Raw Video Stream Data

[Tr4in]

Hello Insta360-Team,

I am using an Insta360 ONE X2 and I wanted to ask what is meant by "raw stream data every frame"? My goal is to stream those frames in real-time using WebRTC to the server. How can I process them? Which color space do these frames have?

@Override
public void onVideoData(VideoData videoData) {
// Callback frequency 500Hz
// videoData.timestamp: The time since the camera was turned on
// videoData.data: Preview raw stream data every frame
// videoData.size: videoData.data.length
}

[djoannid-iti]

Dear @Tr4in ,

did you manage to get the data in Image File or Preview in customized frame in Java ?

@Insta360-team,

an example in the sdk for the raw stream data would be helpful.

Dimos

[Tr4in]

Hey @djoannid-iti

I have sent the raw stream data frame via UDP Socket to my server. I used ffmpeg to process the incoming stream and found out that the video-codec was h264 or h265 depending on the codec I set on the Insta360 X2 camera. The color space was yuvj420p as far as I could remember - but maybe I am wrong about that.

[mordekai91]

Hi @djoannid-iti, did you solve this?

Im stuck trying to decode videoData. If you have any idea it would be nice Thank you :)

[djoannid-iti]

Hi @djoannid-iti, did you solve this?

Im stuck trying to decode videoData. If you have any idea it would be nice Thank you :)

Yes we've managed to do so, please send me an email to djoannid@iti.gr for following up.

[Tianweihaihaihai]

"raw stream data every frame" means unstitched video data. "color space do these frames" is 601

[DawidFulmesOculo]

Hi @djoannid-iti, did you solve this? Im stuck trying to decode videoData. If you have any idea it would be nice Thank you :)

Yes we've managed to do so, please send me an email to djoannid@iti.gr for following up.

Do you still have solution for decoding those data?

[mordekai91]

Hi @djoannid-iti, did you solve this? Im stuck trying to decode videoData. If you have any idea it would be nice Thank you :)

Yes we've managed to do so, please send me an email to djoannid@iti.gr for following up.

Do you still have solution for decoding those data?

Yes! you have to override onOpening() InstaCameraManager method with the following:

@Override
    public void onOpening() {
        createSurfaceView();
    }

 private ImageReader mImageReader;
    private HandlerThread mImageReaderHandlerThread;
    private Handler mImageReaderHandler;

    private void createSurfaceView() {
        if (mImageReader != null) {
            return;
        }
        mImageReaderHandlerThread = new HandlerThread("camera render surface");
        mImageReaderHandlerThread.start();
        mImageReaderHandler = new Handler(mImageReaderHandlerThread.getLooper());
        mImageReader = ImageReader.newInstance(1024, 512, PixelFormat.RGBA_8888, 2);

        mImageReader.setOnImageAvailableListener(reader -> {
            if (reader.getMaxImages() > 0) {
                Image image = reader.acquireLatestImage();
                if (imageCounter % 3 == 0) {
                    Image.Plane plane = image.getPlanes()[0];
                    int pixelStride = plane.getPixelStride();
                    int rowStride = plane.getRowStride();
                    int rowPadding = rowStride - pixelStride * image.getWidth();
                    bitmap = Bitmap.createBitmap(image.getWidth() + rowPadding / pixelStride, image.getHeight(), Bitmap.Config.ARGB_8888);
                    bitmap.copyPixelsFromBuffer(plane.getBuffer());

                   //Here you have the preview as bitmap with equirectangular format

                    image.close();
                } else {
                    image.close();
                }
            }
            imageCounter++;
        }, mImageReaderHandler);
    }

private CaptureParamsBuilder createCaptureParams() {
        return new CaptureParamsBuilder()
                .setCameraType(InstaCameraManager.getInstance().getCameraType())
                .setMediaOffset(InstaCameraManager.getInstance().getMediaOffset())
                .setCameraSelfie(InstaCameraManager.getInstance().isCameraSelfie())
                .setLive(false)
                .setStabType(InstaStabType.STAB_TYPE_CALIBRATE_HORIZON)
                .setResolutionParams(1024, 512, 10)

                **//add next lines**
                .setRenderModelType(CaptureParamsBuilder.RENDER_MODE_PLANE_STITCH).setScreenRatio(2, 1)
                .setCameraRenderSurfaceInfo(mImageReader.getSurface(), mImageReader.getWidth(), mImageReader.getHeight());

    }

Hope it works for you!