bytedeco / javacv

Java interface to OpenCV, FFmpeg, and more
Other
7.53k stars 1.58k forks source link

How to get byte[] more efficient? #2044

Open gordonyfg opened 1 year ago

gordonyfg commented 1 year ago

Application: Java client(FrameGrabber)->websocket.sendBinary->php Server(Imagick->readImageBlob)

grabber = FrameGrabber.createDefault(1);
grabber.setFormat("dshow");
grabber.setBitsPerPixel(8);
grabber.setImageWidth(1920);
grabber.setImageHeight(1080);
grabber.start();
System.out.println("Before frameToByteArray Time: " + System.currentTimeMillis());
byte[] imageBytes = frameToByteArray(frameMat);
System.out.println("After frameToByteArray Time: " + System.currentTimeMillis());
// Send image data as binary message
websocket.sendBinary(imageBytes);
public static byte[] frameToByteArray(Frame frame) throws IOException {
    BufferedImage image = Java2DFrameUtils.toBufferedImage(frame);
    ByteArrayOutputStream os = new ByteArrayOutputStream();
    javax.imageio.ImageIO.write(image, "jpg", os);
    return os.toByteArray();
}

Result

Before frameToByteArray Time: 1687021694023
After frameToByteArray Time: 1687021694124

Time needed for such Frame to byte[] operation is about 100ms

Is there more efficient way? Can i get byte[] directly from Frame and php Imagick can decode it?

saudet commented 1 year ago

We can copy from the ByteBuffer in Frame.image[0], that's efficient.

gordonyfg commented 1 year ago

thank you. i tried that in 2 ways.

ByteBuffer byteBuffer = (ByteBuffer) frameMat.image[0];
byte[] bytes = new byte[byteBuffer.remaining()];
byteBuffer.get(bytes, 0, bytes.length);
websocket.sendBinary(bytes);
int width = frameMat.imageWidth;
int height = frameMat.imageHeight;
ByteBuffer buffer = (ByteBuffer) frameMat.image[0]; 
byte[] bytes = new byte[width * height];
buffer.get(bytes); // Copy the buffer contents to the byte array
websocket.sendBinary(bytes);

however, in my php code

$imageSrc = new \Imagick();
$imageSrc->readImageBlob($binaryData);

It prompts "An error has occurred: no decode delegate for this image format `' @ error/blob.c/BlobToImag" at line "readImageBlob". I've check Imagick, the built-in 'delegates' should be able to detect and convert any image format. Also, the frameToByteArray function can generate a byte[] that Imagick can recognize.

Anything i can setup with FrameGrabber to config the image format of ByteBuffer? like setVideoCodec?

saudet commented 1 year ago

We can use FFmpegFrameRecorder to encode the Frames into something else like JPEG or PNG format, if that's what you're looking for.

Vansonccc commented 1 year ago

How can i use recorder to encode frames to JPG format? Literally I just cover to BufferedImage and then use ImageIO to write currently.

We can use FFmpegFrameRecorder to encode the Frames into something else like JPEG or PNG format, if that's what you're looking for.

/

saudet commented 1 year ago

FFmpeg accepts a printf() like format as filename to output to JPEG files, so something like new FFmpegFrameRecorder("output_%03d.jpeg", width, height) should work.

gordonyfg commented 1 year ago

How can i use recorder to encode frames to JPG format? Literally I just cover to BufferedImage and then use ImageIO to write currently.

We can use FFmpegFrameRecorder to encode the Frames into something else like JPEG or PNG format, if that's what you're looking for.

/

I tried bufferedImage and ImageoIO, but still there is around 100ms latency for the conversion (frame to bufferedImage to byteArray). I really want to make it as short as possible.

I look into sending the frame.image[0] again. @saudet Is this ByteBuffer start with something like $JPEG = "\xFF\xD8\xFF" $GIF = "GIF" $PNG = "\x89\x50\x4e\x47\x0d\x0a\x1a\x0a" $BMP = "BM" $PSD = "8BPS" $SWF = "FWS" ? I am trying to decode it in the php server directly. but no luck.