Convert org.bytedeco.javacv.Frame to base64

[deimsdeutsch]

Hi,

Can you please tell me how to convert org.bytedeco.javacv.Frame to base64 ? I am sorry i am asking here this but i have been searching this for weeks.

frame grabbed at 240000
loop end with frame: org.bytedeco.javacv.Frame@76ad8565
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00000000667b6cbf, pid=7996, tid=0x0000000000005010
#
# JRE version: Java(TM) SE Runtime Environment (8.0_161-b12) (build 1.8.0_161-b12)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.161-b12 mixed mode windows-amd64 compressed oops)
# Problematic frame:
# V  [jvm.dll+0x1e6cbf]
#
# Failed to write core dump. Minidumps are not enabled by default on client versions of Windows
#
# An error report file with more information is saved as:
# C:\Users\Admin\Desktop\war\hs_err_pid7996.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#
[udp @ 0000000023da4f80] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
[udp @ 0000000023da5080] 'circular_buffer_size' option was set but it is not supported on this build (pthread support is required)
Input #0, rtsp, from 'rtsp://admin:admin123@192.168.1.193:554/Streaming/Channels/101':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.240000, bitrate: N/A
    Stream #0:0: Video: h264 (Main), yuv420p(progressive), 1920x1080, 25 fps, 25 tbr, 90k tbn, 50 tbc

    public static String grab(String SOURCE_RTSP)
    {
        try {
            FFmpegFrameGrabber grabber = new FFmpegFrameGrabber(SOURCE_RTSP);
            /**
             * "timeout" - IS IGNORED when a network cable have been unplugged
             * before a connection and sometimes when connection is lost.
             *
             * "rw_timeout" - IS IGNORED when a network cable have been
             * unplugged before a connection but the option takes effect after a
             * connection was established.
             *
             * "stimeout" - works fine.
             */
            grabber.setOption(
                    TimeoutOption.STIMEOUT.getKey(),
                    String.valueOf(TIMEOUT * 1000000)
            ); // In microseconds.
            grabber.start();

            Frame frame = null;
            /**
             * When network is disabled (before brabber was started) grabber
             * throws exception: "org.bytedeco.javacv.FrameGrabber$Exception:
             * avformat_open_input() error -138: Could not open input...".
             *
             * When connections is lost (after a few grabbed frames)
             * grabber.grab() returns null without exception.
             */
            while ((frame = grabber.grab()) != null) {
                System.out.println("frame grabbed at " + grabber.getTimestamp());
                break;
            }
            grabber.close();

            System.out.println("loop end with frame: " + frame);

            Java2DFrameConverter converter = new Java2DFrameConverter();
            BufferedImage buf = converter.getBufferedImage(frame);            

            return BufToBase64(buf);

        } catch (FrameGrabber.Exception ex) {
            System.out.println("exception: " + ex);
        }

        return null;
    }

Thanks..

[teocci]

@deimsdeutsch why do you want to convert a frame in a base64? probably there is another way to work around. Remember that a Frame instance (frame in your code) is like a container. It contains a metadata and either an image buffer or an audio sample buffer. If you want to extract the image first you must validate if it contains a not null frame.image. Something like this:

if (frame.image != null) {
    BufferedImage buf = converter.convert(frame);
    if (buf != null) {
        return BufToBase64(buf);
    }
}

[xiaozhubenben]

public static String BufferedImageToBase64(BufferedImage image) {
    ByteArrayOutputStream outputStream = new ByteArrayOutputStream();
    try {
        ImageIO.write(image, "jpg", outputStream);
    } catch (IOException e) {
        e.printStackTrace();
    }
    BASE64Encoder encoder = new BASE64Encoder();
    return encoder.encode(outputStream.toByteArray());
}

[Osiris-Team]

@deimsdeutsch Your solution is almost correct, here it is fixed:

            Java2DFrameConverter converter = new Java2DFrameConverter();
            BufferedImage buf = converter.getBufferedImage(frame);
            byte[] arr = compressImage(buf);
            String base64Img = new String(Base64.getEncoder().encode(buf)));
        ...

    public static byte[] compressImage(BufferedImage image) throws IOException {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        ImageIO.write(image, "jpg", baos); // Using JPEG format for compression
        baos.flush();

        byte[] byteArray = baos.toByteArray();
        baos.close();

        return byteArray;
    }

Note that this is pretty CPU demanding and thus not exactly suitable if you want high FPS, thus I would want a better solution too myself.

Optimal solution would be converting the Frame directly to base64.

Update, found a slightly more performant solution:

    static {
        Loader.load(opencv_java.class);
    }

        public static final OpenCVFrameConverter.ToMat toMatConverter = new OpenCVFrameConverter.ToMat();

    public static byte[] frameToJPG(Frame frame) {
        // Convert OpenCV Mat to JPEG byte array

        // Create a ByteArrayOutputStream to store the JPEG data
        MatOfByte outputStream = new MatOfByte();

        // Convert OpenCV Mat to JavaCV Mat
        Mat convertedMat = toMatConverter.convertToOrgOpenCvCoreMat(frame);

        // Encode the Mat as JPEG and write it to the ByteArrayOutputStream
        Imgcodecs.imencode(".jpg", convertedMat, outputStream);

        // Get the byte array from the ByteArrayOutputStream
        byte[] jpgData = outputStream.toArray();

        // Release resources
        convertedMat.release();

        return jpgData;
    }

[martin-1415]

Hi, how to import MatOfByte? @deimsdeutsch

EDIT: Ah ok, you are not using bytedeco in your code.

[saudet]

@martin-1415 You'll need to call Loader.load(opencv_java.class) as documented here: https://github.com/bytedeco/javacpp-presets/tree/master/opencv#documentation