Asking for help with hardware decoding to convert into JPG images.

[DarkiT]

Thanks for the great work you have done with this library!

I am using the hardware_decoding example for hardware decoding, and I have obtained Frame.Data(). However, I keep encountering errors when trying to save the frame as a JPG image, which prevents me from proceeding.

I plan to take timed screenshots with my camera and send them to WeChat, but due to the aforementioned errors, I can't continue. Could someone help me? Here is the decoder information for my camera's VLC streaming.

Can you provide a demo for saving a frame to an image?

[asticode]

Could you share the code and the error you're getting?

Anyway check out this part of the Demuxing/Decoding example on how to properly generate a go image with the data of a frame. Once you get an image.Image, you can use jpeg.Encode or png.Encode to save it to a file. Beware as you may need to convert to proper RGBA in order to get good result.

[DarkiT]

This is my streaming information.

Input #0, rtsp, from 'rtsp://192.168.1.175':
  Metadata:
    title           : Media Presentation
  Duration: N/A, start: 0.040000, bitrate: N/A
  Stream #0:0: Video: hevc (Main), yuv420p(tv), 2560x1440, 25 fps, 25 tbr, 90k tbn

Now it's working properly, but the conversion efficiency is too low. How can I use RKMPP hardware acceleration to achieve image transcoding?

if s.inputStream.CodecParameters().MediaType() == astiav.MediaTypeVideo {

    fd := finalFrame.Data()

    var err error
    if i, err = fd.GuessImageFormat(); err == nil {
        if err := fd.ToImage(i); err != nil {
            log.Error("guessing image format failed: %v", err)
        }
    } else {
        nv12, err := fd.Bytes(1)
        if err != nil {
            log.Error("copying frame data to the image failed: %v", err)
            return
        }
        i = NV12ToRGBA(nv12, finalFrame.Width(), finalFrame.Height())
    }
}

// NV12ToRGBA converts NV12 formatted data to an RGBA image.
func NV12ToRGBA(nv12 []byte, width, height int) *image.RGBA {
    // Create an RGBA image
    rgba := image.NewRGBA(image.Rect(0, 0, width, height))

    // Iterate through each pixel in the NV12 data and convert it to RGBA
    for y := 0; y < height; y++ {
        for x := 0; x < width; x++ {
            // Calculate indices for Y, U, and V components
            yIndex := y*width + x
            uvIndex := width*height + (y/2)*width + (x/2)*2

            // Extract Y, U, and V components
            yComp := float64(nv12[yIndex])
            uComp := float64(nv12[uvIndex])
            vComp := float64(nv12[uvIndex+1])

            // Convert YUV to RGB
            r := clamp(yComp + 1.402*(vComp-128))
            g := clamp(yComp - 0.344136*(uComp-128) - 0.714136*(vComp-128))
            b := clamp(yComp + 1.772*(uComp-128))

            // Set the RGBA values
            rgba.Set(x, y, color.RGBA{R: r, G: g, B: b, A: 255})
        }
    }

    return rgba
}

// Helper function to clamp a value to the range [0, 255]
func clamp(v float64) uint8 {
    if v < 0 {
        return 0
    }
    if v > 255 {
        return 255
    }
    return uint8(v)
}

[asticode]

You could use a filter instead (checkout format and possibly hwdownload as well as the example) and if your goal is to transcode checkout this example

[Tryanks]

If your target platform is RKMPP, you may need to use go-astiav/bump/n6.1 (see issues #27 and #57). For hardware encoding and decoding, consider using ffmpeg-rockchip. The mainline FFmpeg does not support Rockchip Mpp.

[asticode]

Closing this issue, but feel free to reopen it if needed 👍