Can I Decode-> Torch -> Encode a video without passing it through CPU (numpy)?

[sotelo]

Hello @RomanArzumanyan,

First of all, thank you for continuing your work in VALI! I think that it's an amazingly useful piece of software. In my application, I used to use VPF for reading, processing (with PyTorch) and writing a video without moving the data to CPU. I'm porting my code to VALI and there's a couple of questions that I have in the process:

1. What's the expected way to bring the data back from torch into a nvc.Surface object for encoding?
2. What's the expected way to convert back nvc.PixelFormat.RGB into nvc.PixelFormat.NV12 to write back the video?

We get ValueError: Unsupported pixel format conversion with this code:

to_nv12 = nvc.PySurfaceConverter(
    nvc.PixelFormat.RGB, nvc.PixelFormat.NV12, gpu_id
)

[RomanArzumanyan]

Hi @sotelo Thank you for the kind talk.

1) Please take a look at tests/test_PySurface.py unit test file. Within it, conversion between Surface and Tensor is tested in both directions.

2) Colour conversion is done with the help of NPP library which unfortunately doesn’t support (at least at the moment) conversion from rgb to nv12. However, you can convert rgb to yuv444 and give that surface to encoder. It does support yuv444 just fine. The only thing you need to do is to specify yuv444 pixel format in PyNvEncoder constructor.

[sotelo]

Thanks for the quick answer @RomanArzumanyan!

I was able to successfully convert between Surface and Tensor after looking at the code example. However, when I try to convert the surface from rgb to yuv444 I get the error: TaskExecInfo.UNSUPPORTED_FMT_CONV_PARAMS.

Here's my code (using the video in the test examples):

import PyNvCodec as nvc

# url = "/data/jose/Turnbull.mp4"
url = "/u/home/code/VALI/tests/data/test.mp4"

class StopExecution(Exception):
    def _render_traceback_(self):
        return []

# GPU-accelerated decoder
pyDec = nvc.PyDecoder(
    url,
    {},
    gpu_id=0)

# GPU-accelerated converter
pyCvt = nvc.PySurfaceConverter(
    pyDec.Format(),
    nvc.PixelFormat.RGB,
    gpu_id=0)

# Colorspace conversion context. Here a dummy set of parameters is used
# just for illustration purposes.
cc_ctx = nvc.ColorspaceConversionContext(
    nvc.ColorSpace.BT_709,
    nvc.ColorRange.MPEG)

# Raw decoded Surface
surf_src = nvc.Surface.Make(
    format=pyDec.Format(),
    width=pyDec.Width(),
    height=pyDec.Height(),
    gpu_id=0)

# Raw Surface, converted to RGB
surf_dst = nvc.Surface.Make(
    format=nvc.PixelFormat.RGB,
    width=pyDec.Width(),
    height=pyDec.Height(),
    gpu_id=0)

surface_yuv444 = nvc.Surface.Make(
    format=nvc.PixelFormat.YUV444,
    width=pyDec.Width(),
    height=pyDec.Height(),
    gpu_id=0,
)

to_yuv444 = nvc.PySurfaceConverter(
    nvc.PixelFormat.RGB,
    nvc.PixelFormat.YUV444,
    gpu_id=0,
)

success, details = pyDec.DecodeSingleSurface(surf_src)
if not success:
    print(details)

# Convert tot RGB
success, details = pyCvt.Run(surf_src, surf_dst, cc_ctx)
if not success:
    print(details)

frame = torch.from_dlpack(surf_dst)
frame = torch.reshape(frame, (pyDec.Height(), pyDec.Width() * 3))

surface = nvc.Surface.from_dlpack(
    torch.utils.dlpack.to_dlpack(frame), nvc.PixelFormat.RGB)

success, details = to_yuv444.Run(surface, surface_yuv444, cc_ctx)

print(success)
print(details)

[RomanArzumanyan]

Hi @sotelo

That’s the usual color conversion parameters business. Only JPEG cooor range is supported in rgb 2 yuv444 conversion by npp: https://github.com/RomanArzumanyan/VALI/blob/c4ce78d5e06beda2ada8f680078044b8cd3d4654/src/TC/src/TaskConvertSurface.cpp#L654

I should probably make cconv params argument completely optional and use default supported parameters if nothing is given by user.

Cconv parameters support was intended to improve color conversion accuracy but it looks to me now that it should be optional and not mandatory.

[RomanArzumanyan]

@sotelo

Please LMK if the issue is resolved and can be closed.

[sotelo]

Yes, thank you.