search

PyAV-Org / PyAV

Pythonic bindings for FFmpeg's libraries.
https://pyav.basswood-io.com/
BSD 3-Clause "New" or "Revised" License
2.52k stars 365 forks source link

Can't use stream added from template for anything other than remuxing #730

Closed alexhutnik closed 2 years ago

alexhutnik commented 3 years ago

Overview

When I create a new Container, and add a new stream to it using a template, I can only remux packets. I cannot decode and then re-encode packets.

Expected behavior

I would expect to be able to roundtrip a packet like decode packet -> frame -> encode frame -> packet.

Actual behavior

When trying to do this roundtrip, I am presented with an error.

Traceback:

File "av/stream.pyx", line 152, in av.stream.Stream.encode
File "av/codec/context.pyx", line 476, in av.codec.context.CodecContext.encode
File "av/codec/context.pyx", line 391, in av.codec.context.CodecContext._send_frame_and_recv
File "av/error.pyx", line 336, in av.error.err_check
av.error.ValueError: [Errno 22] Invalid argument

Investigation

I've dug about as far in to the pyav source as I can, but I'm not very familiar with Cython

Reproduction

input_container = av.open('in.mp4', 'r')
input_stream = input_container.streams.get(video=0)[0]

output_container = av.open('out.mp4', 'w')
output_stream = output_container.add_stream(template=input_stream)

for frame in input_container.decode(input_stream):
    # encode and mux
    for packet in output_stream.encode(frame):
        packet.pts = frame.pts
        packet.time_base = frame.time_base
        # packet.stream = output_stream
        output_container.mux(packet)

output_container.close()

Versions

Research

I have done the following:

Additional context

I only installed ffmpeg so I could inspect some files. I did not build PyAV against it.

alexhutnik commented 3 years ago

Turns out #507 is having a similar issue. One commenter mentions that avcodec_copy_context is deprecated and that two other methods should be used. I've implemented these and the relevant data structures, and yet the problem persists. See this commit: https://github.com/alexhutnik/PyAV/commit/1213f02dc28a14aa803753a8b57f9eae9e07604a

alexhutnik commented 3 years ago

Something else I noticed. When copying the input codec via the template option the codec context created in the output doesn't strictly qualify as an encoder. See the following assertion fails:

input_container = av.open('in.mp4', 'r')
input_stream = input_container.streams.get(video=0)[0]

output_container = av.open('out.mp4', 'w')
output_stream = output_container.add_stream(template=input_stream)

# this fails
assert output_stream.codec_context.codec.is_encoder == 1

for frame in input_container.decode(input_stream):
    # encode and mux
    for packet in output_stream.encode(frame):
        packet.pts = frame.pts
        packet.time_base = frame.time_base
        # packet.stream = output_stream
        output_container.mux(packet)

output_container.close()

Yet, when I create the codec by hand, the assertion passes.

This seems to matter in https://github.com/FFmpeg/FFmpeg/blob/069d2b4a50a6eb2f925f36884e6b9bd9a1e54670/libavcodec/encode.c#L312 where the codec is checked to see if it supports encoding. If that test fails, it returns error code 22, which is the behavior we're seeing.

alexhutnik commented 3 years ago

I think I've got it figured out. PyAV is copying the input codec as is. However, that codec is constructed to be a decoder. In ffmpeg examples (transcoding.c), they get the ID of the input codec, and look up the matching encoder by that ID. I've implemented this functionality in PyAV and will submit the PR.

CLIsVeryOK commented 3 years ago

I think I've got it figured out. PyAV is copying the input codec as is. However, that codec is constructed to be a decoder. In ffmpeg examples (transcoding.c), they get the ID of the input codec, and look up the matching encoder by that ID. I've implemented this functionality in PyAV and will submit the PR.

yep, I met the same problem, when will this be fixed? thanks~

yudonglin commented 3 years ago

I met the same issue, but only for version 8.0.3 with python 3.9. Version 8.0.2 with python 3.8 (because this version has no wheel for 3.9) will work just fine. Hopefully, this issue will be fixed soon.

alexhutnik commented 3 years ago

I met the same issue, but only for version 8.0.3 with python 3.9. Version 8.0.2 with python 3.8 (because this version has no wheel for 3.9) will work just fine. Hopefully, this issue will be fixed soon.

Did you try building 8.0.3 against python 3.9? You could just install the wheel you built for yourself.

CLTanuki commented 3 years ago

Reproducing:

Not reproducing:

CLTanuki commented 3 years ago

After passing some options to the recorder, the issue is gone:

'vbsf': 'hevc_mp4toannexb',
'x264opts': 'keyint=24:min-keyint=24:no-scenecut',
jlaine commented 2 years ago

The work-in-progress PR #910 reworks how codec contexts are created for streams, I believe it should solve your issue. Would you mind giving it a spin?

CLIsVeryOK commented 2 years ago

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university

github-actions[bot] commented 2 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

CLIsVeryOK commented 2 years ago

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university

yangmin666 commented 1 year ago

I think I've got it figured out. PyAV is copying the input codec as is. However, that codec is constructed to be a decoder. In ffmpeg examples (transcoding.c), they get the ID of the input codec, and look up the matching encoder by that ID. I've implemented this functionality in PyAV and will submit the PR.

hello my friend! I met the same problem and really did not know how to fix it. Now I use the latest version 10.0.0 for PyAV. How could I solve this problem?

CLIsVeryOK commented 1 year ago

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university

yangmin666 commented 1 year ago

Something else I noticed. When copying the input codec via the template option the codec context created in the output doesn't strictly qualify as an encoder. See the following assertion fails:

input_container = av.open('in.mp4', 'r')
input_stream = input_container.streams.get(video=0)[0]

output_container = av.open('out.mp4', 'w')
output_stream = output_container.add_stream(template=input_stream)

# this fails
assert output_stream.codec_context.codec.is_encoder == 1

for frame in input_container.decode(input_stream):
    # encode and mux
    for packet in output_stream.encode(frame):
        packet.pts = frame.pts
        packet.time_base = frame.time_base
        # packet.stream = output_stream
        output_container.mux(packet)

output_container.close()

Yet, when I create the codec by hand, the assertion passes.

This seems to matter in https://github.com/FFmpeg/FFmpeg/blob/069d2b4a50a6eb2f925f36884e6b9bd9a1e54670/libavcodec/encode.c#L312 where the codec is checked to see if it supports encoding. If that test fails, it returns error code 22, which is the behavior we're seeing. How to create the codec by hand?

Arseny-N commented 1 year ago

I case someone is looking for a workaround. This SO post shows one way to fix it. Just copy the codec parameters manually. It worked for me.

codec_name = in_stream.codec_context.name  # Get the codec name from the input video stream.
fps = in_stream.codec_context.rate  # Get the framerate from the input video stream.
out_stream = test_output.add_stream(codec_name, str(fps))
out_stream.width = in_stream.codec_context.width  # Set frame width to be the same as the width of the input stream
out_stream.height = in_stream.codec_context.height  # Set frame height to be the same as the height of the input stream
out_stream.pix_fmt = in_stream.codec_context.pix_fmt  # Copy pixel format from input stream to output stream
CLIsVeryOK commented 1 year ago

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university

RenaKunisaki commented 9 months ago

The workaround doesn't help for me. I get "Cannot rebase to zero time" for every packet sent to the muxer, even when the frame's dts/pts aren't zero.

CLIsVeryOK commented 9 months ago

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university

ReggieMarr commented 8 months ago

Can confirm, I'm also having the same problem

This is my python application:

import av
import av.datasets
import base64
import copy

SOURCE_FILE = os.getenv('SOURCE_FILE', "launch.mp4")
DESTINATION = os.getenv('DESTINATION', "launch_muxed.ts")

def mux_video(source: str, destination: str):

    # Change 'input.mp4' with actual path to mp4 file
    input_container = av.open(source)
    # Get the video stream from the input container
    video_stream = next(s for s in input_container.streams if s.type == 'video')

    # metadata you want to add
    metadata = {'somekey':'some value'}

    output_container = av.open(destination, 'w', format="mpegts")
    # Create new video stream in the output container with the same codec
    out_stream = output_container.add_stream(template=video_stream)
    #out_stream = test_output.add_stream(template=in_stream)  # Using template=in_stream is not working (probably meant to be used for re-muxing and not for re-encoding).

    # TODO find out why this returns None
    # fps = video_stream.codec_context.rate  # Get the framerate from the input video stream.
    # From ffprobe
    fps = "572000/23857"
    meta_stream = output_container.add_stream("srt", str(fps))
    meta_stream.time_base = video_stream.time_base

    # Loop over packets in input container
    for packet in input_container.demux(video_stream):
        # Decode the packet into frames
        for frame in packet.decode():
            # Mux the frame to the output container
            # img_frame = frame.to_image()
            # out_frame = av.VideoFrame.from_image(img_frame)  # Note: to_image and from_image is not required in this specific example.
            # out_packet = out_stream.encode(out_frame)  # Encode video frame
            # output_container.mux(out_packet)
            for packet in out_stream.encode(frame):
                packet.pts = frame.pts
                packet.time_base = frame.time_base
                # packet.stream = output_stream
                output_container.mux(packet)
            # Create a new subtitle packet for each frame
            # Encode binary data as base64
            binary_data = b'some binary data'
            encoded_data = base64.b64encode(binary_data).decode()

            meta_packet = av.Packet(encoded_data)
            meta_packet.pts = frame.pts
            meta_packet.time_base = meta_stream.time_base
            meta_packet.stream = meta_stream
            output_container.mux(meta_packet)

    # Write metadata to the output container
    output_container.mux(metadata)

    # Close the containers
    input_container.close()
    output_container.close()

def main():

    mux_video(SOURCE_FILE, DESTINATION)

if __name__ == '__main__':
    main()

And it gives me this response:

❯ python3 send_video_data.py                                                                                            
Traceback (most recent call last):
  File "send_video_data.py", line 146, in <module>
    main()
  File "send_video_data.py", line 143, in main
    send_video(SOURCE_FILE, DESTINATION)
  File "send_video_data.py", line 103, in send_video
    for packet in out_stream.encode(frame):
  File "av/stream.pyx", line 149, in av.stream.Stream.encode
  File "av/codec/context.pyx", line 479, in av.codec.context.CodecContext.encode
  File "av/frame.pyx", line 50, in av.frame.Frame._rebase_time
ValueError: Cannot rebase to zero time.
(astro_env_3.8.10)
CLIsVeryOK commented 8 months ago

您的来信已收到，谢谢！陈雷同济大学测绘与地理信息学院Thanks for your attention.Chen Lei College of survey and geo-information of Tongji university