Using "segment" muxer to fragment a large mp4 file into segments crashes after writing first two segments

[jmoraleda]

Hi Mike,

I am getting to know your library using it for a project and I really like it! Thank you for creating it! I am trying to create a fragmenter to break a large mp4 into smaller ones for streaming without transcoding, just by copying packets. Basically the same thing that the following ffmpeg command line does:

ffmpeg -i my_large_input_file.mp4 -codec copy -f segment -segment_time 3 out%03d.ts

I thought I would start with the remux code at https://github.com/mikeboers/PyAV/blob/44a46cc265cf012b2ddfb88aec4f564374241d93/scratchpad/remux.py

and then just switch the muxer type to "segment" but I cannot get the original code in remux.py to work. In the stable 3.3 version I get the error:

00 <av.Packet of #0, dts=-1001, pts=0 at 0x7fa05a1b8340>
    in:  <av.VideoStream #0 h264, yuv420p 640x360 at 0x7fa05a1b8178>
    out: <av.VideoStream #0 h264, yuv420p 640x360 at 0x7fa05a1b82a8>
Traceback (most recent call last):
  File "/home/jorge/vc/focalized_visualization/src/core/remux_original.py", line 78, in <module>
WARNING:libav.mp4:Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
WARNING:libav.mp4:Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
WARNING:libav.mp4:Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
WARNING:libav.mp4:Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
WARNING:libav.mp4:Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
WARNING:libav.mp4:Packet with invalid duration -9223372036854775808 in stream 0
    output.mux(packet)
  File "av/container/output.pyx", line 178, in av.container.output.OutputContainer.mux (src/av/container/output.c:3375)
  File "av/container/core.pyx", line 230, in av.container.core.ContainerProxy.err_check (src/av/container/core.c:4052)
  File "av/utils.pyx", line 76, in av.utils.err_check (src/av/utils.c:1673)
av.AVError: [Errno 22] Invalid argument: 'test.mp4'

and in the latest version compiled from the master branch from github I get:

00 <av.Packet of #0, dts=-1001, pts=0; 15011 bytes at 0x7f9d17715868>
    in:  <av.VideoStream #0 h264, yuv420p 640x360 at 0x7f9cfaa22048>
    out: <av.VideoStream #0 h264, None 640x360 at 0x7f9cfaa22128>
Traceback (most recent call last):
  File "/home/jorge/vc/focalized_visualization/src/core/remux_original.py", line 78, in <module>
    output.mux(packet)
  File "av/container/output.pyx", line 195, in av.container.output.OutputContainer.mux (src/av/container/output.c:4133)
  File "av/container/output.pyx", line 201, in av.container.output.OutputContainer.mux_one (src/av/container/output.c:4363)
  File "av/container/output.pyx", line 132, in av.container.output.OutputContainer.start_encoding (src/av/container/output.c:3052)
AttributeError: 'NoneType' object has no attribute 'update'

I am wondering if you understand what it is going on. In both cases the command line that I am using is: python3 remux.py input.mp4 test.mp4 (I trivially modify remux.py to use python3). This happens regardless of the video file that I use for input. One such file is : http://www.sample-videos.com/video/mp4/360/big_buck_bunny_360p_1mb.mp4

I am in linux (debian stretch up to date) and I installed the development libraries of libav via apt-get.

[mikeboers]

Honestly... I don't know if the remux code in there ever worked. Given a pile of the recent overhauling it might be the closest to working it even has been, but I haven't tried in a while.

There is so much going on in the ffmpeg command itself dealing with timestamps and other garbage.

I do have someone that has been doing a lot of work with PyAV and HLS. I'm going to kick them an email to see if they have some input for you.

[mikeboers]

That second part (the lack of an update method) is definitely my bad.

[jmoraleda]

Great. Thank you for getting back to me. I would love to hear from anyone who has HLS working. Also, let me know if there is anything I can do to help debug this.

[GalanB]

Hi, Jorge. I've been working with PyAV and HLS a lot lately and will offer whatever help I can.

I should state up front that I am using the HLS muxer, not the 'segment' muxer, and I'm outputting .ts segments not .mp4 files, and I don't quite understand why you want 3 second .mp4 files for "streaming", but be that as it may, since you find the output of your sample ffmpeg statement fits your needs, I'll try to focus on how I think you could get that to work.

A few things come to mind. For one, as Mike said, PyAV has been getting quite an overhaul lately, so be prepared that if you opt to use the latest version, many of the previous examples will need some modifications now to work with the new model. Mike has been making it much more symmetric between how decoding and encoding now work, and it's modeled a lot after how ffmpeg does it. The introduction of the CodecContext in particular will affect how you need to code for the new model.

I agree that for your desired functionality the remux.py example seems like a good starting point. But it may not yet have been complete. I know for my case, since I did need to do a full transcode, I found it helpful to refer to ffmpeg's own transcoding.c example to learn the ideal approach. In your case, the ffmpeg remuxing.c example may likewise prove helpful. I'd keep that in mind just in case you need to go there.

In your first example from the Stable 3.3 release, it looks like the issue may have been the output filename. The segment muxer wants a template, such as the out%03d.mp4 that you used in your ffmpeg example. I would try changing that first. (You do have a format='segment' parm in the av.open() call to ensure the segment muxer is getting invoked, right?) Also, you don't show your code, but hopefully you're setting an options dictionary with your segment_time=3 option, if that's your desired duration, and maybe some additional 'segment' options? Since you want to "stream" these files, you may also find the segment_format_options movflags=+faststart option helpful.

In your 2nd example on the latest release, it does look like the lack of an 'update' method as Mike states is the immediate issue. But I wouldn't be surprised if once you get past that you may encounter new issues related to the new model. And did you pass down the '-codec copy' option from your ffmpeg example to ensure transcoding is not necessary? I do think that would make your remuxing job as simple as possible. In my case I also had to do a lot of work preserving and rebasing the timestamps during the transcode because my stream was live. It'd be good if you didn't have to deal with that.

I think if I were you I'd try to get it to work under the 3.3 release first, as it is the one that's had the most testing and that the examples are designed to work with.

One last thing. If you are 'copying' (preserving) the encoded packets instead of transcoding them, your IDR frames may not align well with your desired 3-second segment sizes. It could make for some uneven segment durations. If so and you find yourself needing a full transcode (demux/decode/encode/mux) then you may find the new model more desirable, and since you're working with files instead of live streams like I was, it's been tested for that and with your zero-based PTS's it may work fine.

--Galan

[jmoraleda]

Hi Galan,

Thank you for your detailed response! This is the remux code that I referred to in my post. It is almost identical to the one in the repository, which is why I didn't post it the first time:

import array
import argparse
import logging
import sys
import pprint
import subprocess

from PIL import Image

from av import open, time_base

logging.basicConfig(level=logging.DEBUG)

def format_time(time, time_base):
    if time is None:
        return 'None'
    return '%.3fs (%s or %s/%s)' % (time_base * time, time_base * time, time_base.numerator * time, time_base.denominator)

arg_parser = argparse.ArgumentParser()
arg_parser.add_argument('input')
arg_parser.add_argument('output')
arg_parser.add_argument('-F', '--iformat')
arg_parser.add_argument('-O', '--ioption', action='append', default=[])
arg_parser.add_argument('-f', '--oformat')
arg_parser.add_argument('-o', '--ooption', action='append', default=[])
arg_parser.add_argument('-a', '--noaudio', action='store_true')
arg_parser.add_argument('-v', '--novideo', action='store_true')
arg_parser.add_argument('-s', '--nosubs', action='store_true')
arg_parser.add_argument('-d', '--nodata', action='store_true')
arg_parser.add_argument('-c', '--count', type=int, default=0)
args = arg_parser.parse_args()

input_ = open(args.input,
    format=args.iformat,
    options=dict(x.split('=') for x in args.ioption),
)
output = open(args.output, 'w',
    format=args.oformat,
    options=dict(x.split('=') for x in args.ooption),
)

in_to_out = {}

for i, stream in enumerate(input_.streams):

    if (
        (stream.type == 'audio' and not args.noaudio) or
        (stream.type == 'video' and not args.novideo) or
        (stream.type == 'subtitle' and not args.nosubtitle) or
        (stream.type == 'data' and not args.nodata)
    ):
        in_to_out[stream] = ostream = output.add_stream(template=stream)

for i, packet in enumerate(input_.demux(tuple(in_to_out.keys()))):

    if args.count and i >= args.count:
        break
    print('%02d %r' % (i, packet))
    print('\tin: ', packet.stream)

    if packet.dts is None:
        continue

    packet.stream = in_to_out[packet.stream]

    print('\tout:', packet.stream)

    output.mux(packet)

output.close()

This is the code that I wrote to use the "segment" muxer to output ts segments. (Converting each segment to mp4 is something that I may experiment with after I get the code working to output segments. I mentioned it because this is why I started originally looking into PyAv since I wanted to go beyond what the ffmpeg executable can do, but it is not relevant to my current problems).

import av
import argparse

arg_parser = argparse.ArgumentParser()
arg_parser.add_argument('input')
arg_parser.add_argument('-output',default='test%03d.ts')
arg_parser.add_argument('-t', '--time', help='Minimum interval of a segment', default='3')
args = arg_parser.parse_args()

input_ = av.open(args.input)
output = av.open(args.output, 'w', format='segment', options={'segment_time':args.time})  

inOutStreamMap = {}  
for stream in input_.streams:
    inOutStreamMap[stream] = output.add_stream(template=stream)

for packet in input_.demux(tuple(inOutStreamMap.keys())):

    if packet.dts is None:
        continue

    print(packet)    
    #print (packet.duration)

    print('\tin:', packet.stream)
    packet.stream = inOutStreamMap[packet.stream]
    print('\tout:', packet.stream)
    output.mux(packet)

output.close()

They both fail in the same way in the output.mux(packet) line, that is why I thought I would make the question about the remux first. In the segmenter program, I am using the default output name (with pattern) and segment_time.

They fail in all of the mp4 files that I have tried both in 3.3 and in the latest code (although the error messages are different as I mentioned in the earlier email). One such example is: http://www.sample-videos.com/video/mp4/360/big_buck_bunny_360p_1mb.mp4

Do you understand what is going on?

Thank you again!

[GalanB]

Hi, Jorge. Thanks for the code listings. That does help.

So in the 2nd example, it's good, clean, simple code that I suspect is very close to working, but I think Mike may need to address the issue of there not being a dictionary in the stream.codec_context.options object with an 'update' value. I tried setting packet.stream.codec_context.options = {a dictionary} in your code so there might be an 'update' there, but it somehow always got nulled out before reaching start_encoding() in output.pyx. Perhaps Mike knows a better workaround than what I was trying? That or it'd take a code change to PyAV.

The reason I mentioned setting the output file to the template pattern was because in your earlier first example the error message said "[Errno 22] Invalid argument: 'test.mp4'", so it seemed like your template filename wasn't getting applied?

I didn't try backrev-ing my PyAV to 3.3 and testing your new code against it. Have you posted what those error messages are? I don't see much (if anything) your new code would lack if called against the 3.3 code, so I was just curious what the error messages are in that case.

--Galan

[jmoraleda]

Hi Galan,

Thank you again. This is the exact error I get with the second program when I use the defaults. It is the same as before.

Traceback (most recent call last):
  File "/home/jorge/vc/focalized_visualization/src/core/remux.py", line 41, in <module>
Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Using AVStream.codec.time_base as a timebase hint to the muxer is deprecated. Set AVStream.time_base instead.
Using AVStream.codec to pass codec parameters to muxers is deprecated, use AVStream.codecpar instead.
Timestamps are unset in a packet for stream 0. This is deprecated and will stop working in the future. Fix your code to set the timestamps properly
Packet with invalid duration -9223372036854775808 in stream 0
    output.mux(packet)
  File "av/container/output.pyx", line 178, in av.container.output.OutputContainer.mux (src/av/container/output.c:3375)
  File "av/container/core.pyx", line 230, in av.container.core.ContainerProxy.err_check (src/av/container/core.c:4052)
  File "av/utils.pyx", line 76, in av.utils.err_check (src/av/utils.c:1673)
av.AVError: [Errno 22] Invalid argument: 'test%03d.ts'

After I saw this, I tried to use just "test.mp4" as output to see if it made any difference, but it does not (that's the error I pasted), so I suspect the problem is something else. Also, out of curiosity, I tried to use an invalid segment template name in the ffmpeg command line. The error ffmpeg reports is:

[mpegts @ 0x561d0a23cac0] Invalid segment filename template 'blah.ts'
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument

[GalanB]

Jorge,

So this is new, though ... "av.AVError: [Errno 22] Invalid argument: 'test%03d.ts'". The templated filename didn't take. If that's the new program against the 3.3 PyAV release, then it looks like the new program doesn't get as far along with the old release as it does with the latest release. I tried your program against my PyAV (latest version plus a few mods for live) and I got past the open() and it didn't fail until the mux().

It'd be good to see what Mike thinks about fixing the lack of 'update'. And beyond that, it's possible you may need to set a few codec_context values to make it work.

--Galan

[hhsprings]

AttributeError: 'NoneType' object has no attribute 'update'

I encounter the same problem.

This error occurs at av/container/output.pyx#L132:

            if not ctx.is_open:

                ctx.options.update(self.options)
                ctx.open()

I tried to fix like this:

            if not ctx.is_open:

                if ctx.options:
                    ctx.options.update(self.options)
                else:
                    ctx.options = dict(self.options)
                ctx.open()

It seems working fine to me. Is this fix right? (I don't understand the PyAV internal well.)

[GalanB]

Hi, Hiroaki.

That's interesting. I'm not sure what the intent there was, either, but your fix does look pretty good to me! @jmoraleda, you might want to try this and recompile PyAV. If it gets you past this point you could be a lot closer to working.

--Galan

[hhsprings]

That's interesting. I'm not sure what the intent there was, either, but your fix does look pretty good to me! @jmoraleda, you might want to try this and recompile PyAV. If it gets you past this point you could be a lot closer to working.

Great. But... this is Cython not Python, so I'm afraid some kind of memory allocation issue, because of my lack of knowlege of PyAV internals. So, even if your code works fine, you should not trust me, and wait for Mike's fix.

[jmoraleda]

Thank you everyone! Sorry for my late reply, I was away from my development computer for a couple of days. @hhsprings, I just tried your patch and the segment code that I posted early works with it and produces valid segments. Thank you again @mikeboers for a great library and everyone for your responsiveness!

I remark that my code above produces the same number of segments as the ffmpeg command line with the same arguments, but segment files produced this way are silghtly larger than those produced by the command line. I do not think this is an issue, but I bring it up for the record.

I was going to close the issue, but I figured you may want to leave it open until this patch or an equivalent one is applied to the code in the repository. Thank you again.

[clyde-tressler]

hi everyone- is there working sample code for wrapping aac as ts segments? thx clyde

[jlaine]

I'm closing this, the following works fine:

python scratchpad/remux.py --oformat segment tests/assets/pyav-curated/pexels/time-lapse-video-of-night-sky-857195.mp4 out%03d.ts