Video Decoding from Custom Stream: Way to Set Buffer Size?

[kentborg]

Oh, am I impressed with how easy PyAV makes using ffmpeg! Thanks.

I am decoding video from a custom stream, I am managing to decode data in RAM and get image frames out. But the decoder wants to buffer data before giving me the first frame out. I would like to get each frame out as promptly as possible, and as I feed more data in, get out new frames--again as promptly as possibly.

Googling around there seems to be an ffmpeg option to control this: codec_ctx->delay (mentioned in http://ffmpeg.org/pipermail/libav-user/2014-December/007672.html).

Is that what I am looking for? Is there a way to set it via an av.open() option?

Thanks!

-kb

[mikeboers]

I exposed that via Stream.delay on the gh150 branch (in commit 1fad83a2).

Let us know if that does what you are looking for, and I'll figure out where the best place to put that in the API is (likely where I just put it).

Cheers!

[kentborg]

Cool!

Only downside is I don't think I can get to it for a couple days. Grrrr.

Thanks,

-kb

[kentborg]

Stupid question: How do I use it? I see the changes in av/stream.pyx, how do I set that in the canonical here-is-how-you-play-a-file example code I started from from https://mikeboers.github.io/PyAV/index.html ?

Thanks,

-kb

[kentborg]

Light of morning (before hitting the road) it looks like it will just show up as a property. Once I build 1fad83a.

src/av/codec.c:286:33: fatal error: libavfilter/avcodec.h: No such file or directory

include "libavfilter/avcodec.h"

I have /usr/include/libavcodec/avcodec.h

A libav vs. ffmpeg issue? On Debian 8.

Try "apt-get install libav-tools" (warned ffmpeg will be removed), same error.

[mikeboers]

How did you install it before? What version did you have before? Can we see a more complete transcript of how you are attempting to build it?

[kentborg]

I did not succeed in building before. So my working case was, in the directory where I am hacking Python:

$ picamera
$ virtualenv venv
$ . venv/bin/activate
$ pip install av
$ pip install image

My failing case (one of my tries, first one today) was:

kentborg@stoneflute:~/python/PyAV$ git checkout gh150
kentborg@stoneflute:~/python/PyAV$ virtualenv venv
Running virtualenv with interpreter /usr/bin/python2
New python executable in venv/bin/python2
Not overwriting existing python script venv/bin/python (you must use venv/bin/python2)
Installing setuptools, pip...done.
kentborg@stoneflute:~/python/PyAV$ . venv/bin/activate
(venv)kentborg@stoneflute:~/python/PyAV$ pip install cython
Requirement already satisfied (use --upgrade to upgrade): cython in ./venv/lib/python2.7/site-packages
Cleaning up...
(venv)kentborg@stoneflute:~/python/PyAV$ python setup.py build_ext --inplace
running build_ext
running reflect
running config
looking for avformat_open_input... found
looking for pyav_function_should_not_exist... missing
looking for av_calloc... missing
looking for av_frame_get_best_effort_timestamp... missing
looking for avformat_alloc_output_context2... missing
looking for avformat_close_input... found
looking for AVStream.index... found
looking for PyAV.struct_should_not_exist... missing
looking for AVFrame.mb_type... found
writing build/temp.linux-x86_64-2.7/include/pyav/config.h
running cythonize
Compiling av/buffer.pyx because it changed.
[1/1] Cythonizing av/buffer.pyx
Compiling av/bytesource.pyx because it changed.
[1/1] Cythonizing av/bytesource.pyx
Compiling av/codec.pyx because it changed.
[1/1] Cythonizing av/codec.pyx
Compiling av/descriptor.pyx because it changed.
[1/1] Cythonizing av/descriptor.pyx
Compiling av/dictionary.pyx because it changed.
[1/1] Cythonizing av/dictionary.pyx
Compiling av/format.pyx because it changed.
[1/1] Cythonizing av/format.pyx
Compiling av/logging.pyx because it changed.
[1/1] Cythonizing av/logging.pyx
Compiling av/option.pyx because it changed.
[1/1] Cythonizing av/option.pyx
Compiling av/packet.pyx because it changed.
[1/1] Cythonizing av/packet.pyx
Compiling av/plane.pyx because it changed.
[1/1] Cythonizing av/plane.pyx
Compiling av/utils.pyx because it changed.
[1/1] Cythonizing av/utils.pyx
Compiling av/_core.pyx because it changed.
[1/1] Cythonizing av/_core.pyx
Compiling av/frame.pyx because it changed.
[1/1] Cythonizing av/frame.pyx
Compiling av/stream.pyx because it changed.
[1/1] Cythonizing av/stream.pyx
Compiling av/audio/fifo.pyx because it changed.
[1/1] Cythonizing av/audio/fifo.pyx
Compiling av/audio/format.pyx because it changed.
[1/1] Cythonizing av/audio/format.pyx
Compiling av/audio/frame.pyx because it changed.
[1/1] Cythonizing av/audio/frame.pyx
Compiling av/audio/layout.pyx because it changed.
[1/1] Cythonizing av/audio/layout.pyx
Compiling av/audio/plane.pyx because it changed.
[1/1] Cythonizing av/audio/plane.pyx
Compiling av/audio/resampler.pyx because it changed.
[1/1] Cythonizing av/audio/resampler.pyx
Compiling av/audio/stream.pyx because it changed.
[1/1] Cythonizing av/audio/stream.pyx
Compiling av/container/core.pyx because it changed.
[1/1] Cythonizing av/container/core.pyx
Compiling av/container/input.pyx because it changed.
[1/1] Cythonizing av/container/input.pyx
Compiling av/container/output.pyx because it changed.
[1/1] Cythonizing av/container/output.pyx
Compiling av/container/streams.pyx because it changed.
[1/1] Cythonizing av/container/streams.pyx
Compiling av/subtitles/stream.pyx because it changed.
[1/1] Cythonizing av/subtitles/stream.pyx
Compiling av/subtitles/subtitle.pyx because it changed.
[1/1] Cythonizing av/subtitles/subtitle.pyx
Compiling av/video/frame.pyx because it changed.
[1/1] Cythonizing av/video/frame.pyx
warning: av/video/frame.pyx:173:17: Non-trivial type declarators in shared declaration (e.g. mix of pointers and values). Each pointer declaration should be on its own line.
warning: av/video/frame.pyx:173:27: Non-trivial type declarators in shared declaration (e.g. mix of pointers and values). Each pointer declaration should be on its own line.
warning: av/video/frame.pyx:173:33: Non-trivial type declarators in shared declaration (e.g. mix of pointers and values). Each pointer declaration should be on its own line.
Compiling av/video/plane.pyx because it changed.
[1/1] Cythonizing av/video/plane.pyx
Compiling av/video/reformatter.pyx because it changed.
[1/1] Cythonizing av/video/reformatter.pyx
Compiling av/video/stream.pyx because it changed.
[1/1] Cythonizing av/video/stream.pyx
Compiling av/video/format.pyx because it changed.
[1/1] Cythonizing av/video/format.pyx
Compiling av/filter/context.pyx because it changed.
[1/1] Cythonizing av/filter/context.pyx
Compiling av/filter/filter.pyx because it changed.
[1/1] Cythonizing av/filter/filter.pyx
Compiling av/filter/graph.pyx because it changed.
[1/1] Cythonizing av/filter/graph.pyx
building 'av.buffer' extension
creating build/temp.linux-x86_64-2.7/src
creating build/temp.linux-x86_64-2.7/src/av
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Iinclude -I/usr/include/python2.7 -Ibuild/temp.linux-x86_64-2.7/include -I/usr/include/python2.7 -Ibuild/temp.linux-x86_64-2.7/include -c src/av/buffer.c -o build/temp.linux-x86_64-2.7/src/av/buffer.o
creating build/lib.linux-x86_64-2.7
creating build/lib.linux-x86_64-2.7/av
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/src/av/buffer.o -lavcodec -lavutil -lavformat -lavresample -lswscale -lavdevice -o build/lib.linux-x86_64-2.7/av/buffer.so
building 'av.bytesource' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Iinclude -I/usr/include/python2.7 -Ibuild/temp.linux-x86_64-2.7/include -I/usr/include/python2.7 -Ibuild/temp.linux-x86_64-2.7/include -c src/av/bytesource.c -o build/temp.linux-x86_64-2.7/src/av/bytesource.o
x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wl,-z,relro -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security build/temp.linux-x86_64-2.7/src/av/bytesource.o -lavcodec -lavutil -lavformat -lavresample -lswscale -lavdevice -o build/lib.linux-x86_64-2.7/av/bytesource.so
building 'av.codec' extension
x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -Iinclude -I/usr/include/python2.7 -Ibuild/temp.linux-x86_64-2.7/include -I/usr/include/python2.7 -Ibuild/temp.linux-x86_64-2.7/include -c src/av/codec.c -o build/temp.linux-x86_64-2.7/src/av/codec.o
In file included from src/av/codec.c:284:0:
include/libswresample/swresample.pyav.h:34:9: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     int swresample_version() { return -1; }
         ^
include/libswresample/swresample.pyav.h:35:17: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     const char* swresample_configuration() { return ""; }
                 ^
include/libswresample/swresample.pyav.h:36:17: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     const char* swresample_license() { return ""; }
                 ^
src/av/codec.c:286:33: fatal error: libavfilter/avcodec.h: No such file or directory
 #include "libavfilter/avcodec.h"
                                 ^
compilation terminated.

[mikeboers]

That is super weird, because the config did find it.

Can you run python setup.py doctor? That should at least tell us what it is trying to build with.

[kentborg]

(venv)kentborg@stoneflute:~/python/PyAV$ python setup.py doctor
running doctor
running config
running reflect
looking for avformat_open_input... found
looking for pyav_function_should_not_exist... missing
looking for av_calloc... missing
looking for av_frame_get_best_effort_timestamp... missing
looking for avformat_alloc_output_context2... missing
looking for avformat_close_input... found
looking for AVStream.index... found
looking for PyAV.struct_should_not_exist... missing
looking for AVFrame.mb_type... found

PyAV: 0.2.4 v0.2.4-92-g1fad83a
Python: 2.7.9 (default, Mar  1 2015, 12:57:24) \n[GCC 4.9.2]
platform: Linux-3.16.0-4-amd64-x86_64-with-debian-8.3
extension_extra:
    libraries: ['avcodec', 'avdevice', 'avformat', 'avresample', 'avutil', 'swscale']
    library_dirs: []
    include_dirs: ['include', '/usr/include/python2.7']
config_macros:
    PYAV_VERSION=0.2.4
    PYAV_VERSION_STR="0.2.4"
    PYAV_COMMIT_STR="v0.2.4-92-g1fad83a"
    PYAV_HAVE_LIBAVRESAMPLE=1
    PYAV_HAVE_AVFORMAT_CLOSE_INPUT=1
    PYAV_HAVE_AVFRAME__MB_TYPE=1

[kentborg]

I think I am have libav installed at the moment (for the doctor output).

[mikeboers]

The options at this point are:

1. PyAV is building something wrong.
2. Your LibAV does not have that header.

Can you find the libav headers, and determine if libavfilter/avcodec.h is missing?

[mikeboers]

Neither FFmpeg or Libav appear to have that header. Huh. I wonder if it was deprecated (or something else)...

[kentborg]

A file by that name exists, but in a different directory:

# updatedb

Then locate finds /usr/include/libavcodec/avcodec.h

Same locate output for both ffmpeg and libav installed cases.

[kentborg]

Um, that was supposed to be a crosshash followed by " updatedb" and got all interpreted...

[mikeboers]

I've been editing your posts to wrap your code in triple back-ticks to format it.

I also realized that this specific problem is in the filters implementation, which is brand new and not fully tested across platforms, so I'm not surprised it is giving you trouble.

If you git cherry-pick 1fad83a onto the master it is much more likely to build for you. :blush:

[kentborg]

That builds. Whew. I'll let you know whether the delay property works for me, but I think it will be a few hours. Gotta pack and drive to Montreal now.

[kentborg]

Back to trying to use PyAV. I copied the results of my build from the PyAV venv to the venv where I am doing my work, and lifting from your Basic Demo, after the assignment to variable video, I print video.delay and it doesn't complain--good sign--it prints 0. And the demo runs. But no matter what I set it to it really wants to buffer--on my test data I need over a thousand frames before it starts reading any additional data while outputting frames.

I reverted to the pip install version, and it complained about my accessing the delay property (so that part was real) but it behaves the same way, not reading additional data until the exact same frame.

Am I using it wrong?

Thanks,

-kb

[mikeboers]

Are you able to provide a sample of your file? Or a way to generate one with the same behavior?

(Sent from my phone.)

On Mar 4, 2016, at 12:10 PM, kentborg notifications@github.com wrote:

Back to trying to use PyAV. I copied the results of my build from the PyAV venv to the venv where I am doing my work, and lifting from your Basic Demo, after the assignment to variable video, I print video.delay and it doesn't complain--good sign--it prints 0. And the demo runs. But no matter what I set it to it really wants to buffer--on my test data I need over a thousand frames before it starts reading any additional data while outputting frames.

I reverted to the pip install version, and it complained about my accessing the delay property (so that part was real) but it behaves the same way, not reading additional data until the exact same frame.

Am I using it wrong?

Thanks,

-kb

— Reply to this email directly or view it on GitHub.

[kentborg]

I can upload my Python file, but it won't let me upload a 1MB mpeg test file. I could e-mail it...

-kb

[kentborg]

Heck, it won't let me upload a .py file either.

[mikeboers]

You can upload your small sample to https://www.dropbox.com/request/26cFYq3DBTWM0MKVjEuH

[mikeboers]

I've isolated the reads to avformat_find_stream_info(...), which is what seems to determine time bases, frame rates, etc..

I could potentially add a skip_stream_info kwarg to av.open(...), so that it doesn't figure those out. I'm not sure what the repercussions are.

I have also identified where the buffer size is set for Python IO, and could expose it, but I don't know where that could be helpful.

[kentborg]

avformat_find_stream_info(...), which is what seems to determine time bases, frame rates, etc.

My stream has distinctly funny frame rates, I make it by feeding only the frames I want, when I want, to code based on the Raspberry Pi hello_encode example. mplayer isn't very pleased with the result. Maybe I could make my stream more conventional, but I haven't figured out how.

where the buffer size is set for Python IO

That sounds promising: I would like to make the read buffering as small as possible and have it still work, to have it spit out a frame as promptly as possible when it has enough data. I would like it to make further read calls only when it really needs more data, because maybe this frame is really new and its compressed data didn't exist until a moment ago. And a moment later data for yet another frame will be available, but not quite yet. I expect I will lie on that seek() to the and tell() calls it makes. Yeah, sure, I have lots of data--just not yet. Some of the read calls will stall until I have more data. But constant timing isn't the point, I am looking to recreate the original frames I fed in, as frames again.

Thanks,

-kb

[mikeboers]

I'm not sure adjusting the buffer size will do what you want. I think it is just how much the library will request, not how much it actually needs.

It would be interesting to see if we could signal that you wont permit seeking, and then use a custom file-like object (e.g. a wrapper around a list of strings, popping one off on every read request).

[kentborg]

This was interesting: http://ffmpeg.org/pipermail/libav-user/2014-December/007672.html

But I don't fully understand it.

-kb

[kentborg]

Remember me? I'm wondering whether you have any new insights here.

The reason I pester is I finally have an end-to-end implementation of my project. Terribly incomplete, but all the key pieces are in place, I have convinced it can all work and I know mostly how to make it all work. This is MPEG annoyance is big remaining wart...

Earlier you wrote:

I'm not sure adjusting the buffer size will do what you want. I think it is just how much the library will request, not how much it actually needs.

I am tossing compressed data that I concluded corresponds to exact input frame boundaries. I fed in frames one at a time, and collected the that came out each time. I figure that if I fed it a frame and if returned compressed data, then that same compressed data should be dang close to what is needed to reproduce something resembling the original frame. Or am I misunderstanding MPEG?

Thanks,

-kb

[kentborg]

Um, "...tossing..." as "...tossing around...", not discarding!

Sorry,

-kb

[mikeboers]

Lots here to remind myself about. I wonder if #155 would help (e.g. exposing raw codecs without formats).

Is what you uploaded to me (ages ago 😞) still roughly what is going on? You are streaming (effectively) a whole container file, and want to pull out frames with as little buffered data as possible.

I wonder if B-frames are giving you a hard time (although the sample you gave me does not seem to have any). Looking at the file you gave me in March, I have to decode 12 packets before getting a frame (with the current master). Looks like the GOP size is ~40. So those don't line up...

I really wish I could keep digging into this right now, but I have a really crazy deadline hanging over my head. You are welcome to come chat at me on Gitter (https://gitter.im/mikeboers/PyAV), but I can't do a ton myself.

[kentborg]

On 05/30/2016 06:20 PM, Mike Boers wrote:

Lots here to remind myself about. I wonder if #155 https://github.com/mikeboers/PyAV/issues/155 would help (e.g. exposing raw codecs without formats).

Let me look at that. Maybe it would help.

Is what you uploaded to me (ages ago 😞) still roughly what is going on? You are streaming (effectively) a whole container file, and want to pull out frames with as little buffered data as possible.

I am feeding a frame at a time in and grabbing the compressed data chunks I am returned. On the other side I would like to feed the compressed data a chunk at a time, and get out an uncompressed frame as quickly as possible.

I wonder if B-frames are giving you a hard time (although the sample you gave me does not seem to have any).

My encoding code was pretty closely lifted from a Raspberry Pi sample /opt/vc/src/hello_pi/hello_encode/encode.c (if that means anything to you), I haven't figured out how get it to start a new GOP, right now I am being brutal and just closing/reopening it (and doing that sooner than I was).

I really wish I could keep digging into this right now, but I have a really crazy deadline hanging over my head.

Forget about it for the while. I have plenty of other stuff to work on, like how to gracefully force a keyframe (or get the code I am using to automatically insert one for me).

Thanks for your time--now get back to more important work!

-kb

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.