is it possible to mount mov/mp4 as a folder with pngs/jpgs?

[zhuker]

what would it take to implement it?

like so: # ls /storage/videos

video1.mp4
video2.mov

# ffmpegfs /storage/videos /mnt/ffmpegfs # find /mnt/ffmpegfs

/mnt/ffmpegfs/video1.mp4/00001.png
/mnt/ffmpegfs/video1.mp4/00002.png
...
/mnt/ffmpegfs/video1.mov/00001.png
/mnt/ffmpegfs/video1.mov/00002.png

[zhuker]

Predict the target image size at least approximately (currently fixed to an arbitrary 40 K)

not sure this will work with jpegs but for png if i could specify some option like --exact-sizes it would turn on ffmpeg option -compression_level 0 and the size would be 100% predictable header + width * height * bytes_per_pixel

[nschlia]

one more thing i am missing is ability to read random frames without waiting for decode of previous ones. this would be very handy for debugging.

I guess this should be possible. For video or audio files it's almost impossible but as we want separate, independent frame images it could be done. Added to the list.

also find /mnt/ffmpegfs is not feasible on our 100TB video storage would be nice to just be able to: cp /mnt/ffmpegfs/long/path/to/video/here/00000001.png ~

This is on my list, I forgot to mention.

not sure this will work with jpegs but for png if i could specify some option like --exact-sizes it would turn on ffmpeg option >-compression_level 0 and the size would be 100% predictable header + width height bytes_per_pixel

Added.

This is the complete list now:

I am afraid this is going to take me a few weeks to complete, so please stay tuned :)

• Allow direct access without prio ls (cp /mnt/ffmpegfs/long/path/to/video/here/00000001.png ~)
• Allow selecting the target image quality (using --videobitrate= or a new parameter)
• Predict the target image size at least approximately (currently fixed to an arbitrary 40 K)
• For png if it could specify some option like --exact-sizes it would turn on ffmpeg option -compression_level 0 and the size would be 100% predictable header + width height bytes_per_pixel
• Update the file sizes once known
• Change the internal cache format to support any size of images (remove the 20 M limit)
• Speed up caching
• Select the best match target pixel format (use av_find_best_pix_fmt_of_2 or so)
• Ability to read random frames without waiting for decode of previous ones

[nschlia]

when viewing on mac i copied from the virtual directory first by just using cp 000000001.png ~

And this as well. Seems your Mac does not like my image format. Have to figure out why.

[nschlia]

when viewing on mac i copied from the virtual directory first by just using cp 000000001.png ~

And this as well. Seems your Mac does not like my image format. Have to figure out why.

I tried that on my (oldie Snow Leopard OS) and the images were OK. Did you do the cp twice? Maybe the first time it copied 40 KB only. Still have to fix the size...

Maybe you want to wait a few days, I just fixed the pixel format conversion. Now a best match format is chosen, for PNG it selects RGB48BE now when I convert from ProRes, for example. I did not publish the patch yet.

[zhuker]

re images on mac - the very same bit by bit png file opens ok in Gimp but not ok in Preview i assume it's the declared and actual pixel formats

[nschlia]

re images on mac - the very same bit by bit png file opens ok in Gimp but not ok in Preview i assume it's the declared and actual pixel formats

Maybe the next update fixes it. I also added the direct access thing so you don't need to ls first. Unfortunately I'll be in Chicago for the next one or two weeks and I am not sure if I am able to work on this.

You can send me an email to nschlia@oblivion-software.de if you want to so I can keep you up to date without bloating this issue.

[nschlia]

Here's the update with the following new functionality for video frame images:

• Random access to any image
• Improved cache supporting images of any size

The random access strategy works as follows:

When a virtual directory with frame images is opened, ffmpegfs starts decoding from the beginning of the file. If any image of the list is opened, it skips to 25 frames before it and starts decoding, (almost) immediately providing the requested image. If the defaults are not changed (which is recommended), decoding is suspended after 30 seconds and terminated after 60 seconds of inactivity. Once there is another frame image accessed, decoding continues or restarts those 25 frames before that very image.

This should save a lot of disk space as only the images requested get decoded plus an overhead of maybe 100 to 150 frames. That means that normally the whole file will never be recoded, only a fraction of it unless everything is accessed. We might be talking of hundreds of megabytes, mayb even giga- or terabytes that would fill up.

I decided to start 25 frames earlier as it only takes a fraction of a second and maybe the user wants to go back a few frames. If not, we are talking about a few hundred milliseconds wasted, so who should care.

Notes:

• I refrained from creating uncompressed PNGs. The format is lossless, encoding time seems to be negligible, so better save valuable disk space.
• The first image is repeated once, so if frame no. 200 is accessed it is actually 199 but I guess this does not really matter.

To create test files, run these commands (see test/mkvid):

ffmpeg -y -f lavfi -i color=c=blue:s=1920x1280:d=60:r=30000/1001 -g 1 -c:v prores -vf \
 "drawtext=fontsize=800: \
  fontcolor=white:x=(w-text_w)/2:y=(h-text_h)/2: \
  text='%{frame_num}': start_number=1" \
 frame_test_ntsc.mov

ffmpeg -y -f lavfi -i color=c=blue:s=1920x1280:d=60:r=25 -g 1 -c:v prores -vf \
 "drawtext=fontsize=800: \
  fontcolor=white:x=(w-text_w)/2:y=(h-text_h)/2: \
  text='%{frame_num}': start_number=1" \
 frame_test_pal.mov

This will create 60 second movie at 1920x1280 resolution, where each frame contains its number. The NTSC version at ~29.997 frames per second ist a good test to check if the frame to pts calculations and vice versa work on non-integer frame rates.

[nschlia]

There are still some things to do:

• Speed is still an issue. For some reason decoding is quite slow, I'll have to figure out why.
• Need to do some testing, all works fine under Debian KDE and Ubuntu Gnome. Tests with Windows and Apple OS must still be made.
• There's much room for optimisations.

BTW, I like that random access thing. I've been missing that for video and audio as well. I'll extend that feature soon. Using the enhanced caching code this should be feasible.

[nschlia]

• Enhanced caching strategy: Already decoded frames will not be recoded, saving a lot of time. Frames that already exist in cache are simply skipped until EOF or missing frames are found.
• The prescan cache was reduced to 3 frames as with the new strategy access is very fast.
• Enabled profile guided optimisation. This is a big trade of: Compile time to runtime. Building takes much longer now, but the resulting code should be optimised to the machine its built on.

[nschlia]

Currently all that's left is the size prediction and that the first frame is duplicated, so each image frame is one behind actually. But this should both be only minor flaws.

[nschlia]

Found the reason for the duplicated first frame: Happens when deinterlacing is on. If it's turned off, the frames are correct. So it's not a bug, it's a feature.

[nschlia]

Found two problems:

• Sometimes the real number of frames falls a bit shorter than the predicted value. If e.g. the source file contains e,g. 30 frames, but 32 were predicted, the client sometimes waits forever if it accesses frame 31 or 32. I'll have to find a way to correct this number once known.
• The cache works well on most source files but I found at least one where it goes berserk. Although it seeks to the right position (if already decoded frames), it only reports errors and only returns some of the results.

I'll have to investigate that and find a fix.

[zhuker]

locks up at latest commit 4328b66bc53c3c479ae373c4ac28ad83354258fa

1. mount a folder with https://kote.videogorillas.com/vmir/uk/demo/R3A-shot54-sbs.mp4 ffmpegfs -d --cachepath=/home/zhukov/cacheffmpegfs --expiry_time=1d --desttype=png /storage/vmir/uk/demo/ /home/zhukov/mnt/
2. ls mnt/R3A-shot54-sbs.mp4 - ok 💚
3. cp mnt/R3A-shot54-sbs.mp4/00000000067.png ~ locks up 🛑

open flags: 0x8000 /R3A-shot54-sbs.mp4/00000000067.png
2019-05-06 05:31:23 DEBUG  : [/storage/vmir/uk/demo/R3A-shot54-sbs.mp4] Starting decoder thread.
2019-05-06 05:31:23 INFO   : [/storage/vmir/uk/demo/R3A-shot54-sbs.mp4] Transcoding to png. Predicted size unlimited (0 bytes).
2019-05-06 05:31:23 INFO   : [/storage/vmir/uk/demo/R3A-shot54-sbs.mp4] Opening input file.
2019-05-06 05:31:23 DEBUG  : [/storage/vmir/uk/demo/R3A-shot54-sbs.mp4] Opened input codec for stream #0: H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
2019-05-06 05:31:23 INFO   : [/storage/vmir/uk/demo/R3A-shot54-sbs.mp4] Video in: h264@37.02 Mbps [00:02.760]
2019-05-06 05:31:23 WARNING: [/storage/vmir/uk/demo/R3A-shot54-sbs.mp4] Unsupported video codec 'png' for format png.
2019-05-06 05:31:23 INFO   : [/home/zhukov/mnt/R3A-shot54-sbs.png] Opening output file.
2019-05-06 05:31:23 DEBUG  : [/home/zhukov/mnt/R3A-shot54-sbs.png] Opening format type 'png'.
2019-05-06 05:31:23 DEBUG  : [/home/zhukov/mnt/R3A-shot54-sbs.png] Output pixel format: rgb24
2019-05-06 05:31:23 DEBUG  : [/home/zhukov/mnt/R3A-shot54-sbs.png] Pre-buffering up to 102400 bytes.
[h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 DEBUG  : [/home/zhukov/mnt/R3A-shot54-sbs.png] Pre-buffer limit reached.
2019-05-06 05:31:23 DEBUG  : [/storage/vmir/uk/demo/R3A-shot54-sbs.mp4] Decoder thread is running.
   open[140396172708752] flags: 0x8000 /R3A-shot54-sbs.mp4/00000000067.png
   unique: 91, success, outsize: 32
unique: 92, opcode: READ (15), nodeid: 75, insize: 80, pid: 26316
read[140396172708752] 131072 bytes from 0 flags: 0x8000
[h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] illegal short term buffer state detected
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 22
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 22
[h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] illegal short term buffer state detected
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 22
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 22
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 24
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 24
[h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] illegal short term buffer state detected
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 24
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 24
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 26
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 26
[h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] illegal short term buffer state detected
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 26
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 26
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 28
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 28
[h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] illegal short term buffer state detected
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 28
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 28
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 30
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 30
[h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] illegal short term buffer state detected
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 30
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 30
[h264 @ 0x7fb08c003100] reference picture missing during reorder
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] reference picture missing during reorder
[h264 @ 0x7fb08c003100] Missing reference picture, default is 32
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] Missing reference picture, default is 32
[h264 @ 0x7fb08c003100] illegal short term buffer state detected
2019-05-06 05:31:23 ERROR  : [h264 @ 0x7fb08c003100] illegal short term buffer state detected

NOTE: prores transcode of the same file works as expected 👍 https://kote.videogorillas.com/vmir/uk/demo/R3A-shot54-sbs.mov

[nschlia]

I've seen that, too. Seems to be caused if you try to access a non-i-frame directly. I'll check how to fix that one. I tested with Prores i-frame only files and never had that problem, only with H264 mp4.

[zhuker]

it would be good to return some kind of error to cp if frame can not be decoded for any reason (e.g. ENOTSUP?)

[nschlia]

it would be good to return some kind of error to cp if frame can not be decoded for any reason (e.g. ENOTSUP?)

I am doing that actually, but it seems the error code gets lost inside ffmpegfs on the way. The messages come from FFmpeg, either from the decoder or the encoder. Seems it gets stuck on something. I'll figure that out (probably the decoder needs the i-frame to properly decode the p/b-frame.).

[nschlia]

It's like I have expected, for videos with p/b-frames I have to go back to the last i-frame and start decoding from that key frame. In your video https://kote.videogorillas.com/vmir/uk/demo/R3A-shot54-sbs.mp4, number 1, 27 and 54 are key frames. So if you want to open e.g. frame 37, the decoder needs to rewind to frame 27 and start decoding from there.

That's all. I'll implement that.

[nschlia]

• Fixed a bug that caused crashes due to GPFs/Bus Errors
• Fixed hangs because frames were recoded over an over See https://github.com/nschlia/ffmpegfs/commit/85b9572822784e678119ff24e2da1d1e46adff12

[nschlia]

Seek requests now go to the correct decoder thread.

[zhuker]

IT WORKS!!! 🥇 magic thank you so much

if only i could copy files without listing directory first it would be the answer to our prayers

[nschlia]

:~/test/test$ cp -v /home/norbert/test/out/orig.mp4.mp4/00000000008.png .
'/home/norbert/test/out/orig.mp4.mp4/00000000008.png' -> './00000000008.png'
:~/test/test$ cp -v /home/norbert/test/out/orig.mp4.mp4/00000000018.png .
'/home/norbert/test/out/orig.mp4.mp4/00000000018.png' -> './00000000018.png'
:~/test/test$ cp -v /home/norbert/test/out/orig.mp4.mp4/00000000058.png .
'/home/norbert/test/out/orig.mp4.mp4/00000000058.png' -> './00000000058.png'
:~/test/test$ ll /home/norbert/test/out/orig.mp4.mp4/00000000008.png    
-rw-rw-rw- 1 norbert norbert 2715307 Mai  5 12:18 /home/norbert/test/out/orig.mp4.mp4/00000000008.png
:~/test/test$ ll /home/norbert/test/out/orig.mp4.mp4/00000000028.png 
-rw-rw-rw- 1 norbert norbert 40960 Mai  5 12:18 /home/norbert/test/out/orig.mp4.mp4/00000000028.png

Workz :)

[zhuker]

nice!

[nschlia]

Great. Think we can close this issue, has grown large enough... Any bugs or additional requests should be handled by new issues.