[FEATURE] Subtitle Support

[JRGonz]

I do not see any documentation about subtitle tracks. My guess is that these are not supported?

In my use case I am not doing anything special and using it in order to keep any video files I have consistent. So I am just using a shell and cp to copy them over to a final directory. Conversion to a common subtitle format would be awesome but I am sure that is far more complicated than copying the subtitle tracks over.

[nschlia]

Currently FFmpegfs simply discards closed captions. I am not sure if subtitle conversion is possible, need to get into the matter to find out. I'll keep it on the list of feature requests and maybe add it in the future. Honestly what I know about subtitle format is next to nothing. Is there a preferable/common format?

[JRGonz]

Conversion would be difficult I'm sure but in my case (and others) copying would suffice. I believe the ffmpeg option would be -c s:copy. I know an mp4 container would not support this but mkv support may. With regard to using an MP4 container burning them in would be an option but I think few people would like to do that.

And example:

Sorry if I am not articulating this well but something like --container=MKV --videocodec=H265 --subtitles-copy. Someone in the irc channel said that the H265 is an upcoming feature. I think I should have asked for an mkv feature first. :)

[nschlia]

Simply copying would be relatively easy, as long as the target supports subtitles (at all and that particular format). MP4 for example allows subtitles, as far as I know. TS definetely.

Sorry if I am not articulating this well but something like --container=MKV --videocodec=H265 --subtitles-copy. Someone in the irc channel said that the H265 is an upcoming feature. I think I should have asked for an mkv feature first. :)

That would be --desttype=MKV --videocodec=H265 --subtitles-copy when subtitles had been added. H265 is already supported. I tried to add MKV a while ago but did not succeed, because the format is not suitable for streaming. Probably I could try again, maybe there is an extension to MKV available now that makes it possible. But using format extensions is usually quite a hassle, as you can see from MP4 - there are quite a few derivates to choose from, depending on the playback software. Not really intuitive and useful.

Couldn't you use TS? The format is well suited for streaming (not a surprise as TS stands for transport stream) and supports a wide range of subtitle codecs.

[JRGonz]

I did not know there was a --desttype=MKV. This is in dev right and hasn't been pushed out as an apt pkg yet (ubuntu)?

In my case I am not streaming. I am just using cp to take .avi, .mp4, .mkv, etc files from the mounted ffmpegfs dir and re-encode them into h265 mkv while copying the remaineder of the tracks. So mounted I would just be taking files dumped into a dir and moving them to permanent storage but re-encoded as a quick and easy way to avoid any further steps and creating files that are all the same. Doing this would be a huge reduction in space while hardly breaking quality and a huge increase in efficiency since ffmpegfs would do all the work.

I know this probably isn't the use case that ffmpegfs had in mind but it would be nice to just add it to ffmpegfs in fstab and forget about it.

[JRGonz]

Sorry. I did not answer your question. The goal is to universally use mkv as the container.

[nschlia]

I did not know there was a --desttype=MKV. This is in dev right and hasn't been pushed out as an apt pkg yet (ubuntu)?

Not yet. MKV support is not implemented.

In my case I am not streaming. I am just using cp to take .avi, .mp4, .mkv, etc files from the mounted ffmpegfs dir and re-encode them into h265 mkv while copying the remaineder of the tracks.

I should have mentioned that whenever you copy files through FFmpegfs, this actually is streaming. The point is that the file is encoded while being read. More of it becomes available while it is already being played back (or copied, makes no difference).

Some formats, standard MP4 or MKV, require the file to be finalised, i.e., completely recoded, to be playable. This means that parts of it get updated in the end. So if you copy the file before this has taken place, you will miss out some changes. Easier to explain for WAV: The format has a header which contains the file size. I trick that by filling in the header before the file is recoded. Works for WAV as it is easily possible to calculate these fields. Unfortunately not for MP4 or MKV...

I know this probably isn't the use case that ffmpegfs had in mind but it would be nice to just add it to ffmpegfs in fstab and forget about it.

This is exactly what I do. But you have to stick to "streamable" formats, best choice would be TS (H265 is currently slow, H264 is much faster, and you could use HW support. I recode to H264 at 11-13 times speed with VAAPI. The software encoder comes to 3-4 times only. H265 is under 2x mostly...). WebM also works well, but the encoder for VP8/9 is slow. Even with hardware support not as fast as H264,

Anyways, you could create a feature request for MKV and I'll see if it is possible,

[nschlia]

Several containers support subtitles, see https://en.wikipedia.org/wiki/Comparison_of_video_container_formats#Subtitle_formats_support

[nschlia]

case of external subtitles: can ffmpeg detect external subtitles and faux the mkv as one file as well? (Meaning ffmpegts will treat an external .idx/.sub.ass/etc file as an mkv track when viewed by the user in the mounted directory? (from

121)

Yes, that should be possible.

[nschlia]

Subtitle support has been implemented, but some points are missing or not working as expected:

Although exact timestamps from the input streams are written to the output, the title timing is different.

For testing I have used test5.mkv (taken from this test suite: https://github.com/ietf-wg-cellar/matroska-test-files). It contains a subtitle stream with captions at these PTS:

pts=42
pts=3625
pts=6458
pts=7375
pts=41958
pts=43833

This is at 0,042, 3.625, 6,458 seconds, and so on. These PTS are written plain vanilla to the output, yet the resulting PTS (when read with ffprobe) are different:

pts=1054
pts=4637
pts=7470
pts=8387
pts=42970
pts=44845

This is very strange. I'll have to debug the FFmpeg API code to figure that out.

Support for external subtitle files

FFmpegfs should be able to read external subtitle files, e.g. SRT. This needs to be implemented. Need to fix the above problem first, though.

Conversion from text to bitmap

This would be nice to have:

Currently captions are only transferred bitmap to bitmap or text to text. MKV supports both, no problem. TS and MP4 support text only. At least conversion text to bitmap should be possible, though.

[nschlia]

Although exact timestamps from the input streams are written to the output, the title timing is different.

This is very strange. I'll have to debug the FFmpeg API code to figure that out.

The strangeness has been resolved: av_write_frame shifts the time stamps to avoid negative values. The riddle is why negative PTS occur in audio/video anyway. But for now the values are OK relative to the PTS in audio and video.

[nschlia]

Title timing problem has been resolved with 4cfafe92a70605fd1cbb07d2b9d837899029a2be, the frames are properly sorted before feeding them into av_write_frame.

BTW, before anyone mentions, av_interlaced_write_frame would solve this, but delay disk writes. This is not an option as it can block playback until the whole file is available (in memory). Not good :)

Sorting frames is also a prerequisite to support external subtitle files (like .srt), because they need to be synchronised.

[nschlia]

Found a few problems and fixed them:

• For Subrip and WebVTT, a header must be added if missing
• Corrected checks if target format supports compatible subtitles (both must be bitmapped or text based)

Done some refactoring required for the next steps:

• Add separate SRT file support
• Seems that some sources start with 1 or even 0 subtitle tracks that appear much later. These streams be added to the output on-the-fly.

[nschlia]

Seems that some sources start with 1 or even 0 subtitle tracks that appear much later. These streams be added to the output on-the-fly.

Fixed with 52766cf

[nschlia]

Totally revamped frame buffer, there is a single buffer now that holds audio/video frames and subtitles in a single spot. Gone C++17 for that, using std::variant and std::shared_ptr objects (they were available since C++11 or so, but there were some details missing that were required like std::map:extract and such).

This greatly improves audio/video synchronicity and will allow external subtitle files to be incorporated.

Done a great deal of refactoring as well to clean up the code and maintain an overview over it.

Next thing will be reading external subtitle files and inserting them into the output file.

[nschlia]

Finally I have completed the external subtitle file functionality. External SRT or VTT files are read and incorporated as streams into the output file. This seemed to be a simple task, but turned out to be much bigger. I had to revamp the whole packet queue system using C++17 to do that. Formerly FFmpegfs had three queues, one for audio, video and subtitles respectively. These had to be merged into a single queue, sorted by time stamp. This way only I could insert all subtitle frames at the beginning and read them out sequentially during transcoding. Changes have been done and are being tested now. Will be published once stable.