Netflix Timing Fixer - Githubissues

matmaggi commented 3 years ago

Hello, I think this is a mission for @OmrSi or anyone interested.

My idea is a "Netflix Timing Fixer" following the newest Netflix Timing Guidelines (that can be seen here)

How would it work?

We would time the whole subtitle file to audio (in and out) regardless of the shot changes/scene changes. Then we would apply this "Netflix Timing Fixer" and it will follow this rules:

*SC = shot change/scene change in this case.

IN-CUES:

Always have to be 12 frames before the SC or 12 frames after the SC or on the SC, so:

When 1 to 7 frames before SC, move to the SC [-7 to -1 ➝ 0]
When 8 to 11 frames before SC, move to 12 frames before SC [-11 to -8 ➝ -12]
When 1 to 9 frames after SC, move to the SC [1 to 9 ➝ 0]
When 10 to 11 frames after SC, move to 12 frames after SC [10 to 11 ➝ 12]

OUT-CUES:

Always have to be 12 frames before SC or 12 frames after SC or 2 frames before SC (never on the SC), so:

When 1 frame before SC, move to 2 frames before SC [-1 ➝ -2]
When 3 to 7 frames before SC, move to 2 frames before SC [-7 to -3 ➝ -2]
When 8 to 11 frames before SC, move to 12 frames before SC [-11 to -8 ➝ -12]
When 1 to 5 frames after SC, move to 2 frames before SC [1 to 5 ➝ -2]
When 6 to 11 frames after SC, move to 12 frames after SC [6 to 11 ➝ 12]

Condition:

Respecting 2 or 12 frame gap between subtitles (3-11 frames gap are not allowed)

How can it be done?

I actually know nothing about programming, but using or incorporating some parts of SubTimingsBeautifier (by @Flitskikker) into SE could be really helpful, since it already does some part of the job. Having to readequate to above rules. Also, the new Netflix Quality Checker in the SE has some great info too #4470.

Why do this?

Because Netflix has become somewhat the head of the subtitling industry and several clients always say "follow Netflix guidelines", so it could be very helpful for all that uses these timing guidelines.

I tried to make this easy to explain, so I hope it's clear enough. Thank you all for all your effort throughout the years.

OmrSi commented 3 years ago

Hey. :) This https://github.com/SubtitleEdit/subtitleedit/pull/4470 is applied in 3.5.18 BETA, not the original 3.5.18.

As you can see here, they should be working well:

Here is the link of the beta: https://github.com/SubtitleEdit/subtitleedit/releases/download/3.5.18/SubtitleEditBeta.zip

m0ck69 commented 3 years ago

Good idea @matmaggi @OmrSi SE only shows a raport, will there be an option to fix all issues automatically?

matmaggi commented 3 years ago

Thank you @OmrSi. So now Netflix Quality Checker is working great!

About my suggestion, I was thinking about a button to correct the timings automatically the way I described instead of manual adjusting them. And we would use the checker to see if everything is alright at the end. The two together could be very powerful!

OmrSi commented 3 years ago

@m0ck69 @matmaggi It's on the to-do list. The issue is there are too many factors, especially for the timing. Most of the time, the user has to choose what to do.

For example, If the reading speed is above the threshold and you choose to auto-fix that, it would extend the out-cue, but what if the user had already extended the out-cue to 12 frames after the voice ended?

Another example, what if the shot change was a fake shot change? it would adjust the timing based on it, but it shouldn't. On the other hand, there is no way for it to know it's a fake shot change.

Yet another example is your image: https://user-images.githubusercontent.com/48259053/104123325-63ea5c00-5329-11eb-982f-08b9e68d3b2f.png

What should it do in this case? How would it know whether to extend the out-cue to 2 frames before the shot or 13 frames before it or 12 frames after it? What about the in-cue, should it be snapped to the shot change or set to 12 frames after it?

And other examples of endless edge cases. All of these factors make it a bit challenging to auto-fix.

By the way, check out these shortcuts if you haven't:

matmaggi commented 3 years ago

Well, as stated, I was thinking about using it when we time subtitles exactly at the in-time and out-time, so it wouldn't be expected to have already 12 frames past it.

In my image, I would do what I have described in the following NF guideline.

In-cue: When 1 to 7 frames before SC, move to the SC [-7 to -1 ➝ 0] Out-cue: When 8 to 11 frames before SC, move to 12 frames before SC [-11 to -8 ➝ -12]

I think this tool is supposed to be used right after timing the whole file (or template), then translating in a file "fixed". So you'll adjust some lines when needed and then run the Netflix Quality Checker at last, fixing manually a few (hopefully) errors that could have passed.

That's why I think this auto-correct timing would be really helpful.

Flitskikker commented 3 years ago

@matmaggi Good idea! I was thinking about integrating SubTimingsBeautifier's logic into SE some day, so maybe it's finally time to start working on that.

I already worked on an experimental version of SubTimingsBeautifier with a setting to leave X frames between the out cue and shot cut, but the other rules ("green zone", "red zone") were a PITA to implement.

When creating a feature like this, I feel it should be configurable. Netflix might be a big player in the subtitle industry, their guidelines are definitely NOT the best. Several of their rules clash with sensible subtitling guidelines.

For example, 12 frames is 0,5 second. Moving an in-cue 11 / 12 frames forward because coincidentally there is a shot cut will cause the subtitle to be shown 0,5 second after the speaker is audible. Same goes in reverse for the out-cues. This comes down to these basic guidelines:

A subtitle should NEVER appear AFTER dialogue has started. A subtitle should NEVER disappear BEFORE dialogue has ended.

Also, subtitles should stay on-screen for about 500 to 1000 ms after the dialogue has ended, to give the viewer the chance to actually read the subtitle.

Furthermore, I never understood the "2 frames before shot cut" rule. This leads to unnecessary "flashing" on-screen for viewers. It's better to hide the subtitle exactly on the shot cut, so there's only 1 "flash". SubTimingsBeautifier was originally designed this way.

TL;DR: A feature like this could be really useful, but it should not use Netflix's bad rules hardcoded.

Flitskikker commented 3 years ago

Some technical challenges:

SubtitleEdit and SubRip use milliseconds under the hood. 25 FPS will give nicely rounded timecodes, but a lot of content is 23,976 / 24 FPS, which will lead to rounding issues.

In practice, playing an SRT using a media player may result in some subtitles (dis)appearing too late or too early, which is bad when you're working with shot cuts, of course.

SubTimingsBeautifier solved this by extracting the exact timecodes from the video file using ffmpeg. However, extracting the timecodes takes a while, so it's pretty tedious when working with many files in a row.

Using mpv in combination with the shortcuts to move frame by frame seems to give accurate timecodes, so I hope it's possible somehow to tap into that.

OmrSi commented 3 years ago

@matmaggi Right now, I just use it when I'm done with the file to check for any issues left. I'm not saying it's impossible, just challenging, that's why it's on the to-do list. :D

@Flitskikker The green zone and red zone are gone now, by the way. Also, SE has SubtitleFormat.MillisecondsToFrames to work with frames, if that helps.

matmaggi commented 3 years ago

@Flitskikker Thank you. Yes, being configurable it woulkd be very helpful indeed. I just said Netflix guidelines because all my clients follow their rules, so for me it would be a great improvement. But if we could configure Netflix or other clients guidelines, it would be awesome!

A subtitle should NEVER appear AFTER dialogue has started. A subtitle should NEVER disappear BEFORE dialogue has ended.

About this statement, I totally agree, but that's not what my clients tell me to do, so I do what they tell me. haha

SubTimingsBeautifier solved this by extracting the exact timecodes from the video file using ffmpeg. However, extracting the timecodes takes a while, so it's pretty tedious when working with many files in a row.

I really liked to extract the exact timecodes with SubTimingsBeautifier using ffmpeg. I don't think this will be a problem if we have a good quality tool.

@OmrSi Yes, I understand. Thank you anyway.

Flitskikker commented 3 years ago

I worked out a prototype UI for this feature tonight.

The following panel could be added somewhere in the settings:

(Treat*)

While Netflix doesn't use the red/green zone anymore, it helps visualizing setting the threshold (i.e. move the cue to 12 frames from the shot cut, or snap the cue to the shot cut). For the previews, I used Netflix's design from their style guide. The previews will show the settings and maybe allow interaction.

Everything should be configurable like this, either for Netflix or to replicate SubTimingsBeautifier's logic.

--

These settings will then be used by:

A new option Tools > Beautify timings... (or something like that)
Shot cut snapping in the waveform (e.g. out-cues will be snapped 2 frames before a shot cut)
Perhaps also add it to Fix Common Errors?
Perhaps syntax/error coloring in the waveform?

--

Finally, SubTimingsBeautifier's feature "Extend out-cues where/until possible by ...", which takes shot cuts into account, can be added to Tools > Adjust durations... as a checkbox.

Any thoughts? Did I miss anything?

matmaggi commented 3 years ago

@Flitskikker Wow! That's fantastic! I think that's perfect. It seems that everything is there. With some testing we can say for sure.

nkh0472 commented 3 years ago

I worked out a prototype UI for this feature tonight.

The following panel could be added somewhere in the settings:

(Treat*)

Looks cool!

chenlung commented 3 years ago

Thank you for all your work, @Flitskikker.

I'm not sure I 100% agree with one of your posts, however.

A subtitle should NEVER disappear BEFORE dialogue has ended.

What about shot changes?

Also, subtitles should stay on-screen for about 500 to 1000 ms after the dialogue has ended

Maybe in some scenarios, but this might be difficult for dialogue in quick succession or when the dialogue continues very briefly into a shot change.

Furthermore, I never understood the "2 frames before shot cut" rule. This leads to unnecessary "flashing" on-screen for viewers. It's better to hide the subtitle exactly on the shot cut, so there's only 1 "flash". SubTimingsBeautifier was originally designed this way.

One Blu-ray authoring software requires a minimum of four frames, so I would need this.

TL;DR: A feature like this could be really useful, but it should not use Netflix's bad rules hardcoded.

Agreed.

matmaggi commented 3 years ago

I worked out a prototype UI for this feature tonight.

The following panel could be added somewhere in the settings:

(Treat*)

While Netflix doesn't use the red/green zone anymore, it helps visualizing setting the threshold (i.e. move the cue to 12 frames from the shot cut, or snap the cue to the shot cut). For the previews, I used Netflix's design from their style guide. The previews will show the settings and maybe allow interaction.

Everything should be configurable like this, either for Netflix or to replicate SubTimingsBeautifier's logic.

--

These settings will then be used by:

A new option Tools > Beautify timings... (or something like that)

Shot cut snapping in the waveform (e.g. out-cues will be snapped 2 frames before a shot cut)

Perhaps also add it to Fix Common Errors?

Perhaps syntax/error coloring in the waveform?

--

Finally, SubTimingsBeautifier's feature "Extend out-cues where/until possible by ...", which takes shot cuts into account, can be added to Tools > Adjust durations... as a checkbox.

Any thoughts? Did I miss anything?

Hi @Flitskikker, sorry to disturb you, but do you have any news about your amazing panel? Thanks.

Flitskikker commented 3 years ago

@matmaggi Apologies for the very late reply. I've been working on this on my own repo: https://github.com/Flitskikker/subtitleedit/tree/feature/timecodes-fixer

Unfortunately I've had very little spare time in the last few months because of work... 😪 I'd really need a week of holiday to fully focus on this, I think. Netflix keeping changing their guidelines also isn't much of help either. 😅

If I recall correctly, I was almost done with the UI, so that leaves the actual logic and further integration, such as #5134, based on the selected settings.

It's still on my radar!

matmaggi commented 1 year ago

Hello, just checking. Any news on this? Thank you!

Flitskikker commented 1 year ago

@matmaggi No big news, unfortunately. I have just been swamped with my two jobs and also some personal stuff. I'm trying to work on it locally whenever I have some spare time, but it's more complex than I imagined.

I went ahead and actually blocked some space in my schedule next week to dive in it again. Sorry to keep you waiting so long...

Flitskikker commented 1 year ago

@matmaggi Alright, I have the main logic mostly working now, and I'm working towards an MVP.

You can go to Tools > Beautify time codes... or use the icon on the toolbar (old or new style 😉):

Then you can choose to extract the exact timings using ffprobe (useful for 23.976 FPS content due to rounding -- not really needed for 25 FPS) and enable shot change snapping:

You can configure the profile as shown before:

Netflix' rules (not my personal preference) are shown in the example. There are also presets for e.g. SDI and the "default" preset from good ol' SubTimingsBeautifier.

Netflix already moved away from red and green zones in the meantime, but I find it useful to visualize what is actually happening (move towards or away from shot changes).

Still a bit overwhelming of course, so there is also a "simple mode" helper wizard thingy:

Again NF rules as an example, loaded by the handy button (I presume most subtitlers are forced to work with similar rules these days).

Beautifying itself seems to work, but I can't really screenshot that of course.

Snapping in the wave form is now also done according to this profile, so out cues would snap 2 frames before shot changes with this profile.

Before I make a PR for it, I will do some testing with several files and also compare it with STB's output. The logic was a real braintwister to write with all those zones and conditions, so there are probably some edge cases that I need to tweak.

There are still some things that could be improved, such as adding it to the batch converter and command line, an option to beautify only selected lines... But let's focus on version 1.0 first, I think.

I also struggle with writing unit tests for this. There are some simple functions that I can write tests for (e.g. IsOutCueOnShotChange), but a lot of the logic is more "visual" with actual subtitle files and lists of shot changes. I'm afraid it would take ages to create these programmatically in a black box. Any ideas, @niksedk ? All i18n and tab indices are ready.

Also, @OmrSi, some shortcuts depending on shot changes might need to be updated to also use the new rules.

Again my apologies that it took so long... Hopefully a release is now finally nearby.

matmaggi commented 1 year ago

OMG, @Flitskikker this is beyond amazing and super helpful for lots of people, I think. Looking forward to seeing it!

tharsim commented 1 year ago

This is looking amazing. Thank you, @Flitskikker

Flitskikker commented 1 year ago

Thank you!

Two other changes I forgot to mention:

"Adjust durations" and "Apply duration limits" are now also shot change-aware.

SubtitleEdit / subtitleedit

Netflix Timing Fixer #4697

How would it work?

IN-CUES:

OUT-CUES:

Condition:

How can it be done?

Why do this?