asticode / go-astisub

Manipulate subtitles in GO (.srt, .ssa/.ass, .stl, .ttml, .vtt (webvtt), teletext, etc.)
MIT License
594 stars 111 forks source link

Implement custom `bufio.SplitFunc` for WebVTT #118

Open nakkamarra opened 1 month ago

nakkamarra commented 1 month ago

From an excerpt of the WebVTT spec regarding the WebVTT parsing algorithm:

Let input be the string being parsed, after conversion to Unicode, and with the following transformations applied: Replace all U+0000 NULL characters by U+FFFD REPLACEMENT CHARACTERs.

Replace each U+000D CARRIAGE RETURN U+000A LINE FEED (CRLF) character pair by a single U+000A LINE FEED (LF) character.

Replace all remaining U+000D CARRIAGE RETURN characters by U+000A LINE FEED (LF) characters.

I don't believe the current astisub WebVTT parsing logic is covering this part of the WebVTT parsing algorithm correctly. The bufio.Scanner is using the default which is ScanLines and I'm not sure this is handling the WebVTT parsing correctly (i.e a file with exclusively \r terminations will not be replaced and thus not parsed properly). So I'm thinking maybe implementing a new bufio.SplitFunc and feeding it to the scanner.Split(...)

This somewhat ties into another issue I am currently trying to make a PR to solve #110, where I'd like to:

  1. read the first 6 bytes and make sure they are WEBVTT
  2. read the remainder of the content up until the first occurrence of two successive line terminations
  3. write that content into a new field of the Subtitles struct that has each line

Thoughts? Maybe there's a better way for me to achieve what I'm trying to do.

asticode commented 1 month ago

From what I understand, there are 2 problems you're trying to fix:

1) handle \r 2) handle optional data after WEBVTT keyword

Regarding 1), implementing a custom bufio.SplitFunc capable of parsing \n\r seems like a good idea Regarding 2), I would update this logic, trimming BOMBytes prefix, checking the header is there, scanning until 2 empty lines have been found and storing its value in a Subtitles.Comments attribute