Add full-width space to delimiters for tokenization

erengy / anitomy

Anime video filename parser

Mozilla Public License 2.0

278 stars 22 forks source link

Add full-width space to delimiters for tokenization #17

Closed rzumer closed 10 months ago

rzumer commented 2 years ago

Just a suggestion since I have seen some videos where the title and episode number are separated by a full-width space (　). It looks like currently only half-width spaces and underscores are included. I may write a PR later if I have the time.

erengy commented 10 months ago

Added U+3000 (ideographic space) as well as U+00A0 (no-break space) and U+200B (zero width space) in 5e7cb4af49a765fdde54d8f1762d33722a632c0d.

https://github.com/erengy/anitomy/blob/3c17e851d89c4605340a9a42bd8a1e1e0e3de1d0/include/anitomy/detail/delimiter.hpp#L5-L16