File names with characters with accents (or Greek characters) are not recognised when importing playlists

vasilisvg commented 4 years ago

Whenever I update or add an m3u playlist, I get quite a few errors that say that No file in the library matches playlist entry. It seems like all these songs are songs that have a special character in the file name, like an é or Greek letters like a π. It seems like the problem only occurs when the special characters are in the name of the file. When the special characters are in the path only there is no problem. So for instance, the song Quem bô ê by Cesária Évora is not imported:

Sep 25 18:37:45 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Cesária Évora/Cabo Verde/09 Quem bô ê.m4a'

All other songs from that same record were imported without any issues. I checked, and these songs are all in the library, and Forked daapd has no problem playing them when I play them in the web interface.

I run forked-daapd 27.2 on a Raspberry Pi 3B+ with Raspbian Lite.

Here are a few more lines from the latest scan:

Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../xyce/vue/11 chaussée.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Αλίκη Καγιαλόγλου/Live In Utrecht/Κάπου Υπάρχει Η Αγάπη Μου.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Baba Zula/XX/1-13 Baba Zula - Abdülcanbaz (live @ Resistance Festival in Piraeus).m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Kostas Bezos and the White Birds/Kostas Bezos and the White Birds/13 Καρδιά απ` αγάπη ορφανη (Heart Orphaned By Love).m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Dexter Johnson/Live à l'Étoile/06 Caminho de São Tomé.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../DIEUF-DIEUL de Thiès/Aw Sa Yone Vol. 1/08 Yandé.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Royal Band de Thiès/Kadior Demb/10 Hommage Á Mbaye Fall.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Mar Seck/Vagabonde/07 Kaïra.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../DIEUF-DIEUL de Thiès/Aw Sa Yone Vol. 1/07 Hommage à Cheikhou Oumar Foutiyou.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Baba Zula/XX/2-06 Baba Zula - Hopçe (dub).m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Idrissa Diop & Cheikh Tidiane Tall/Diamonoye Tiopité/09 Massani Cissé.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Idrissa Diop & Cheikh Tidiane Tall/Diamonoye Tiopité/08 Kaële.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Bombino/Agamgam 2004/10 Ténéré (2 Guitars Version).m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Idassane Wallet Mohamed/Issawat/07 Inarègh.m4a'
Sep 25 18:37:52 t65 forked-daapd[1743]: [  LOG]     scan: No file in the library matches playlist entry '../Tidiane Thiam/Siftorde/03 Hommage a Toumani Diabaté.m4a'

ejurgensen commented 4 years ago

Yes, I recall looking at this issue before, and if it is the same then the cause is that the character encoding in the playlist and the filesystem is not the same. So when forked-daapd searches playlists for the filename it receives from the OS it doesn't find a match. I'm sorry to say I couldn't find a solution to the problem.

vasilisvg commented 4 years ago

Ah, I think I understand. And if I understand this paragraph about M3U8 correctly this could be solved if somehow support for m3u8 playlists could be added. I have no idea if this is true and how much work it would be though …

ejurgensen commented 4 years ago

Yes, maybe that would help. How are you currently producing the m3u playlists? If you have the option of changing the character encoding then you might be able to fix the issue that way.

Btw, I got it the wrong way around in my reply above: forked-daapd will read the filename from the playlist and then try to match with filename's stored when scanning the fs.

vasilisvg commented 4 years ago

I use Swinsian on my Mac to export playlists. This generates m3u8 files. I then simply remove the 8 from the extension and change the paths to match the path on the Raspberry PI.

vasilisvg commented 4 years ago

I also tried adding #EXTENC: UTF-8 to the second line of one of the m3u files, but unfortunately this doesn’s seem to help either.

vasilisvg commented 4 years ago

Ok, I think I start to see what the problem is. I copied one of the words it cannot find (Hopçe) and pasted it into the search field in the web interface. It couldn’t find it. Then I typed Hopçe and it was able to find it. Somehow Swinsian seems to do some weird encoding. The ç it generates seems to be two characters, instead of the one character it should be. I’ll reach out to the developer of Swinsian to see if they can help me out. It turns out to be an issue with my Nextcloud installation. I need to update the character encoding in the databse

ejurgensen commented 4 years ago

If you send the m3u and your /var/cache/forked-daapd/songs3.db to me at espenjurgensen@gmail.com perhaps I can identify the encodings.

ejurgensen commented 4 years ago

Taking Yandé.mp4 as an example it looks like the m3u as 3-byte encoding of the é (65 cc 81), while the fs has 2-byte (c3 a9)

00000000: 6d33 753a 5961 6e64 65cc 812e 6d70 340a  m3u:Yande...mp4.
00000010: 2064 623a 5961 6e64 c3a9 2e6d 7034 0a     db:Yand...mp4.

ejurgensen commented 4 years ago

Seems they are two different ways of encoding the é: http://www.fileformat.info/info/unicode/char/0301/index.htm http://www.fileformat.info/info/unicode/char/00e9/index.htm

vasilisvg commented 4 years ago

Thank you so much for your help. It turned out to be an issue with Nextcloud. I had to enable 4-byte support in the database to fix it.

owntone / owntone-server

File names with characters with accents (or Greek characters) are not recognised when importing playlists #1101