Unicode DeviceSQL strings in PDB are UTF-16-BE, not UTF-16-LE

[brunchboy]

I heard from another person who is implementing parsing of PDB files, and he was working with some Russian text, and discovered we were wrong to think these strings are UTF-16LE. Here is what he said, and I validated this by creating a playlist containing the same string in its name:

I could have something off, but here is what I'm seeing about the strings. My PDB includes a Russian song called Покинула чат ("left the chat"). The first letter here is U+041F. All the Cyrillic letters start with 04, but the spacebar between the words is the same U+0020 as in English. Here's how the track name looks in the pdb in hex:

If I skip the 0 and read little endian, I get back the desired "Покинула чат"

If I don't skip and read big endian, I get back the incorrect " окинулаРGат" It gets a lot of the letters right because there is usually a 04 every other byte, but the first letter (which turns out as U+001F "Information separator one") and the characters around the space get messed up (because of the momentary switch from leading 04 to leading 00). English titles come out right either way, because the leading 00s for each ASCII character in UTF-16 make it forgiving.

[brunchboy]

And this may not be worth dealing with, but the same person who discovered the above also noticed that the first string pointer (which you call str_u1) in track rows is not always empty. When present, it holds the ISRC (International Standard Recording Code) that uniquely identifies the track.

https://github.com/flesniak/python-prodj-link/blob/2709ac77170ff13f9441ea65020e4553475c27c0/prodj/pdblib/track.py#L40

string 0 in the Track row string array is the ISRC (International Standard Recording Code). It showed up for me for some new music files I imported after buying on iTunes. the string encoding here is a bit different, it starts as a 0x90 string with its length uint16, but is followed by 0x00 (maybe Mac-only again) and 0x03 and then is actually ASCII encoded. see screenshot for USUM72004304:

I don’t think I’m going to try to do anything with this part myself; I don’t have any use for the ISRC information (which is not always there anyway), and trying to figure out how to deal with a 90 string that isn’t actually Unicode (maybe the 03 byte that comes before the first character is our clue? what a mess!) seems like too much effort for little value. 😄

[negimeister]

I have noticed the same "every second character looks correct" issue with Japanese titles. There definitely is an issue with encoding.