playlist_map

[whartley94]

Hello!

First off just want to say thanks for the access to such a sick project!

I've been trying to use some of the code from the pdblib folder to write a python script that generates .m3u playlists (for Traktor) from/onto a Rekordbox exported USB. I'm basically using load_file in the pdbdatabase.py script on the .pdb on my Rekordbox exported USB and then a small bit of my own code to generate the playlists. I would say I've got this working most of the way but I've noticed that my playlists in Traktor don't have all the tracks in the original playlists from Rekordbox.

I've tried to look into why this might be, by examining the playlist_map generated from the pdb. Choosing a playlist I know has x tracks in (looking at Rekordbox), I do a basic for loop checking how many containers in the playlist_map actually have the playlist_id of said playlist. From what I can see, the playlist_map generated only has containers for a portion of tracks in the playlist I am trying to convert, the same portion that end up in the Traktor playlist I export. I'm wondering if anyone has an intuition as to why this could be. Could there be a problem with the code that generates the playlist_map or am I just misinterpreting it's structure.

Please let me know if there's more information I can provide. I'm not sure how else to test things.

Thanks, Will

[whartley94]

Just thinking, maybe I could export a small pdb file to upload for people to look at?

[flesniak]

Hi Will, do I understand your problem correctly: Using PDBDatabase.getPlaylist() does not return all the tracks that should be in this playlist? If you are accessing PDBDatabase["playlist_map"] directly, you may have a look at the getPlaylist function to see how to filter playlist_map.

If you can upload a small sample PDB file, that would be very useful if you think it is a file format issue.

[whartley94]

Hi!

You're understanging correctly yes. I've tried using getPlaylist but it wasn't returning all my tracks. I then was looking at the elements of playlist map to see if there might be an issue with the filtering, but I think it seems they're not all in the playlist map either.

Sorry it's taken me a while to respond, I've been trying to replicate the problem with a smaller playlist and then additionally check what a CDJ shows (whether it has all the tracks). Will let you know once I have done this!

[whartley94]

Ok so have managed to replicate on a smaller playlist. Here I'm syncing one playlist named 'Bass Set 4 n 5' to my USB, which has 218 songs in it. I can confirm that all 218 songs are synchronised to the USB by using os.walk over the 'Contents' folder. Evaluating len(list(mydb.get_playlist(6))), where the ID for the playlist in mydb['playlists'] is 6, returns only 54. Similarly len(mydb['playlist_map']) = 54 also, as there's only the one playlist synced to the USB.

I've uploaded the contents of the 'rekordbox' folder on the USB here, including the pdb: https://drive.google.com/open?id=1BgUSMRkuKO9KmxJUx-kjlzrujRvVzG8n

I have just noticed that there's two .pdb files. I am currently running this all on 'export.pdb'. I'm not sure what 'exportExt.pdb' is, but if I try to use load_file on it I get: RuntimeError: PDBDatabase: failed to parse the complete file (32768/73728 bytes parsed).

Any thoughts would be greatly appreciated :) Thanks!

[brunchboy]

I’ve been following along and I can report that Crate Digger has problems with this export as well (not surprising, as it was inspired by discoveries in this project). When I look in the playlist index for ID 6, I find 216 (not 218) entries, but only 54 of those have non-zero track IDs. I will try poking at the database file using the Kaitai Struct web IDE and my PDB mapping file and see if I can figure anything out (although the playlist structures and the tables in general are so tedious to try to follow). I’ll report back if I discover anything. Of course if you figure out anything at your end I’d love to hear it! Hopefully we can get both of our projects fixed.

I don’t believe anyone knows what might be present in an exportExt.pdb file… presumably tables that newer players support but older player’s don’t. The Kaitai Struct web IDE can’t parse it using the existing PDB mapping file, in any case.

[brunchboy]

Well, after a quick scan through in the web IDE, it looks like those 54 tracks are the only rows in the playlist table that are marked as valid and present. There are tons and tons of playlist entry pages which lack any valid rows, however. The state of our current understanding of the PDB format is captured in this document. If you can study the content of your file and find out where we are wrong, that would be a great help. I don’t know when I will have time. Does the exported playlist work properly on your hardware? What hardware is it?

[flesniak]

Okay, so I think I made at least some progress. As @brunchboy said, ony 54 of the 218 playlist entries are marked as valid/enabled by the reverse index. However, the remaining 164 playlist mappings do exist in the database. It turns out that both entry_count_large and entry_count_small do not count them, thus they are not parsed.

block <index>: <type> next <index> # <deduced entry count> (<small>|<large>|<override>)
idx   18: block_playlist_map next  53 #  28 (   0|  28| 284) size 3408 free    8 u1  7453 u3   1 u4   0 u5  52 u9    0 u10  0
idx   53: block_playlist_map next 110 #  28 (   0|  28| 123) size 3408 free    8 u1 10222 u3  33 u4  20 u5  52 u9    0 u10  0
idx  110: block_playlist_map next 218 #  54 (  54|  28| 144) size 3408 free    8 u1 10516 u3  33 u4   7 u5  52 u9    0 u10  0

Above you see the parsed headers of the three largest playlist_map blocks, all showing a size of 3408 bytes. As entries are 12 bytes in size in this table, this results in 284 rows per block. Most of the entries (except block 110) are not enabled in the reverse index, but instead flagged in entry_enabled_override. I suppose this means that these entries override a row from another block, but I'm not sure as such rows also appear in block 18 which is the first playlist_map block (and thus does not have any previous rows to override).

The page header value previously tagged u7 is definitely related to that, as it is exceptionally large for these three blocks and even matches the entry count for block 18. Thus, I tagged it overridden_entries for now. Unfortunately, this additional entry count does not always match the total entry count of 284 rows for these three blocks. For blocks 53 and 110, there are still some entries missing and I don't know how to calculate the total entry count. I think we can't just do size/entry_size as some entries may have padding in between. Probably values u3 and u4 help here as they are zero for all other playlist_map blocks, but I didn't figure that out yet.

I'll investigate this further. Thanks a lot @whartley94 for reporting this. Apparently our understanding of the PDB database format is still incomplete.

@brunchboy: exportExt.pdb is unrelated to this issue. My XDJ-1000 can read the full 218 track playlist just with export.pdb. I'm still wondering what the ext file contains, though.