search

Deep-Symmetry / crate-digger

Java library for fetching and parsing rekordbox exports and track analysis files.
Other
137 stars 18 forks source link

Add song structure (phrases) #3

Closed mganss closed 5 years ago

mganss commented 5 years ago

This adds basic support for phrase analysis data (PSSI tags).

A few things to note:

brunchboy commented 5 years ago

Thank you! I am very excited to see this, even though I won’t have time to look at the details until this weekend.

brunchboy commented 5 years ago

I can’t think of a better way to encode this than your switch on the phrase_id. You have done a great job of following the KSY style guide. The only tweak I would suggest is that on line 337, where you have a hardcoded size of 18 to consume the remaining bytes of the song_structure_entry we don’t yet have an interpretation for, it would be safer to use len_entry_bytes - 6, so if the structure grows in a future release, the parsing continues to work for the parts we understand. Would you like to push that tweak, or have me do it after merging?

The crazy values and inconsistencies you found look very similar to what we’ve found in the rest of the structure.

Thank you so much for contributing this! I would like to do some testing myself, especially when it comes to integrating it with Beat Link Trigger to display the values, but none of my files have this tag in them. It sounds like I need to buy a performance mode license and then I will be able to perform this kind of analysis, is that correct? Where do I find it in the UI then? And if you export the track to a USB, does this tag come along as well?

mganss commented 5 years ago

I've added the tweak.

AFAIK phrase information can only be shown in Performance mode. It is displayed below the waveforms if you enable Preferences → View → Layout → Phrase (Enlarged Waveform) and/or Phrase (Full Waveform).

It should be possible, however, to do phrase analysis in Export mode. I'm not 100% sure, though. You need to have it enabled in Preferences → Analysis → Track Analysis → Phrase or in the analysis dialog. Will check USB export later.

brunchboy commented 5 years ago

There is no Phrase subsection in my Preferences → Analysis section, so it must only appear after purchasing a Performance mode license. Thanks for confirming that, I will go ahead and purchase one to proceed further with testing this and incorporating it into the Java classes, and Beat Link and Beat Link Trigger.

mganss commented 5 years ago

Phrase analysis information is not included in a USB export 😞

brunchboy commented 5 years ago

Oh no! Good thing I didn’t buy my license yet, being busy with implementing cue comment text support. Phrase analysis will be of no practical value to Beat Link then, because it will have no way of accessing it.

brunchboy commented 3 years ago

Say, @mganss have you noticed that rekordbox 6 is now exporting PSSI information for tracks that have phrase analyses? I expect this is to allow the CDJ-3000 to work with their lighting controller without rekordbox. I would love to be able to start taking advantage of this in Beat Link Trigger, but the content seems corrupt compared to the analysis you did of the version found in the rekordbox filesystem. Would you be willing and able to help me study this and figure it out?

brunchboy commented 3 years ago

My current understanding of the structure of the tag has moved out of the old PDF into the new Antora documentation site, here: https://djl-analysis.deepsymmetry.org/rekordbox-export-analysis/anlz.html#song-structure-tag

mganss commented 3 years ago

@brunchboy I hadn't noticed. Sure, I'd love to help. It took me a while to get started again using the Kaitai Web IDE because the current .ksy from master triggers a bug in Kaitai, specifically https://github.com/kaitai-io/kaitai_struct/issues/825 🙄 It works if you add an id to the four character code sections:

      - id: magic
        contents: "PCP2"

I'll try and dig more, but at first glance the PSSI section in the exported file looks very different from the one in the filesystem (for Rekordbox 6 it's now in the "share" subdirectory). The structure in the filesystem has stayed the same, though.

brunchboy commented 3 years ago

Ah, I didn’t know about this new share subdirectory, thanks! Yes indeed, I have some PSSI tags in there. I also gave up on the Kaitai Web IDE and was just using a hex editor to look at what was going on, so thanks for the tip on how to get around that as well, although putting an id on those tags will change the objects that get parsed from them to add a field for the ids, which I don’t really want. Maybe we need separate .ksy files for exploration and for use in code until that bug is fixed. And they seem to be falling behind on fixing bugs, alas.

I posted a side-by-side comparison of the PSSI for the same track generated by rekordbox 5 and rekordbox 6 in Beat Link Trigger’s Gitter channel, and it looks very much like the structure is the same but the content gets encrypted starting with byte 0012.

brunchboy commented 3 years ago

Here is the PSSI section from rekordbox 5:

          0 1  2 3  4 5  6 7  8 9  a b  c d  e f   0123456789abcdef
00000000: 5053 5349 0000 0020 0000 01d0 0000 0018  PSSI... ........
00000010: 0012 0001 0000 0000 0000 01ff 0001 0000  ................
00000020: 0001 0001 0001 0001 0000 0000 0000 0000  ................
00000030: 0000 0000 0000 0000 0002 0021 0002 0000  ...........!....
00000040: 0000 0000 0000 0000 0000 0000 0001 003d  ...............=
00000050: 0003 0041 0002 0000 0000 0000 0000 0000  ...A............
00000060: 0000 0000 0000 0000 0004 0051 0003 0000  ...........Q....
00000070: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000080: 0005 0071 0002 0000 0000 0000 0000 0000  ...q............
00000090: 0000 0001 0000 0000 0006 0081 0002 0000  ................
000000a0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000b0: 0007 0091 0002 0000 0001 0001 008d 0091  ................
000000c0: 0099 0000 0001 009d 0008 00a1 0005 0001  ................
000000d0: 0000 0000 0000 0000 0000 0000 0000 0000  ................
000000e0: 0009 00bd 0005 0001 0000 0000 0000 0000  ................
000000f0: 0000 0000 0001 00df 000a 00e1 0005 0001  ................
00000100: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000110: 000b 00f1 0005 0001 0000 0000 0000 0000  ................
00000120: 0000 0000 0000 0000 000c 0105 0003 0000  ................
00000130: 0000 0000 0000 0000 0000 0000 0000 0000  ................
00000140: 000d 0111 0002 0000 0000 0000 0000 0000  ................
00000150: 0000 0000 0000 0000 000e 0121 0002 0000  ...........!....
00000160: 0000 0000 0000 0000 0000 0001 0000 0000  ................
00000170: 000f 0161 0002 0000 0001 0001 0161 0171  ...a.........a.q
00000180: 0179 0000 0001 017d 0010 0181 0005 0001  .y.....}........
00000190: 0000 0000 0000 0000 0000 0000 0001 01bf  ................
000001a0: 0011 01c1 0005 0001 0000 0000 0000 0000  ................
000001b0: 0000 0000 0000 0000 0012 01dd 0006 0001  ................
000001c0: 0000 0000 0000 0000 0000 0000 0000 0000  ................

And here it is from rekordbox 6:

          0 1  2 3  4 5  6 7  8 9  a b  c d  e f   0123456789abcdef
00000000: 5053 5349 0000 0020 0000 01d0 0000 0018  PSSI... ........
00000010: 0012 ddf2 000c f700 bf00 fa1b fbfc f3fb  ................
00000020: 05fb fb07 f3dc f301 0cf7 00bf 00fb e4fb  ................
00000030: fdf3 fb05 fafb 06f3 ddf1 002d f702 bf00  ...........-....
00000040: fbe4 fbfd f3fb 05fa fb06 f3dd f301 0cca  ................
00000050: 00bc 00ba e4f9 fdf3 fb05 fafb 06f3 ddf3  ................
00000060: 000c f700 bf00 fbe4 fbf9 f3aa 05f9 fb06  ................
00000070: f3dd f300 0cf7 00bf 00fb e4fb fdf3 fb05  ................
00000080: fafe 0682 ddf1 000c f700 bf00 fbe4 fbfd  ................
00000090: f3fb 05fb fb06 f3dd f306 0c76 00bd 00fb  ...........v....
000000a0: e4fb fdf3 fb05 fafb 06f3 ddf3 000c f700  ................
000000b0: bf07 fb75 fbff f3fb 05fb fb07 f350 f391  ...u.........P..
000000c0: 0c6e 00bf 00fa e466 fdfb fba4 fafe 06f2  .n.....f........
000000d0: ddf3 000c f700 bf00 fbe4 fbfd f3fb 05fa  ................
000000e0: fb0f f360 f305 0cf6 00bf 00fb e4fb fdf3  ...`............
000000f0: fb05 fafb 06f2 dd2c 0006 f7e1 bf05 fbe5  .......,........
00000100: fbfd f3fb 05fa fb06 f3dd f300 0cf7 00bf  ................
00000110: 00f0 e40a fdf6 fb04 fafb 06f3 ddf3 000c  ................
00000120: f700 bf00 fbe4 fbfd f3f7 04ff fb05 f3dd  ................
00000130: f300 0cf7 00bf 00fb e4fb fdf3 fb05 fafb  ................
00000140: 06fe dce2 000e f700 bf00 fbe4 fbfd f3fb  ................
00000150: 05fa fb06 f3dd f300 0cf9 019e 00f9 e4fb  ................
00000160: fdf3 fb05 fafb 06f3 ddf3 000d f700 bf00  ................
00000170: fbeb fa9c f3f9 05fa fb07 f3dc f261 0d86  .............a..
00000180: 01c6 00fb e4fa fc8e fb15 fb7a 06f6 ddf2  ...........z....
00000190: 000c f700 bf00 fbe4 fbfd f3fb 05fb fab9  ................
000001a0: f3cc f2c1 0cf2 00be 00fb e4fb fdf3 fb05  ................
000001b0: fafb 06f3 ddf3 000c f712 bedd fbe2 fbfc  ................
000001c0: f3fb 05fa fb06 f3dd f300 0cf7 00bf 00fb  ................
mganss commented 3 years ago

The bytes from style forward are "encrypted" in chunks of 19 bytes each by XOR'ing the plaintext with a key. Here are the first 19 bytes of a sample ciphertext (rekordbox export file):

E3 F8 06 12 FD 06 C5 06 03 39 00 03 F9 01 0B 01 01 0D F9

And here's the plaintext (rekordbox file system):

00 01 00 00 00 00 00 00 02 D3 01 00 00 00 00 01 00 01 00

The key for this file is:

E3 F9 06 12 FD 06 C5 06 01 EA 01 03 F9 01 0B 00 01 0C F9

Unfortunately, the key is different for each file. I don't know yet how the keys are chosen. There is a second track in my export that uses the same key, though (same number of phrases, too). Perhaps there is a fixed collection of keys?

brunchboy commented 3 years ago

Ooh, this is a huge breakthrough! I hoped it was something this simple but did not expect to get this far this fast. There must be a way to determine the keys from the files themselves, so the players know how to do it. Thanks so much, I will start poking further after work, but do let me know if you have any further insights! 🎉

brunchboy commented 3 years ago

The number of phrases is the last unencrypted value in the tag, perhaps that is used as a seed for a pseudo-random number generator to produce the key?

mganss commented 3 years ago

Even more simple. This is the "main key":

CB E1 EE FA E5 EE AD EE E9 D2 E9 EB E1 E9 F3 E8 E9 F4 E1

Add the number of phrases to each byte to get the individual key for the file.

mganss commented 3 years ago

I had hoped the key would spell out a phrase but it doesn't seem to 😞

brunchboy commented 3 years ago

You rule! 🥇

brunchboy commented 3 years ago

And yes, it looked like it might have been letters or something cute.

brunchboy commented 3 years ago

All right, I am going to add a processing step to the .ksy file to unmask the content. That means it will no longer work with the clean versions in the shared folder of the filesystem. If you need support for that, we’ll need to use an alternate version of the .ksy I think.

brunchboy commented 3 years ago

Actually, I have a much better idea, @mganss: I will add a static method to the new UnmaskSongStructureTag class which you can call to configure it to do nothing, so if you need to parse an unmasked PSSI tag from the shared folder, you can call that first and just do it as you used to. Although if you are not using Crate Digger to parse things using this .ksy file, then you will need to implement your own custom processing class that behaves the same way in whatever language you are using.

In either case though, everything after len_entries is now nested one layer deeper, in body, in order to be processed when needed.

brunchboy commented 3 years ago

All right, that’s pushed, you can now call org.deepsymmetry.cratedigger.pdb.UnmaskSongStructureTag.disable() to parse un-masked .EXT files in the snapshot version of Crate Digger.

mganss commented 3 years ago

👍🏻 Is there a way to distinguish between export and internal files? If not, perhaps the unmask method could run some heuristic to tell if the bytes it has been passed are encrypted or not.

Also I've been wondering why Pioneer is playing these little games? I'm sure they're not really trying to encrypt the data. Perhaps it's a legal thing and they want to keep control over who they're letting use the data. If it's us they probably don't care (or even secretly encourage us), but if it's a competitor they might be able to sue them because they "broke" the encryption.

brunchboy commented 3 years ago

They’ve always been control freaks but I suspect this was driven by anger over Denon selling hardware and saying “oh, you have a rekordbox USB? No problem, stick it in and DJ with all your cues!”

And I doubt they are monolithic. There are probably some developers who secretly encourage us, and some managers, lawyers, or bean-counters who hate us and don’t understand the value we bring to the ecosystem. 😄

brunchboy commented 3 years ago

I think the only way to know if a file is exported is by knowing where you found it. You could guess by whether the style seemed insane, but I'd rather leave the decision to the calling code.

Be-ing commented 3 years ago

Also I've been wondering why Pioneer is playing these little games? I'm sure they're not really trying to encrypt the data. Perhaps it's a legal thing and they want to keep control over who they're letting use the data.

That seems likely considering the vendor ID strings found for the CDJ HID handshake.

brunchboy commented 3 years ago

I’ve got Beat Link successfully retrieving and understanding these tags from players (even ones older than the CDJ-3000) now, both via dbserver queries, and by downloading the file over NFS (both approaches are necessary to cope with all the combinations of players, rekordbox, or USB/SD cards that one might find in a performance setting). Also, Kevinnns has made progress on figuring out how the “mood” options are represented in the files as well. @mganss are you interested in that? The discussion is happening on the new Deep Symmetry Zulip instance: https://deep-symmetry.zulipchat.com/join/u7acmljriw3wml7gneylxmul/ (specifically, in the “beat link trigger” stream and the topic “phrase support design” within that.)