search

Deep-Symmetry / crate-digger

Java library for fetching and parsing rekordbox exports and track analysis files.
Other
135 stars 18 forks source link

Document and implement support for exportExt.pdb files #11

Closed eouree closed 1 week ago

eouree commented 4 years ago

Hi guys, hope you are all well.

I have already read in the Rekordbox Export Structure Analysis manual that the database export file is the export.pdb. I have created a testing one on Rekordbox using analysed tracks with extra features like colours and expected from export.pdb that the π‘Žπ‘›π‘Žπ‘™π‘¦π‘§π‘’_π‘π‘Žπ‘‘β„Ž field will contain/point an .EXT file. Instead, all π‘Žπ‘›π‘Žπ‘™π‘¦π‘§π‘’_π‘π‘Žπ‘‘β„Ž fields containing/pointing to .DAT files only.

Am suspecting that exportExt.pdb points to .EXT files. If not then how .EXT files are pointed? Does anyone knows how this works? What is the purpose of exportExt.pdb?

Parsing the exportExt.pdb on Web IDE using the .ksy file seems not working.

Can anyone help please?

Thanks

brunchboy commented 4 years ago

Hello, nice to hear from you!

There isn’t anything that I know that is not already in the analysis document: the purpose of that document is to capture and share everything I know. So you are exploring and creating new knowledge. When you do figure things out, please share them, either in this issue or as a pull request that updates the document. (I am in the process of porting both the dysentery and Crate Digger analysis documents from LaTeX to more modern and convenient Asciidoctor/Antora sites, with the help of a new tool I have created to draw the byte field diagrams, so it will be easier to work with them and offer pull requests soon.)

As far as I know, the way to find the .EXT file is to simply use string replacement on the suffix of the _analyzepath. That is what Beat Link does: https://github.com/Deep-Symmetry/beat-link/blob/master/src/main/java/org/deepsymmetry/beatlink/data/CrateDigger.java#L435-L441

I have no idea about the purpose of exportExt.pdb. I have never had any need to parse those files to accomplish my goals in Beat Link. If you study any examples and figure something out, I would very much like to know what you learn.

Out of curiosity, what are you trying to build or accomplish? It is always interesting to hear from people working in this space.

brunchboy commented 4 years ago

Also, if you could fix the .ksy so that it parses your exportExt.pdb successfully, that would be a great pull request. If you can’t, please share the file here along with the error you are seeing, in case someone else can help fix it. Or if it turns out to be the same problem reported in #10 that would be interesting to know as well.

eouree commented 4 years ago

Hello, thanks for the response and also for the document. What am trying to accomplish is to parse analysis files (.DAT and .EXT) originally created not from Rekordbox but from DJ mixer CDJ 2000 NXS2 and then transfer to Rekordbox. Any advice or recommendation would be appreciated.

Presently am using a USB with tracks that has been used on different PIONEER DJ mixers. I noticed that for some tracks there are multiple corresponding analysis files exist in different folders. So string replacement on the suffix of the analyze_path in order to get the .EXT file is not safe. For this reason am trying to find out where the PIONEER DJ mixers (and Rekordbox ) store the mapping information. For .DAT files are stored in the export.pdb but for .EXT files is not sure yet.

brunchboy commented 4 years ago

Wait, mixers can do that? How?

The way we figured out this much was by studying the files and watching network traffic captures with managed switches so we could span ports and see all communication between players and mixers, looking for patterns. As we figured out small patterns we were able to see how they related to each other and look for bigger patterns.

Please share anything you discover, it would be greatly appreciated!

TheBrenny commented 2 years ago

As far as I know, the way to find the .EXT file is to simply use string replacement on the suffix of the _analyzepath. That is what Beat Link does: https://github.com/Deep-Symmetry/beat-link/blob/master/src/main/java/org/deepsymmetry/beatlink/data/CrateDigger.java#L435-L441

Here's a corrected link for those in the future!: https://github.com/Deep-Symmetry/beat-link/blob/64d3c04726/src/main/java/org/deepsymmetry/beatlink/data/CrateDigger.java#L435-L441

brunchboy commented 2 years ago

Thanks! I didn’t know how to do commit-based perma-links on GitHub back then, or I forgot to for this one. πŸ˜„

brunchboy commented 2 weeks ago

As discussed in #36, we now can parse these files, and I will update the analysis document and Java implementation when I get back from North Coast Music Festival.

brunchboy commented 2 weeks ago

The java implementation has been updated to support parsing exportExt.pdb files, but on further thought, this approach is not particularly useful: it does not try to index the tags, and does try (pointlessly) to index all the things that are found in an export.pdb file. When I am back from North Coast and have more time, I will refactor this so that there are two separate classes, the classic Database that only supports the old files, and a new DatabaseExt that only supports the new ones, and indexes and exposes the tag information.

brunchboy commented 1 week ago

All right, this is done, and the export analysis document is updated!