brunchboy
commented
7 years ago Hello! Thank you so much for writing this, it is always delightful to find more people doing research about these topics, especially when they are willing to share. I probably won’t have time to explore this information in detail until the weekend, but I can already see there are some interesting new discoveries here. It was fun to see you explain the UDP packet that causes a CDJ to load a track, I had always been a bit curious about that.
I would love to learn the details of the track waveform files! Have you seen the user interface that my own lighting software draws on the graphical display of the Ableton Push 2 controller? You can see an animated capture of it on this page if you scroll down past the large photograph. It would be amazing to be able to replace the metronome graph on the left with waveforms from the master CDJ.
Also, did you look at beat-link as a starting point for your own Java Link library? It has a fair bit of features already, and its license should be compatible with whatever use you want to make of it. (Although I would very much appreciate pull requests for things like adding waveform retrieval! 😁) And have your light technicians played with beat-link-trigger at all? I’ve had people use it at music festivals and big clubs around the world to synchronize visuals, and even had some headlining artists use it to synchronize synthesizers with tracks on their CDJs.
Finally, I have been very careful to not try to implement or document any sort of access to the music data itself. I think that if people start doing that, and big DJs and record labels hear about it, they will start fearing their music is in danger of being stolen, and will pressure Pioneer to encrypt the protocol, which will completely destroy our ability to do cool things like synchronizing our light and laser shows, visuals, and synthesizers to it.
Fraeyon
marek-memsql
Hello there ! Since i am working like people there to discover all of Pioneer ProDJ Link's hidden secrets I decided to participate to this projet by bringing you some thoughts I have about how Pioneer CDJs/ Rekordbox are exchanging data through NFS implementation.
First, to introduce myself briefly I am a French engineering student involved in an association bringing to students live DJing, light shows and sound services all across the region. One day, some of our light technicians wondered if we were able to visualize in realtime what the DJ is doing on the players to sync perfectly our light show and that's how I started working on the Link protocol, at least a year ago now. The ideal would be to be able to get track's waveform and current play position within track.
Then I discovered this GitHub repo and by reading all of your analysis PDF I was surprised by the quality of the work, and this document helped me going further in my exploration of the protocol by lighting some shadowy areas of packet content I didn't understood.
And as I'm really close to achieve what I wanted to do (remember, get live waveform data from playing CDJs) I wanted to share some discoveries and thoughts I have about how the whole process works. So let's go !
NFS v2 is the file sharing protocol used in ProDJLink.
About Link protocol's continuous packet flow from ports 50000/1/2 : There are some special packets that seem to be sent juste before an NFS or TCP negociation.
A 36 Byte packet to queried device port 50002 will always precede a the first NFS portmap->export->mount negociation. This packet seems to contain only the player's ID. No idea if there are ID limitations like in metadata queries. After sending the packet, the players will enter a waiting state, and the NFS-serving device will then send a specific response to each player, one after each other
By receiving this packet, the player will negociate a NFS filehandle for each root path of the server. In the case you are querying a laptop running Rekordbox the exported paths will be your drive, designated by their native mount letter (C, D , ...)
This negociation will take three steps : a first packet will ask for 'mount/export' service port by sending a Portmap packet to port 50111 of the server. The next packet will ask for available exports (by returning drive letters associated with their hosting IP/Mask, in this case those of the server). The final packets will be Mount packets, asking for drive nfs's filehandles. (Cf. NFS v2 spec)
At this point, the player is ready to load track data from the server but we have to tell it to do so. Indeed, it seems that track loading is driven by a combination of TCP and NFS requests as the first is used to retrieve track data (and a very important data : it's path) and the latter serves the raw files and their content.
The connection to the DeviceSQL server from the device is the same as described in your analysis document.
The process of TCP connection is as follows : 1. Get the port of device SQL server by send the 00 00 0f REKORDBOX packet. 2. Do the '11 00 00 00 01' 5-bytes handshake with the server (packet ids being ff ff ff fe as described in the analysis). 3. Issue the '00 00' 37-byte request to the server and wait for a 42 byte response.
What do I call a '00 00' request ? It's a name I gave to all 42-bytes requests starting with the separator '11 87 23 49 ae 11', followed by the packet count and then by the request type. It seems that requests are identified by a 2-byte identifier right in the middle of the 4 bytes following the packet ID. Because the first (10) and last (0F) bytes of the request field never changes only two bytes are used and a '00 00' request is a request with bytes 10 00 00 0F following the packet ID.
A '11 field' is a 4 byte field separated from others by a '11' byte. If you look precisely at TCP response packets from CDJs or Rekorbox, these separators are always present in the messages and contain various data.
On reception of a specific 192 bytes packet issued by the server, the device will always connect to it (if not done so, by using the process described before) and send to the server a "30 07" query. But I have no idea of what this request stands for...
Request may be 52 bytes long. In this case, acknowledgment messages may be 52 bytes long too, but not necessarily.
A 62 byte TCP request usually is a '30 00' (EXEC) request. This packet will trigger the request on the server, and the response will be sent back.
Some (EXEC) paquets doesn't need to be preceded by a 42-byte request packet and will make the server respond immediatly to the request.
Some request types :
To disconnect from DeviceSQL server, send a 32 byte packet with ID "ff ff ff fe" and call the "01 00" request in it. The server will acknowledge and then you can close the socket !
The general process of loading a track is as follows :
(EDIT) To make a CDJ device to load a track from Rekordbox (the process that takes place when you drag'n'drop a track on a CDJ displayed in the "LINK" ribbon of rekordbox), you have to send a 88 byte UDP packet from port 50002 of the sender machine to the port 50002 of the destination player containing the rekordbox ID of the requested track at bytes 86-90. Then, the device will ask for path metadata with request "21 02" and NFS operations will occur.
Being a little rush about writing this document I know some points may be unclear, don't hesitate to contact me and I will be able to give more details about my findings. I'm also working on the JAVA version of a Pioneer Link library to support my original software.
In the hope you'll find this findings useful enough to take the job further ;D , Yours,
HunterDKS