no tags found when you call a second track on CDJnxs2

[knickita]

hi! experimenting with a CDJ2000NXS2 i found that when i start your script (monitor-qt.py) it recognize the track info (title, artist, album) that is playing. but when i change track it crash saiyng that there is no artist/album/track in reply.keys(). I introduced a check to prevent this (just submitted a merge request), but still i don't know how to solve the problem.

keep up the good work, i'm really interested in contributing.

[flesniak]

Interesting find, thanks for your report! Can you copy the full shell output (debug messages) including the python backtrace? I'm wondering why it's using dbclient at all, because it should prefer pdb parsing. Probably the pdb parsing problem is related to the construct-v2.9 changes. Things you could do to help me fix it:

1. Give me full debug output + Backtrace
2. Try branch "fixes" which contains minor improvements to the pdb code
3. Try master at commit cb86fdf60667c8ac3495dcc31fd763f42b1f8f5d

I'll gladly be trying to fix this bug, but I'll skip the pull request until I fully understand the problem.

[knickita]

following your 3 steps:

1. Give me full debug output + Backtrace

i have done some tests with branch "construct-v2.9", in all my tests i had never been able to see the actual positionof the track (elapsed time or remain time) neither the waveform of the track. i just see the BPM and the track info (title,/artist/album)

tested on a CDJ2000nxs2 with firmware version 1.84 using construct==2.9.45

1) a track is already playing, i start the script, it recognize the track:

DEBUG: DataStore: 0x7f873f890620 initialized
DEBUG: DataStore: 0x7f873f8906d0 initialized
DEBUG: DataStore: 0x7f873f890678 initialized
DEBUG: DataStore: 0x7f873f890728 initialized
DEBUG: DataStore: 0x7f873f8907d8 initialized
DEBUG: DataStore: 0x7f873f890780 initialized
DEBUG: DataStore: 0x7f873f890888 initialized
INFO: Gui: Creating player 0
DEBUG: Renderer "GeForce GTX 960M/PCIe/SSE2" OpenGL "4.6.0 NVIDIA 390.48"
INFO: Listening on 0.0.0.0:50000 for keepalive packets
INFO: Listening on 0.0.0.0:50001 for beat packets
INFO: Listening on 0.0.0.0:50002 for status packets
DEBUG: DataProvider starting
DEBUG: ProDj: starting main loop
INFO: Player number set to 5
INFO: Starting virtual cdj with player number 5
INFO: New Player 33: DJM-900NXS2, 169.254.31.104, c8:3d:fc:02:1f:68
INFO: Guessed own interface enp2s0 ip 169.254.12.12 mask 255.255.0.0 mac 20:47:47:c7:42:ac
INFO: New Player 2: CDJ-2000NXS2, 169.254.189.166, c8:3d:fc:02:bd:a6
DEBUG: Gui: reassigning default player 0 to player 2
INFO: New Player 5: Virtual CDJ, 169.254.12.12, 20:47:47:c7:42:ac
DEBUG: query link info to 169.254.189.166 struct {'type': 'link_query', 'model': 'Virtual CDJ', 'player_number': 5, 'extra': {'source_ip': '169.254.12.12'}, 'content': {'remote_player_number': 2, 'slot': 3}}
DEBUG: Media usb in player 2 changed
INFO: Gui: player 2 loaded bare file 7, requesting info
DEBUG: DataProvider: enqueueing track_info request with params (2, EnumIntegerString.new(3, 'usb'), 7)
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 7) from pdb
DEBUG: PDBProvider: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 7)
WARNING: DataProvider: pdb failed [PDBProvider: invalid request type track_info]
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 7) from dbc
DEBUG: DBClient: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 7)
INFO: Player 2 Link Info: usb "CDJ", 0 tracks, 0 playlists, 266/1996MB free
DEBUG: Media usb in player 2 changed
INFO: DBClient port of player 2: 1051
DEBUG: DBClient: initial packet reply 1
INFO: DBClient: connected to player 2
DEBUG: DBClient: query_list request: {'transaction_id': 1, 'type': 'track_info_request', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 7}]}
DEBUG: DBClient: query_list track_info_request: 10 entries available
DEBUG: DBClient: render query {'transaction_id': 2, 'type': 'render', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 10}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 10}, {'type': 'int32', 'value': 0}]}
DEBUG: DBClient: parse_metadata menu_header
DEBUG: DBClient: parse_metadata menu_footer
DEBUG: Gui: dbclient_callback track_info source player 2 to widget player 2

then i click "next track button" on the cdj2000nxs2 and the error pops up:

INFO: Gui: player 2 loaded bare file 18719, requesting info
DEBUG: DataProvider: enqueueing track_info request with params (2, EnumIntegerString.new(3, 'usb'), 18719)
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 18719) from pdb
DEBUG: PDBProvider: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 18719)
WARNING: DataProvider: pdb failed [PDBProvider: invalid request type track_info]
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 18719) from dbc
DEBUG: DBClient: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 18719)
DEBUG: DBClient: query_list request: {'transaction_id': 3, 'type': 'track_info_request', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 18719}]}
DEBUG: DBClient: query_list track_info_request: 4294967295 entries available
DEBUG: DBClient: render query {'transaction_id': 4, 'type': 'render', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 4294967295}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 4294967295}, {'type': 'int32', 'value': 0}]}
DEBUG: DBClient: parse_metadata menu_header
DEBUG: DBClient: parse_metadata menu_footer
DEBUG: Gui: dbclient_callback track_info source player 2 to widget player 2
Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/home/nick/repositories/python-prodj-link/dataprovider.py", line 200, in run
    self._handle_request(*request[:-1])
  File "/home/nick/repositories/python-prodj-link/dataprovider.py", line 177, in _handle_request
    callback(request, *params, reply)
  File "/home/nick/repositories/python-prodj-link/gui.py", line 512, in dbclient_callback
    self.players[player_number].setMetadata(reply["title"], reply["artist"], reply["album"])
KeyError: 'artist'

2) a track is already playing, i start the script, it recognize the track:

DEBUG: DataStore: 0x7fa76dce2620 initialized
DEBUG: DataStore: 0x7fa76dce26d0 initialized
DEBUG: DataStore: 0x7fa76dce2728 initialized
DEBUG: DataStore: 0x7fa76dce2780 initialized
DEBUG: DataStore: 0x7fa76dce27d8 initialized
DEBUG: DataStore: 0x7fa76dce2678 initialized
DEBUG: DataStore: 0x7fa76dce2830 initialized
INFO: Gui: Creating player 0
DEBUG: Renderer "GeForce GTX 960M/PCIe/SSE2" OpenGL "4.6.0 NVIDIA 390.48"
INFO: Listening on 0.0.0.0:50000 for keepalive packets
INFO: Listening on 0.0.0.0:50001 for beat packets
INFO: Listening on 0.0.0.0:50002 for status packets
DEBUG: DataProvider starting
DEBUG: ProDj: starting main loop
INFO: Player number set to 5
INFO: Starting virtual cdj with player number 5
INFO: New Player 2: CDJ-2000NXS2, 169.254.189.166, c8:3d:fc:02:bd:a6
INFO: Guessed own interface enp2s0 ip 169.254.12.12 mask 255.255.0.0 mac 20:47:47:c7:42:ac
DEBUG: Gui: reassigning default player 0 to player 2
INFO: New Player 5: Virtual CDJ, 169.254.12.12, 20:47:47:c7:42:ac
INFO: New Player 33: DJM-900NXS2, 169.254.31.104, c8:3d:fc:02:1f:68
DEBUG: query link info to 169.254.189.166 struct {'type': 'link_query', 'model': 'Virtual CDJ', 'player_number': 5, 'extra': {'source_ip': '169.254.12.12'}, 'content': {'remote_player_number': 2, 'slot': 3}}
DEBUG: Media usb in player 2 changed
INFO: Gui: player 2 loaded bare file 7, requesting info
DEBUG: DataProvider: enqueueing track_info request with params (2, EnumIntegerString.new(3, 'usb'), 7)
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 7) from pdb
DEBUG: PDBProvider: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 7)
WARNING: DataProvider: pdb failed [PDBProvider: invalid request type track_info]
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 7) from dbc
DEBUG: DBClient: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 7)
INFO: Player 2 Link Info: usb "CDJ", 0 tracks, 0 playlists, 266/1996MB free
DEBUG: Media usb in player 2 changed
INFO: DBClient port of player 2: 1051
DEBUG: DBClient: initial packet reply 1
INFO: DBClient: connected to player 2
DEBUG: DBClient: query_list request: {'transaction_id': 1, 'type': 'track_info_request', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 7}]}
DEBUG: DBClient: query_list track_info_request: 10 entries available
DEBUG: DBClient: render query {'transaction_id': 2, 'type': 'render', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 10}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 10}, {'type': 'int32', 'value': 0}]}
DEBUG: DBClient: parse_metadata menu_header
DEBUG: DBClient: parse_metadata menu_footer
DEBUG: Gui: dbclient_callback track_info source player 2 to widget player 2

then i select the other track from the CDJ's screen pushing the rotary encoder and it doesn't throw errors but doesn't change the track title in the view window:

INFO: Gui: player 2 loaded bare file 18719, requesting info
DEBUG: DataProvider: enqueueing track_info request with params (2, EnumIntegerString.new(3, 'usb'), 18719)
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 18719) from pdb
DEBUG: PDBProvider: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 18719)
WARNING: DataProvider: pdb failed [PDBProvider: invalid request type track_info]
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 18719) from dbc
DEBUG: DBClient: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 18719)
DEBUG: DBClient: query_list request: {'transaction_id': 3, 'type': 'track_info_request', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 18719}]}
DEBUG: DBClient: query_list track_info_request: 4294967295 entries available
DEBUG: DBClient: render query {'transaction_id': 4, 'type': 'render', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 4294967295}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 4294967295}, {'type': 'int32', 'value': 0}]}
DEBUG: DBClient: parse_metadata menu_header
DEBUG: DBClient: parse_metadata menu_footer
DEBUG: Gui: dbclient_callback track_info source player 2 to widget player 2

then i reselect the first track from the CDJ's screen pushing the rotary encoder...

INFO: Gui: player 2 loaded bare file 7, requesting info
DEBUG: DataProvider: enqueueing track_info request with params (2, EnumIntegerString.new(3, 'usb'), 7)
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 7) from pdb
DEBUG: PDBProvider: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 7)
WARNING: DataProvider: pdb failed [PDBProvider: invalid request type track_info]
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 7) from dbc
DEBUG: DBClient: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 7)
DEBUG: DBClient: query_list request: {'transaction_id': 5, 'type': 'track_info_request', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 7}]}
DEBUG: DBClient: query_list track_info_request: 10 entries available
DEBUG: DBClient: render query {'transaction_id': 6, 'type': 'render', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 10}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 10}, {'type': 'int32', 'value': 0}]}
DEBUG: DBClient: parse_metadata menu_header
DEBUG: DBClient: parse_metadata menu_footer
DEBUG: Gui: dbclient_callback track_info source player 2 to widget player 2

and finally reselect the second track always from the CDJ's screen pushing the rotary encoder, and it changes the title in the view window!

INFO: Gui: player 2 loaded bare file 18719, requesting info
DEBUG: DataProvider: enqueueing track_info request with params (2, EnumIntegerString.new(3, 'usb'), 18719)
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 18719) from pdb
DEBUG: PDBProvider: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 18719)
WARNING: DataProvider: pdb failed [PDBProvider: invalid request type track_info]
DEBUG: DataProvider: trying request track_info (2, EnumIntegerString.new(3, 'usb'), 18719) from dbc
DEBUG: DBClient: handling track_info request params (2, EnumIntegerString.new(3, 'usb'), 18719)
DEBUG: DBClient: query_list request: {'transaction_id': 7, 'type': 'track_info_request', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 18719}]}
DEBUG: DBClient: query_list track_info_request: 10 entries available
DEBUG: DBClient: render query {'transaction_id': 8, 'type': 'render', 'args': [{'type': 'int32', 'value': 66305}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 10}, {'type': 'int32', 'value': 0}, {'type': 'int32', 'value': 10}, {'type': 'int32', 'value': 0}]}
DEBUG: DBClient: parse_metadata menu_header
DEBUG: DBClient: parse_metadata menu_footer
DEBUG: Gui: dbclient_callback track_info source player 2 to widget player 2

2. Try branch "fixes" which contains minor improvements to the pdb code

using exact same equipment and tracks as step 1, construct==2.8.22 exact same behaviour as step 1

3. Try master at commit cb86fdf

using exact same equipment and tracks as step 1, construct==2.8.22 exact same behaviour as step 1 and 2

[flesniak]

Thanks for testing! Now I see the issue - you've hit an untested corner-case. When playing tracks which are not rekordbox-analyzed, the pdb provider (which uses the rekordbox database) can't query the track information. Thus, it falls back to dbclient. Because pdb is preferred, I did not test dbclient a lot. Apparently there is a bug in parsing the reply, but I have to look into this tomorrow.

[brunchboy]

Hey, can you explain what you mean by this? I’m trying to understand what your pdb provider is. Have you figured out how to read media sticks directly, and do you do that to avoid having to query the players? If so, how do you get the media export onto your filesystem? And could you please write up the details of the format the way that I did for the protocol in dysentery? That would be a huge contribution to this effort!

[flesniak]

The pdb provider parses the rekordbox database (named export.pdb, hence the name). Yes, I do not use the dbserver-protocol by default anymore, but as a fallback if pdb fails. I simply download the export.pdb via NFS and parse it. All requests afterwards are answered "locally" by querying the database. When playing non-rekordbox-analyzed tracks, this approach obviously does not work. Then the fallback to dbclient fails, which is what this bug is about. Unfortunately, I wasn't able to test it this weekend. Maybe it's possible to get the loaded file path/inode/whatever and download metadata using NFS to get rid of that dbclient call. Regarding documentation: I would like to if I have some spare time in the future. Unfortunately pretty busy currently. The python-construct implementation of the database parser reads more-or-less vivid, but of course not as good as a real documentation. There are even some very strange things inside the pdb which are commented, but should be explained better for other people to understand the required hacks.

[brunchboy]

Thanks for the clarification! I had done a bit more looking and suspected this was going on, and this confirmation is very exciting. Can export.pdb be loaded via NFS even when there are four CDJs on the network all accessing the same media?

Can you write up the details of how to parse the database in a language-independent way like I did for the dbserver protocol? That would be very helpful to many people!

In order for your dbserver queries to work for non-rekordbox tracks, you need to be sending valid status packets as if you are a CDJ. It is not enough to be sending presence-announcement packets. Alternately, as you say, you might be able to find some way to translate the track “id” values into something that would let you retrieve them over NFS, and then just look for the ID3 data in the track itself. If you make any progress on that front, please let us know.

[flesniak]

Hmm, I can't reproduce it. Yesterday, I was able to make the CDJ send a menu rendering as a reply to a track info request (0x2202), which was very strange. But I can't make it do that now, maybe it was just related to a bad ethernet cable. @knickita Can you capture tcp traffic on port 1051 when the bug happens? Hopefully that clears up things. @brunchboy I can query track info of non-rekordbox tracks (request 0x2202) even when not sending status packets, but presence packets only. Are you sure this was usually necessary?

[brunchboy]

Yes, I was definitely unable to retrieve track info using 0x2202 reliably for most file types (I think it worked best for AAC files, but usually failed for MP3, WAV, and AIFF files if I remember correctly) on my CDJ-2000 nexus players unless I was sending status packets. Red Bull Radio reported similar issues and I believe they had nxs2 players.

Could you please confirm whether the nfs approach works even with four busy players? I am considering trying to implement it, but only if there is a clear benefit because it looks like a huge amount of work, especially considering I am finding the Construct code very hard to understand as a non-Python programmer. Any language-independent reverse engineering notes and diagrams you could share would be hugely appreciated.

[flesniak]

I can confirm that I can query four players without problems - this was why I implemented it. The NFS server on CDJs does not seem to do any checking or limiting of its clients, or at least I have not hit anything yet. I had to write a custom NFSv2-over-UDP implementation as I could not find one anywhere, but I sticked to the RFCs and it turned out working well. Parsing the PDB hat another advantage: Imagine a DJ only using a single USB stick, then I just have to download the PDB once for his whole set. After downloading, every database request is answered locally and thus nearly instantly. Only the waveform files need to be downloaded for each track individually.

[brunchboy]

All right, thanks very much for the confirmation. I realized that downloading the entire PDB would mean local and fast access for the track information—I get the same benefit when I am using a metadata archive (although in that case I have the waveforms, beat grids, cue lists, and artwork locally as well, and those are much bigger than the track information). But working with four players is huge, that is the main issue for many of my users. So… you have just created a vast amount of work for me. 😉

There is an NFS client for Java which supports UDP and NFSv2 (it was even originally written at Sun, where Java and NFS were invented), and I thought that would save me some time, but unfortunately it does not work with CDJs, because Pioneer’s server is nonstandard, and demands UTF-16BE strings for mount paths and file names; the Java client sends ASCII (or, perhaps, UTF-8). In any case, this means I have to do the same as you did, and build it from scratch. Thank you for proving that it works, and giving me an example.

There is also a Java port of Construct. But instead of using that, I think I will use Kaitai Struct. That is language neutral (with good support for both Python and Java among many others), so once I create the definitions they will be usable by people porting these ideas to other languages as well. It looks like the authors of Construct are collaborating with Kaitai. You may be interested in converting your code to use it as well; the visualizer and IDE look very handy.

[brunchboy]

I take that back, at least partially… Kaitai can only parse structures, it can’t build them, so it will only be useful for reading the .pdb database. So there is probably not much value in you changing anything. I will still use it for that, I think, to help make that easier for other languages. The web IDE is already proving tremendously helpful in validating my efforts at writing the .ksy templates. I suspect (but have not yet proven) that the unknown page types are the different indices that the DJ can choose to include in the database.

[flesniak]

Closing this issue due to missing information. If the issue persists with the current version, feel free to open the issue again.