search

Deep-Symmetry / beat-link-trigger

Trigger events and automate shows in response to events on Pioneer CDJs
Eclipse Public License 2.0
431 stars 37 forks source link

Rekordbox in export mode, metadata not working #43

Closed Fra-Ktus closed 6 years ago

Fra-Ktus commented 6 years ago

For me, I can't receive metadata when the music is coming from Rekordbox in export mode...

I also built a prototype in C and came to the same conclusion. Track metadata ready response, Figure 24 in the document describing the protocol, at byte 0x26 I receive 0 or 0xFFFFFFFF depending on the decks I have tried...

I have the same behavior with beat-link-trigger.

Should it work with tracks coming from Rekordbox in export mode?

brunchboy commented 6 years ago

I’m sorry, I don’t know what you mean exactly. What is export mode? But the most basic answer is that it works however you find that it works. I have not been able to test much with rekordbox running because that can’t run on the same machine as Beat Link, and I only have one machine available most of the time.

If you can capture some packets and figure out how to make it work correctly, that would be much appreciated.

Fra-Ktus commented 6 years ago

The export mode is the free mode of Rekordbox, it must just be on the same network than the deck and beat-link-trigger. The deck is then using the tracks from the Rekordbox database. The deck sees the metadata but beat-link-trigger cannot see them and I have the same behavior when I try to implement the protocol myself. I can show the packets, can I send that directly to you?

brunchboy commented 6 years ago

Ok, you are talking about what I still think of as just rekordbox, because I have actual CDJs, and no need to purchase the laptop-based playback mode. 😄

In any case, I am hoping that you can perform the analysis yourself and figure out how to make it work; I am very busy with other tasks at the moment. Otherwise we can leave this issue open and I will investigate it when I have time. If you do have a packet capture of the CDJs loading metadata from rekordbox, especially if you can annotate it with information about the relevant IP addresses and packet numbers, that would definitely be nice to upload to this issue.

brunchboy commented 6 years ago

One thing that is strange to me is that I have a strong feeling that this used to work just fine, even more reliably than requesting metadata from tracks loaded from CDJs. I wonder what changed?

Fra-Ktus commented 6 years ago

The document describes everything nicely but I receive 0 or -1 in the Track metadata ready response... Thanks for the quick response, I will investigate this weekend and keep you updated.

brunchboy commented 6 years ago

I brought home an extra computer this weekend, and I am unable to reproduce this problem. When I load a track on my CDJ-2000 nexus player from rekordbox running on a Mac, Beat Link (which is used by Beat Link Trigger) has no trouble loading all the metadata from rekordbox.

Can you provide more specific details about your setup so we can see how it differs from mine? Did you see anything in the Beat Link Trigger log file when you were failing to get metadata? I discovered that if the Mac pops up a dialog box asking to confirm whether rekordbox should be allowed to access the network, this can cause Beat Link to believe there is no portmapper running on the rekordbox machine, and BLT will not try to get metadata from rekordbox until you take BLT offline and back online again. (For the offline/online fix to work you must be running the latest preview release, otherwise you need to fully quit and relaunch Beat Link Trigger once rekordbox is up and running and allowed to accept network connections.)

brunchboy commented 6 years ago

Another question for you, @Fra-Ktus, in your C prototype, did you remember to send the correct source-type value when requesting the metadata? Tracks loaded from recordbox show as being loaded from “slot” 04, and you need to request them that way.

Fra-Ktus commented 6 years ago

Thank you so much for testing the feature. My Rekordbox is on a Win7 machine on the network. beat-link-trigger is on my MacBook Pro. Running on a 192.168.1.x network with DHCP. I tried going offline online and it did not help. It's good news that it's working on your side I must have something wrong on my network. The pioneer player sees the metadata but I can't download it in beat-link-trigger or in my prototype. In my code, I set Sr and Tr as I received it from the status packet and I declare my virtual drive as number 4... Will keep you updated...

Fra-Ktus commented 6 years ago

Here is the log from beat-link-trigger:

2 errors: Problem requesting metadata, returning null java.net.SocketTimeoutException: connect timed out

2018-Sep-02 11:19:07 INFO [beat-link-trigger.logs:29] - Beat Link Trigger version 0.3.8 built Sun Jun 17 18:25:50 CDT 2018 2018-Sep-02 11:19:07 INFO [beat-link-trigger.logs:30] - Log files can grow to 200000 bytes, with 5 backlog files. 2018-Sep-02 11:19:07 INFO [beat-link-trigger.core:57] - Beat Link Trigger starting. 2018-Sep-02 11:19:08 INFO [beat-link-trigger.core:18] - Trying to go online, Request Track Metadata? true 2018-Sep-02 11:19:12 INFO [beat-link-trigger.core:29] - Went online, using player number 2 2018-Sep-02 11:19:14 INFO [org.deepsymmetry.beatlink.data.MetadataFinder:1528] - Reporting media mounted in SlotReference[player:1, slot:USB_SLOT] 2018-Sep-02 11:20:53 ERROR [org.deepsymmetry.beatlink.data.MetadataFinder:108] - Problem requesting metadata, returning null java.lang.Thread.run Thread.java: 748 org.deepsymmetry.beatlink.data.MetadataFinder$8.run MetadataFinder.java: 1751 org.deepsymmetry.beatlink.data.MetadataFinder.access$800 MetadataFinder.java: 41 org.deepsymmetry.beatlink.data.MetadataFinder.requestMetadataInternal MetadataFinder.java: 106 org.deepsymmetry.beatlink.dbserver.ConnectionManager.invokeWithClientSession ConnectionManager.java: 189 org.deepsymmetry.beatlink.dbserver.ConnectionManager.allocateClient ConnectionManager.java: 119 java.net.Socket.connect Socket.java: 589 java.net.SocksSocketImpl.connect SocksSocketImpl.java: 392 java.net.AbstractPlainSocketImpl.connect AbstractPlainSocketImpl.java: 188 java.net.AbstractPlainSocketImpl.connectToAddress AbstractPlainSocketImpl.java: 206 java.net.AbstractPlainSocketImpl.doConnect AbstractPlainSocketImpl.java: 350 java.net.PlainSocketImpl.socketConnect PlainSocketImpl.java
java.net.SocketTimeoutException: connect timed out

Fra-Ktus commented 6 years ago

Here is a log of my prototype when receiving metadata from the USB stick Send START_PHASE_1_1, time 239036413 Send START_PHASE_1_2, time 239036724 Send START_PHASE_1_3, time 239037037 Send START_PHASE_2_1, time 239037351 Send START_PHASE_2_2, time 239037666 Send START_PHASE_2_3, time 239037982 New track on deck 4 id 7 dr 0 sr 0 tr 0 Dump of received status message: 51 73 70 74 31 57 6D 4A 4F 4C 0A 58 44 4A 2D 37 Qspt1WmJOL.XDJ-7 30 30 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00.............. 04 01 00 F8 01 00 00 00 01 03 01 00 00 00 00 07 ................ 00 00 00 07 00 00 00 05 00 00 00 01 00 00 00 00 ................ 00 00 00 00 00 00 00 13 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 01 00 06 04 00 00 00 00 ................ 00 00 00 04 01 00 00 00 00 00 00 05 31 2E 31 33 ............1.13 00 00 00 00 00 00 00 01 00 A4 FF FE 00 10 8B 43 ...............C 80 00 39 3B 7F FF FF FF 00 00 00 00 00 01 01 FF ..9;............ 00 00 00 AB 01 FF 03 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 ................ 00 10 8B 43 00 00 00 00 00 00 23 1B 0F 01 00 00 ...C......#..... 12 34 56 78 00 00 00 01 01 01 01 01 00 00 00 00 .4Vx............ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 17 00 00 06 30 00 00 03 BE 00 00 00 00 .......0........ 00 00 00 00 .... Open Connection 14 with 192.168.1.174 port 1051 connectWithTimeout to 192.168.1.174 suceed after 0 ms PioneerMetadataStart Dump of sent message: 11 00 00 00 01 ..... PioneerMetadataReceive received 5 PioneerMetadata_CheckConnectOK - Dump of 5 bytes of received message: 11 00 00 00 01 ..... PioneerMetadata_SendQueryContext d=4 Dump of sent message: 11 87 23 49 AE 11 FF FF FF FE 10 00 00 0F 01 14 ..#I............ 00 00 00 0C 06 00 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 04 ..... PioneerMetadataReceive received 42 PioneerMetadata_CheckContextOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 FF FF FF FE 10 40 00 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 00 11 00 00 00 01 .......... PioneerMetadata_MetadataRequest d=4 sr=3 tr=1 track id=7 Dump of sent message: 11 87 23 49 AE 11 00 00 00 01 10 20 02 0F 02 14 ..#I....... .... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 04 01 03 01 11 00 00 00 07 .......... PioneerMetadataReceive received 42 PioneerMetadata_CheckMetaDataRequestOK num metadatas is 11 PioneerMetadata_CheckMetaDataRequestOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 00 00 00 01 10 40 00 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 20 02 11 00 00 00 0B ... ...... PioneerMetadata_RenderMenuRequest d=4 sr=3 tr=1 Dump of sent message: 11 87 23 49 AE 11 00 00 00 02 10 30 00 0F 06 14 ..#I.......0.... 00 00 00 0C 06 06 06 06 06 06 00 00 00 00 00 00 ................ 11 04 01 03 01 11 00 00 00 00 11 00 00 00 0B 11 ................ 00 00 00 00 11 00 00 00 0B 11 00 00 00 00 .............. PioneerMetadataReceive received 42 PioneerMetadata_CheckRenderMenuRequestOK message type 4001 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 40 01 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 01 11 00 00 00 00 .......... PioneerMetadataReceive received 134 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 134 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 01 11 00 00 00 07 11 00 00 00 28 26 ..............(& 00 00 00 14 00 49 00 6E 00 74 00 65 00 6C 00 6C .....I.n.t.e.l.l 00 69 00 67 00 65 00 6E 00 74 00 20 00 57 00 6F .i.g.e.n.t. .W.o 00 72 00 73 00 68 00 69 00 70 00 00 11 00 00 00 .r.s.h.i.p...... 02 26 00 00 00 01 00 00 11 00 00 00 04 11 01 00 .&.............. 01 00 11 00 00 00 05 11 00 00 00 00 11 00 00 01 ................ 00 11 00 00 00 00 ...... PioneerMetadataReceive received 114 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 114 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 01 11 00 00 00 0A 11 00 00 00 14 26 ...............& 00 00 00 0A 00 46 00 75 00 20 00 4D 00 61 00 6E .....F.u. .M.a.n 00 63 00 68 00 75 00 00 11 00 00 00 02 26 00 00 .c.h.u.......&.. 00 01 00 00 11 00 00 00 07 11 00 00 00 00 11 00 ................ 00 00 00 11 00 00 00 00 11 00 00 00 00 11 00 00 ................ 00 00 .. PioneerMetadataReceive received 138 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 138 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 01 11 00 00 00 07 11 00 00 00 2C 26 ..............,& 00 00 00 16 00 43 00 6C 00 6F 00 6E 00 65 00 20 .....C.l.o.n.e. 00 6F 00 66 00 20 00 74 00 68 00 65 00 20 00 55 .o.f. .t.h.e. .U 00 6E 00 69 00 76 00 65 00 72 00 73 00 65 00 00 .n.i.v.e.r.s.e.. 11 00 00 00 02 26 00 00 00 01 00 00 11 00 00 00 .....&.......... 02 11 00 00 00 00 11 00 00 00 00 11 00 00 00 00 ................ 11 00 00 00 00 11 00 00 00 00 .......... PioneerMetadataReceive received 96 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 96 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 00 11 00 00 00 BB 11 00 00 00 02 26 ...............& 00 00 00 01 00 00 11 00 00 00 02 26 00 00 00 01 ...........&.... 00 00 11 00 00 00 0B 11 00 00 00 00 11 00 00 00 ................ 00 11 00 00 00 00 11 00 00 00 00 11 00 00 00 00 ................

PioneerMetadataReceive received 96 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 96 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 00 11 00 00 39 3B 11 00 00 00 02 26 ........9;.....& 00 00 00 01 00 00 11 00 00 00 02 26 00 00 00 01 ...........&.... 00 00 11 00 00 00 0D 11 00 00 00 00 11 00 00 00 ................ 00 11 00 00 00 00 11 00 00 00 00 11 00 00 00 00 ................

PioneerMetadataReceive received 96 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 96 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 00 11 00 00 00 07 11 00 00 00 02 26 ...............& 00 00 00 01 00 00 11 00 00 00 02 26 00 00 00 01 ...........&.... 00 00 11 00 00 00 23 11 00 00 00 00 11 00 00 00 ......#......... 00 11 00 00 00 00 11 00 00 00 00 11 00 00 00 00 ................

PioneerMetadataReceive received 100 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 100 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 01 11 00 00 00 03 11 00 00 00 06 26 ...............& 00 00 00 03 00 44 00 6D 00 00 11 00 00 00 02 26 .....D.m.......& 00 00 00 01 00 00 11 00 00 00 0F 11 00 00 00 00 ................ 11 00 00 00 00 11 00 00 00 00 11 00 00 00 00 11 ................ 00 00 00 00 .... PioneerMetadataReceive received 96 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 96 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 00 11 00 00 00 00 11 00 00 00 02 26 ...............& 00 00 00 01 00 00 11 00 00 00 02 26 00 00 00 01 ...........&.... 00 00 11 00 00 00 0A 11 00 00 00 00 11 00 00 00 ................ 00 11 00 00 00 00 11 00 00 00 00 11 00 00 00 00 ................

PioneerMetadataReceive received 96 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 96 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 00 11 00 00 00 00 11 00 00 00 02 26 ...............& 00 00 00 01 00 00 11 00 00 00 02 26 00 00 00 01 ...........&.... 00 00 11 00 00 00 13 11 00 00 00 00 11 00 00 00 ................ 00 11 00 00 00 00 11 00 00 00 00 11 00 00 00 00 ................

PioneerMetadataReceive received 104 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 104 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 00 11 00 00 00 03 11 00 00 00 0A 26 ...............& 00 00 00 05 00 52 00 6F 00 63 00 6B 00 00 11 00 .....R.o.c.k.... 00 00 02 26 00 00 00 01 00 00 11 00 00 00 06 11 ...&............ 00 00 00 00 11 00 00 00 00 11 00 00 00 00 11 00 ................ 00 00 00 11 00 00 00 00 ........ PioneerMetadataReceive received 116 PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 116 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 00 11 00 00 00 07 11 00 00 00 16 26 ...............& 00 00 00 0B 00 32 00 30 00 31 00 38 00 2D 00 30 .....2.0.1.8.-.0 00 38 00 2D 00 31 00 30 00 00 11 00 00 00 02 26 .8.-.1.0.......& 00 00 00 01 00 00 11 00 00 00 2E 11 00 00 00 00 ................ 11 00 00 00 00 11 00 00 00 00 11 00 00 00 00 11 ................ 00 00 00 00 .... PioneerMetadataReceive received 32 PioneerMetadata_CheckRenderMenuRequestOK message type 4201 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 32 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 42 01 0F 00 14 ..#I.......B.... 00 00 00 0C 00 00 00 00 00 00 00 00 00 00 00 00 ................

PioneerMetadata_BeatGridsRequest d=4 sr=3 tr=1 Dump of sent message: 11 87 23 49 AE 11 00 00 00 03 10 22 04 0F 02 14 ..#I.......".... 00 00 00 0C 06 06 06 00 00 00 00 00 00 00 00 00 ................ 11 04 08 03 01 11 00 00 00 07 .......... PioneerMetadataReceive_ReadData had to iterate 1 times PioneerMetadataReceive received 7389 It took 88 ms to get metadata

Fra-Ktus commented 6 years ago

And here is a log when I can't read metadata from RekordBox I must have done something stupid on my side...

Here in PioneerMetadata_CheckRenderMenuRequestOK I receive 0x4001 but with 0 metadata available...

Send START_PHASE_1_1, time 239168213 Send START_PHASE_1_2, time 239168522 Send START_PHASE_1_3, time 239168835 Send START_PHASE_2_1, time 239169148 Send START_PHASE_2_2, time 239169462 Send START_PHASE_2_3, time 239169777 New track on deck 4 id 10 dr 0 sr 0 tr 0 Dump of received status message: 51 73 70 74 31 57 6D 4A 4F 4C 0A 58 44 4A 2D 37 Qspt1WmJOL.XDJ-7 30 30 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00.............. 04 01 00 F8 01 00 00 00 11 04 01 00 00 00 00 0A ................ 00 00 00 01 00 00 00 02 00 00 00 02 00 00 00 01 ................ 00 00 00 00 00 00 00 03 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 01 00 06 04 00 00 00 00 ................ 00 00 00 04 01 00 00 00 00 00 00 03 31 2E 31 33 ............1.13 00 00 00 00 00 00 00 01 00 E4 FF FA 00 10 6A 7E ..............j~ 80 00 24 54 7F FF FF FF 00 10 6A 7E 00 09 01 FF ..$T......j~.... 00 00 00 0D 01 FF 03 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 ................ 00 10 6A 7E 00 10 6A 7E 00 00 26 35 0F 01 00 00 ..j~..j~..&5.... 12 34 56 78 00 00 00 01 01 01 01 01 00 00 00 00 .4Vx............ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ................ 00 00 00 05 00 00 09 D2 00 00 03 7A 00 00 00 00 ...........z.... 00 00 00 00 .... Open Connection 14 with 192.168.1.174 port 1051 connectWithTimeout to 192.168.1.174 suceed after 1 ms PioneerMetadataStart Dump of sent message: 11 00 00 00 01 ..... PioneerMetadataReceive received 5 PioneerMetadata_CheckConnectOK - Dump of 5 bytes of received message: 11 00 00 00 01 ..... PioneerMetadata_SendQueryContext d=4 Dump of sent message: 11 87 23 49 AE 11 FF FF FF FE 10 00 00 0F 01 14 ..#I............ 00 00 00 0C 06 00 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 04 ..... PioneerMetadataReceive received 42 PioneerMetadata_CheckContextOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 FF FF FF FE 10 40 00 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 00 11 00 00 00 01 .......... PioneerMetadata_MetadataRequest d=4 sr=4 tr=1 track id=10 Dump of sent message: 11 87 23 49 AE 11 00 00 00 01 10 20 02 0F 02 14 ..#I....... .... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 04 01 04 01 11 00 00 00 0A .......... PioneerMetadataReceive received 42 PioneerMetadata_CheckMetaDataRequestOK num metadatas is 0 PioneerMetadata_CheckMetaDataRequestOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 00 00 00 01 10 40 00 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 20 02 11 00 00 00 00 ... ...... PioneerMetadata_RenderMenuRequest d=4 sr=4 tr=1 Dump of sent message: 11 87 23 49 AE 11 00 00 00 02 10 30 00 0F 06 14 ..#I.......0.... 00 00 00 0C 06 06 06 06 06 06 00 00 00 00 00 00 ................ 11 04 01 04 01 11 00 00 00 00 11 00 00 00 00 11 ................ 00 00 00 00 11 00 00 00 00 11 00 00 00 00 .............. PioneerMetadataReceive received 42 PioneerMetadata_CheckRenderMenuRequestOK message type 4001 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 40 01 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 00 11 00 00 00 00 .......... PioneerMetadata_BeatGridsRequest d=4 sr=4 tr=1 Dump of sent message: 11 87 23 49 AE 11 00 00 00 03 10 22 04 0F 02 14 ..#I.......".... 00 00 00 0C 06 06 06 00 00 00 00 00 00 00 00 00 ................ 11 04 08 04 01 11 00 00 00 0A .......... PioneerMetadataReceive_ReadData had to iterate 100 times PioneerMetadataReceive received 52 PioneerMetadata_CheckBeatGridsRequestOK Packet size is wrong: 52 vs 0 - bad delta: 52 It took 164 ms to get metadata

brunchboy commented 6 years ago

I don’t know how to ready your logs, unfortunately, and don’t have time to learn, but the Beat Link Trigger log makes it seem that there might be a firewall preventing it from communicating fully with rekordbox?

Fra-Ktus commented 6 years ago

Not sure about the firewall because it works when tracks come from USB and I see the metadata correctly in the player if I play from USB and Rekordbox. The log from my tool is as close as possible from what is in the paper describing the protocol. Thanks for your help I hope I can post soon the mistake I did, I will try to use a different machine to host Rekordbox and change IP settings...

brunchboy commented 6 years ago

I agree, the fact that the players can get the data makes it not likely a firewall issue. It’s possible you are doing nothing wrong, and there is a subtle difference between what rekordbox on the Mac accepts, and what the Windows version accepts. This is very strange! If you do figure out how to get it to respond, I’d very much appreciate learning about it. At this point if I were investigating, I would use Wireshark to compare the packets from the CDJs to rekordbox with the ones we are sending, and see if there is any difference that might matter.

I’ve spent weeks of my life on such research, unfortunately… perhaps not the healthiest thing! 😉

Fra-Ktus commented 6 years ago

Thanks, I will test with Rekordbox on the Mac and see if it works better, if needed I will fire Wireshark, it's also my friend for many years. I agree it's not healthy to dive into it for many hours but sometimes you have the ha ha moment :-)

Fra-Ktus commented 6 years ago

I can confirm that if Rekorbox runs on macOS then the metadata is downloaded by Beat Link Trigger! In my test Windows 7 or macOS is running on the same hardware. Time to fire Wireshark :-)

Fra-Ktus commented 6 years ago

So I could get it to work! What I did understood wrong is that when a track is coming from Rekordbox you need to query the system running Rekordbox and not the player... That can explain why it does not work with Windows 7 because the port number is different and may be blocked. I will try to reproduce. Side question, there can be several Rekordbox servers on the network, how to know what server to query?

brunchboy commented 6 years ago

That’s good to hear, and yes, you always query the device hosting the database from which the track has been loaded. I have not tested with multiple rekordbox laptops, although I do believe that the players support up to two of them. My expectation is that they will be assigned different device numbers (if I remember correctly, the first one gets device number 0x11 or decimal 17, so probably the second one is 0x12 or decimal 18). I already need to keep a map of device numbers to IP addresses for knowing which player to contact for a given track, the same technique would work here. (The player from which a track was loaded is shown in the status packet of the player playing the track, along with the slot number, track type, and track ID.)

KayPure commented 6 years ago

Hey Guys,

ive got the same issue, since i use rekordbox only in export mode. if i understand correctly, it wont work this way? BLT is working fine, but only the metadata wont be displayed. if i use usb, it works fine.

@Fra-Ktus how did you get this to work?

greetings

brunchboy commented 6 years ago

Yes, if anyone can figure out how to change the queries to work properly with Windows rekordbox, I would greatly appreciate learning that, so I can update dysentery, Beat Link, and Beat Link Trigger.

Fra-Ktus commented 6 years ago

I think the queries are ok because my C prototype works now.

What I see on my side is that the port number of the DB server is not the classical one. It asks me to use port 49184 for the queries, could that be the problem?

What should I use to debug Beat Link Trigger? (C dinosaur coding in C for 20 years here :-) )

Here is a log when Beat Link Trigger try to get the metadata:

2018-Sep-07 10:57:04 INFO [beat-link-trigger.logs:29] - Beat Link Trigger version 0.3.8 built Sun Jun 17 18:25:50 CDT 2018 2018-Sep-07 10:57:04 INFO [beat-link-trigger.logs:30] - Log files can grow to 200000 bytes, with 5 backlog files. 2018-Sep-07 10:57:04 INFO [beat-link-trigger.core:57] - Beat Link Trigger starting. 2018-Sep-07 10:57:05 INFO [beat-link-trigger.core:18] - Trying to go online, Request Track Metadata? true 2018-Sep-07 10:57:10 INFO [beat-link-trigger.core:29] - Went online, using player number 2 2018-Sep-07 10:57:11 INFO [org.deepsymmetry.beatlink.data.MetadataFinder:1528] - Reporting media mounted in SlotReference[player:1, slot:USB_SLOT] 2018-Sep-07 10:57:11 ERROR [org.deepsymmetry.beatlink.data.MetadataFinder:108] - Problem requesting metadata, returning null java.lang.Thread.run Thread.java: 748 org.deepsymmetry.beatlink.data.MetadataFinder$8.run MetadataFinder.java: 1751 org.deepsymmetry.beatlink.data.MetadataFinder.access$800 MetadataFinder.java: 41 org.deepsymmetry.beatlink.data.MetadataFinder.requestMetadataInternal MetadataFinder.java: 106 org.deepsymmetry.beatlink.dbserver.ConnectionManager.invokeWithClientSession ConnectionManager.java: 189 org.deepsymmetry.beatlink.dbserver.ConnectionManager.allocateClient ConnectionManager.java: 121 org.deepsymmetry.beatlink.dbserver.Client. Client.java: 99 org.deepsymmetry.beatlink.dbserver.Client.performSetupExchange Client.java: 120 java.io.IOException: Did not receive message type 0x4000 in response to setup message, got: Message: [transaction: 4294967294, type: 0x0100 (unknown), arg count: 0, arguments: ]

2018-Sep-07 10:57:12 ERROR [org.deepsymmetry.beatlink.data.MetadataFinder:108] - Problem requesting metadata, returning null java.lang.Thread.run Thread.java: 748 org.deepsymmetry.beatlink.data.MetadataFinder$8.run MetadataFinder.java: 1751 org.deepsymmetry.beatlink.data.MetadataFinder.access$800 MetadataFinder.java: 41 org.deepsymmetry.beatlink.data.MetadataFinder.requestMetadataInternal MetadataFinder.java: 106 org.deepsymmetry.beatlink.dbserver.ConnectionManager.invokeWithClientSession ConnectionManager.java: 189 org.deepsymmetry.beatlink.dbserver.ConnectionManager.allocateClient ConnectionManager.java: 121 org.deepsymmetry.beatlink.dbserver.Client. Client.java: 99 org.deepsymmetry.beatlink.dbserver.Client.performSetupExchange Client.java: 120 java.io.IOException: Did not receive message type 0x4000 in response to setup message, got: Message: [transaction: 4294967294, type: 0x0100 (unknown), arg count: 0, arguments: ]

KayPure commented 6 years ago

Did ur prototype work on win machines? Would love to test it, because i really need an option to display played tracks on a video stream.

Fra-Ktus commented 6 years ago

I am not working on a similar tool to Beat Link Trigger but just a custom utility to send commands to a media server.

brunchboy commented 6 years ago

Beat Link Trigger works fine on Windows.

brunchboy commented 6 years ago

@Fra-Ktus to debug this you’ll need to install a Java SDK, and Clojure, and choose a Clojure development environment (the two big choices are CIDER in Emacs, and Cursive in IntelliJ IDEA). But that will be a big learning curve; Clojure is a Lisp, and Lisp is vastly different than C. And there is the whole Java class library infrastructure underneath it. I feel like the journey will be rewarding in the end, but I know it will be very confusing and seem slow going at the start, so I am not going to try to push you to do it.

That said, we already have a bit of key information in the stack trace you quoted:

java.io.IOException: Did not receive message type 0x4000 in response to setup message, got:
 Message: [transaction: 4294967294, type: 0x0100 (unknown), arg count: 0, arguments: ]

This tells me that when BLT sends the database context setup message described in Figure 21 of the dysentery Packet Analysis document, we do not get back the packet we expect (Figure 22), but one whose type field is 0100. So if you can capture the corresponding traffic when a CDJ succeeds at setting up its remotedb session, these are the packets to focus on, see what is different, and if the CDJ is getting a similar packet back, how it proceeds past that point, and how that differs from what I show in the paper. You should be able to examine that without having to worry about learning Clojure or Java. And anything you can discover would be helpful.

KayPure commented 6 years ago

Yeah, it works on windows. But no metadata is displayed when i link my xdj 1000mk2's to rekordbox. If i use ubs sticks, it works fine.

brunchboy commented 6 years ago

Ah, I see, that’s the same problem Fra-Ktus is trying to investigate. I have to admit that people using rekordbox has not been a primary use case I worry about, because I can’t run rekordbox and Beat Link Trigger at the same time, since I usually only have one computer. Just to confirm, you are running rekordbox on a different machine than you are running BLT on, @KayPure ?

KayPure commented 6 years ago

Yes, i do. Ive got a Rekordbox DB on a Laptop and run BLT on a PC. If i run a Mac VM on this PC, the metadata displays correctly. Just like Fra-Ktus said, there must be some issues with windows.

Fra-Ktus commented 6 years ago

Did retest this morning. Applied latest Rekorbox update 5.4.0, running on Windows 7 Beat Link Trigger run on macOS

Beat Link Trigger has the same error 2018-Sep-08 11:37:53 INFO [org.deepsymmetry.beatlink.data.MetadataFinder:1528] - Reporting media mounted in SlotReference[player:1, slot:USB_SLOT] 2018-Sep-08 11:37:53 ERROR [org.deepsymmetry.beatlink.data.MetadataFinder:108] - Problem requesting metadata, returning null java.lang.Thread.run Thread.java: 748 org.deepsymmetry.beatlink.data.MetadataFinder$8.run MetadataFinder.java: 1751 org.deepsymmetry.beatlink.data.MetadataFinder.access$800 MetadataFinder.java: 41 org.deepsymmetry.beatlink.data.MetadataFinder.requestMetadataInternal MetadataFinder.java: 106 org.deepsymmetry.beatlink.dbserver.ConnectionManager.invokeWithClientSession ConnectionManager.java: 189 org.deepsymmetry.beatlink.dbserver.ConnectionManager.allocateClient ConnectionManager.java: 121 org.deepsymmetry.beatlink.dbserver.Client. Client.java: 99 org.deepsymmetry.beatlink.dbserver.Client.performSetupExchange Client.java: 120 java.io.IOException: Did not receive message type 0x4000 in response to setup message, got: Message: [transaction: 4294967294, type: 0x0100 (unknown), arg count: 0, arguments:

Here is the trace from my C prototype that works when running in the same config. I run my app on macOS, Rekorbox on Win7.

I cut the trace to try to make it more readable...

Get port number: PioneerMetadataStart Dump of sent message: 00 00 00 0F 52 65 6D 6F 74 65 44 42 53 65 72 76 ....RemoteDBServ 65 72 00 er. PioneerMetadataReceive received 2 PioneerMetadata_CheckAskPortOK - Dump of 2 bytes of received message: C0 1E .. Open Connection 14 with 192.168.1.176 port 49182

Open connection and the first handshake:

PioneerMetadata_OpenDBServer on port 49182 Dump of sent message: 11 00 00 00 01 ..... PioneerMetadataReceive received 5 PioneerMetadata_CheckConnectOK - Dump of 5 bytes of received message: 11 00 00 00 01 .....

Query Context:

PioneerMetadata_SendQueryContext d=4 Dump of sent message: 11 87 23 49 AE 11 FF FF FF FE 10 00 00 0F 01 14 ..#I............ 00 00 00 0C 06 00 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 04 ..... PioneerMetadataReceive received 42 PioneerMetadata_CheckContextOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 FF FF FF FE 10 40 00 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 00 11 00 00 00 11 ..........

Metadata request:

PioneerMetadata_MetadataRequest d=4 sr=4 tr=1 track id=10 Dump of sent message: 11 87 23 49 AE 11 00 00 00 01 10 20 02 0F 02 14 ..#I....... .... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 04 01 04 01 11 00 00 00 0A .......... PioneerMetadataReceive received 42 PioneerMetadata_CheckMetaDataRequestOK num metadatas is 10 PioneerMetadata_CheckMetaDataRequestOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 00 00 00 01 10 40 00 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 20 02 11 00 00 00 0A ... ......

Render menu request:

PioneerMetadata_RenderMenuRequest d=4 sr=4 tr=1 Dump of sent message: 11 87 23 49 AE 11 00 00 00 02 10 30 00 0F 06 14 ..#I.......0.... 00 00 00 0C 06 06 06 06 06 06 00 00 00 00 00 00 ................ 11 04 01 04 01 11 00 00 00 00 11 00 00 00 0A 11 ................ 00 00 00 00 11 00 00 00 0A 11 00 00 00 00 .............. PioneerMetadataReceive received 42 PioneerMetadata_CheckRenderMenuRequestOK message type 4001 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 42 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 40 01 0F 02 14 ..#I.......@.... 00 00 00 0C 06 06 00 00 00 00 00 00 00 00 00 00 ................ 11 00 00 00 01 11 00 00 00 00 .......... PioneerMetadataReceive received 144

And then the metadata are comming:

PioneerMetadata_CheckRenderMenuRequestOK message type 4101 PioneerMetadata_CheckRenderMenuRequestOK - Dump of 144 bytes of received message: 11 87 23 49 AE 11 00 00 00 02 10 41 01 0F 0C 14 ..#I.......A.... 00 00 00 0C 06 06 06 02 06 02 06 06 06 06 06 06 ................ 11 00 00 00 01 11 00 00 00 0A 11 00 00 00 32 26 ..............2& 00 00 00 19 00 46 00 61 00 6C 00 6C 00 69 00 6E .....F.a.l.l.i.n 00 67 00 20 00 49 00 6E 00 73 00 69 00 64 00 65 .g. .I.n.s.i.d.e 00 20 00 54 00 68 00 65 00 20 00 42 00 6C 00 61 . .T.h.e. .B.l.a 00 63 00 6B 00 00 11 00 00 00 02 26 00 00 00 01 .c.k.......&.... 00 00 11 00 00 00 04 11 00 00 01 00 11 00 00 00 ................ 01 11 00 00 00 00 11 00 00 01 00 11 00 00 00 00 ................

I would like to help more but it looks steep to install all the tools for me... I am used to XCode and Visual Studio...

brunchboy commented 6 years ago

Thanks for this, and I agree it would be steep (and not a great use of your time) to have you try to install Clojure and a development environment. However, would it be possible for you to share a Wireshark capture of the packets when Beat Link Trigger fails? It is the difference between those and the ones you quoted above that I need to figure out what is going on.

brunchboy commented 6 years ago

Oh! I just had a thought. In Beat Link Trigger, what player number are you? If you are using the latest 0.4.0 preview release, it will show that in the Network menu after the Online? checkbox.

KayPure commented 6 years ago

For me, it is 3. We are Player 3

My Xdj´s are Set to 1 and 2

brunchboy commented 6 years ago

That should be fine. I am baffled as to what might be going on! Is your C prototype sending status packets in addition to device-announcement packets, by any chance?

brunchboy commented 6 years ago

Interesting, I do see one difference from what I expect, there are only ten metadata items coming back from rekordbox, while I see eleven usually. But my code should handle that.

But I also see something that may be different between Beat Link and your C prototype: Beat Link asks for the cue list for the track in between obtaining the metadata item count and asking for the actual items, and it does not look like your C prototype is doing that. Am I correct? If so, I will build a new preview build of Beat Link Trigger with a modified version of Beat Link that gets the metadata items before asking for the cue list, and ask you to test that.

brunchboy commented 6 years ago

I have uploaded a new version of the 0-4-0 preview to the releases page. This new vesion loads the metadata items right after setting up the metadata menu request, instead of asking for the cue list in between those two actions, in case that is what is causing problems for Windows rekordbox. Please give it a try when you have a chance, @Fra-Ktus and @KayPure.

KayPure commented 6 years ago

Unfortunately still no metadata.

BLT shows: Player Status, Master or not, Track ID, BPM, Beat

Player Status shows: Bar Steps Player active or not (green or grey player symbol) Tempo Range BPM

Thats all. No Waveforms, no Wavedetail, no Time, no Meta :(

1 2

brunchboy commented 6 years ago

Rats, but thanks for trying. The log file (File ▶️ Open Logs Folder) would show more about what went wrong, but I expect it will show exactly what @Fra-Ktus posted above. We will have to wait until he can compare packet captures.

Fra-Ktus commented 6 years ago

Can I mail you a wireshark trace, will try to do that today but I need to go to the office to do that...

Fra-Ktus commented 6 years ago

I tried the latest preview 0-4-0 and unfortunately, it's the same. My prototype is announcing player 4 and is also sending the keep-alive message, just as described in the protocol documentation. Yes, I just ask metadata, then the beat grid to reconstruct the time, not the playlist.

brunchboy commented 6 years ago

Oops, I did not mean for that commit to auto-close the issue! I want to wait to hear from you both. I have updated the preview release with a new version of the Beat Link library that assembles all dbserver messages into a single buffer and sends them as a single operation, which should hopefully fix the issue that @Fra-Ktus helped me find with his perfect Wireshark captures. @KayPure could you try this latest one too, when you have a chance?

It looks like the problem is that Windows rekordbox is getting confused when a dbserver message is split into multiple packets on the network. Code is supposed to be written in a way that protects against that, because networks sometimes do that, and my own code is careful to handle that properly (and so is rekordbox on the Mac, and on the CDJs themselves), but apparently not the Windows implementation of rekordbox. It replies to the first tiny part of the message, then closes the connection when the rest of the message arrives.

The messages should hopefully not get split any more; at least, I have done everything I can within programming to avoid that. I can’t control what happens once they are on the network, of course.

brunchboy commented 6 years ago

And one interesting point about all this: We had to go to the level of an actual network traffic capture to figure this out, because the software level log files were not showing where messages were being split into multiple packets; at the software level we just can’t tell. Sometimes we need these low level tools. I am very thankful you were able to use them.

Fra-Ktus commented 6 years ago

I happy to report that the latest preview version can have the metadata from Rekorbox running on windows 7! Congratulations! Tell me if you want to see a Wireshark trace for you to confirm that it works as you expect.

KayPure commented 6 years ago

Agree! Windows 10 Machines. Worked like a charme!!!! U guys are awesome ;) Thanks!

Another question, not related to this issue: Is there any way to push the metadata out of the program into, lets say, an html file? i think thats quite not easy huh?

brunchboy commented 6 years ago

Thanks so much for confirming this, both of you! I will close the issue.

@Fra-Ktus I appreciate the offer, but my change was simple enough that I can pretty much visualize what the capture will look like, so there is no need for you to spend the time. Save it for when we next run into a problem! 😁

And @KayPure you could certainly write an expression that would do that. However, you should take a look at the Playlist Writer which is already there, Network ➡️ Show Playlist Writer. That creates a CSV file showing information about all the tracks which have been played. You could feed that as input to a program that would reformat it as HTML. Or you could look at the source code of that as an example of how to write code that would create it as HTML in the first place. Without knowing more details about your intended use case, that’s the best advice I can offer.

KayPure commented 6 years ago

@brunchboy thanks for the infos. Will check the playlist writer. I want to bring the metadata in obs. I could do this with an windowcapture, but that looks not good. I found an obs overlay here on github, but that wont work. I'll try to figure something out with the code, thanks.

brunchboy commented 6 years ago

What is obs? If you are looking for real-time metadata information to integrate with another program, then you probably want to drop down a level and use the same Beat Link library that Beat Link Trigger uses to get it.

KayPure commented 6 years ago

Open broadcaster Studio. To stream video and audio to e.g. Twitch. Yeah, i will take a look and figure something out.

brunchboy commented 6 years ago

Thanks for the context! Interesting, I know a number of other people looking at incorporating my discoveries into Twitch streams. Once I finish my current big project which is keeping me from getting enough sleep, I may take a look at writing an example expression for BLT that would offer an embedded web server that could serve an HTML page containing current metadata information, and that you could customize with a template. Would OBS be able to fetch and use that? Alternately, it could write it to a file if that would be more useful.

KayPure commented 6 years ago

That sounds great. Obs can fetch HTML as a single file or as an source from a browser. So if there is an webserver, obs can fetch the URL with its browser source plugin and display the content.

As i said, there is an project here on github called "prolink overlay" but for me, it wont work. Perhaps that inspires you. It seem that the author isnt active anymore.

brunchboy commented 6 years ago

@KayPure, I was going to tell you that the author of the prolink overlay you mentioned is someone I talk to, because he has been building his software with the help of my dysentery research project, and that you should open an issue in his repository to see if he can help you get it working… but I see you already have! 😄 I expect that as soon as he has some time, he will help you. He just recently started incorporating some of my latest discoveries in his main project related to this.