Unreal-Dan / RekordBoxSongExporter

A hack for Rekordbox on Windows x64 to export track information for realtime integration with OBS
59 stars 6 forks source link

[Question] Real-time BPM update polling rate #26

Open Wietjuh opened 7 months ago

Wietjuh commented 7 months ago

Is there a way we could limit the rt bpm function (or in call frequency/time)

I play with timecodes (vinyl) and vinyl flutters.

So, I would love a function that wouldn't update the file x times per second but once a 15/30 seconds (preferably before it sends it over the network to my other PC)

Maybe its an easy config for me locally, maybe more people would like to be able to define it.

Also I only need whole bpm info, not the 0,01 accuracy, but that can be filtered out at my end.

/edit: And I forgot to make it a readable title, please delete if needed 😂

Unreal-Dan commented 7 months ago

I see so you just want to slow down the rt bpm function so that it updates less often? Is that what I'm understanding?

You're saying you'd like it to run once every 15 seconds, and 30 seconds?

Are you comfortable making modifications to the module and rebuilding it yourself? All that is needed is visual studio there's no extra requirements. If you're not comfortable then I can put together some changes for you but it would probably be faster for you to do it.

This function here is responsible for queueing up a bpm update into the logging system:

https://github.com/Unreal-Dan/RekordBoxSongExporter/blob/master/Module/OlvcCallback.cpp#L146

You could add a simple time check inside that function, something like this:

#include <chrono>

// callback for when the bpm changes on a deck
static void bpm_changed(uint32_t deck_idx, uint32_t old_bpm, uint32_t new_bpm)
{
    // keep track of 8 last call times, one for each possible deck
    static std::chrono::steady_clock::time_point lastCallTime[8];

    // adjust period to be some measure of time in milliseconds to ratelimit to
    // 1000 / 15 = 15 times per second
    const auto period = std::chrono::milliseconds(1000 / 15);

    // grab curtime and calculate elapsed since last update on this deck
    auto now = std::chrono::steady_clock::now();
    auto elapsed = now - lastCallTime[deck_idx];

    // if we haven't waited long enough to queue the bpm change, then just discard this change
    if (elapsed < period) {
        // Discard the call as it is within the rate limit period
        return;
    }

    // otherwise update the last call time and proceed with queuing the bpm change
    lastCallTime[deck_idx] = now;

    // lookup a player for the deck
    djplayer_uiplayer *player = lookup_player(deck_idx);
    // make sure the player has a track loaded on it
    if (!player || !player->getTrackBrowserID()) {
        return;
    }
    info("BPM change deck %u: %.2f -> %.2f", deck_idx, old_bpm / 100.0, new_bpm / 100.0);
    // Push a message to the output file writer indicating the deck changed bpm
    push_deck_update(deck_idx, UPDATE_TYPE_BPM);
}

I honestly don't know if this will work the way we want, but I'm quite certain it should.

You're going to want to configure the code and rebuild the module how you want, here's a changeset that demonstrates adding this code to the program.

https://github.com/Unreal-Dan/RekordBoxSongExporter/commit/4666d3d70e14d2d1c4fbe514a7db7857df9acab5

Let me know how this goes and if you need any help

Wietjuh commented 7 months ago

as i read this i think i can handle it.

My goal is, because my decks flutter .3 bpm while playing a track, to round of the bpm value to a whole and only push it to the text file every 15 seconds or so as that would be enough updates for my bot.

It rounding of can be done very easy at my end by just reading whole values with my bot.

They way you implemented as i can see is it pushes the data (rekordbox) to the hook (rbse) anyway, than we set a timer to check if enough time has past before we push to textfiles.

Its different than what i had in mind and maybe a tad compicated for my skil level to approach it like this, but why not. Ill try to build it tomorrow. Gonna be a first for me tho.

TBH if i can mod a config file in the program folder i'd be very happy. I got time to wait and im willing to cover the costs if you do it for me.

So maybe let me know when u have a little time to drop it in yourself. In the mean time ill try installing visual tomorrow and see what i can do. Kinda curious too.

I think with github i'd also be able to do more with it. But as said, my skil level with code is abominal. I can read, but writing is a whole new thing..

Kind regards and thanks for the quick reply

ill keep you updated

ps The whole reason i want this is because it sens hundreds of updates over the network and it even showed up in my logs as one of the most intens processes on my network. (if you dont count the bandwith but the amount of packages)

erikrichardlarson commented 7 months ago

Maybe if you just round bpm and last_bpm to the nearest integer then the bpm_change will trigger as much as you'd expect? https://github.com/Unreal-Dan/RekordBoxSongExporter/blob/25e897a0523713ce2ef47bb692b9119b0d4414a1/Module/OlvcCallback.cpp#L112

Unreal-Dan commented 7 months ago

Maybe if you just round bpm and last_bpm to the nearest integer then the bpm_change will trigger as much as you'd expect?

https://github.com/Unreal-Dan/RekordBoxSongExporter/blob/25e897a0523713ce2ef47bb692b9119b0d4414a1/Module/OlvcCallback.cpp#L112

This is right, that would be an optimal way to do it in the engine. The bpm is an integer that represents the real bpm * 100, so 128.50 is 12850 in that integer, if I recall correctly. If you do something like:

bpm -= (bpm % 100);

This will round the bpm down to the nearest whole value, so something like 12850 will round down to 12800.

If you do this before storing the old_bpm[deck_idx] = bpm then it will always only track the whole-bpm changes.

Perhaps I can add a separate tag that only tracks the whole number bpms this way.

Wietjuh commented 7 months ago

That last suggestion would be the sollution I think.

Please let me know if you need more info, but I think we are on the right track here!

Wietjuh commented 7 months ago

I tried with VS last week, but I can tell you it's not a program that I understand in a day. So I'd rather have you make the change or get me a test build with the bpm this way. I'm sorry.

erikrichardlarson commented 7 months ago

Will create a new build right now and send you way

erikrichardlarson commented 7 months ago

@Unreal-Dan Looks like capstone is missing some header files which is preventing the build from being successful, are you able to include those header files in the repo here?

erikrichardlarson commented 7 months ago

@Wietjuh Was able to build the project, try out this build, uses the bpm change above. @Unreal-Dan Looks like an older version of capstone was needed, built 4.0.1 and it worked. RekordBoxSongExporter.zip

Wietjuh commented 7 months ago

I'll try today @erikrichardlarson, thanks for the build.

I should really get to know VS once. Made some plans to be able to use my stream desk as a desktop, so it might come one day.

BTW, did I read the code correctly it updates 15 times a second? And only after it change a full bpm it'll update the txt file.

Wietjuh commented 7 months ago

So, I'm really sorry but it still pumps out waaaay to many changes (like in the 100s)

Maybe just update once every couple of seconds (my preference would be 10 seconds) and I'd really would like the whole bpm number as a result if possible.

But we are getting there, this wasn't a strain on my network as it used to be. And I'm sure it will be amazing to see it work out caus the test I did with the lighting and all the artwork on stream all synced up with the music went really well

erikrichardlarson commented 7 months ago

No worries, can send over another update. Can update to send out bpm update if last update was more than 10 seconds ago or if there's a large change in BPM, maybe a change of 5 or more? Just want to be sure that larger updates aren't slow / missed in real-time.

Unreal-Dan commented 7 months ago

Sorry guys, been busy, I'll put in some time tonight and see if I can put together a new feature that let's him control the rate of updates

Wietjuh commented 7 months ago

U both are the best! If we can get this to work I can set so much automation based on this info, the MIDI over network doesnt work with most streamer bots. I will show some results if you like when this all goes live!

As said, if we can do the round off to full bpm and if I can set the rate that would be amazing (even if it would be in a config file or whatever)

Really getting excited for the possibilities

erikrichardlarson commented 7 months ago

@Wietjuh Try this build, this actually floors the BPM, and should result in less unnecessary updates. @Unreal-Dan Had to update the get_deck_bpm method in the output_file class so the bpm is floored when written out as well. RekordBoxSongExporter.zip

Unreal-Dan commented 7 months ago

@Wietjuh Try this build, this actually floors the BPM, and should result in less unnecessary updates. @Unreal-Dan Had to update the get_deck_bpm method in the output_file class so the bpm is floored when written out as well. RekordBoxSongExporter.zip

ahahaha yeah it's not quite so elegant in that the bpm change triggers an update event to the output file system, then that system fetches the bpm again and converts it to a string. Definitely not the best way to approach that, it was an evolution of discoveries combined with just enough laziness that resulted in that culmination.

What I'm thinking is I'm going to add a tag that rounds the bpm (floor of bpm + 0.5) and only triggers updates when the rounded valued changes.

Then on top of that, I will add a secret configuration value to the config file that allows the server update rate to be capped at a customizable rate, extra realtime updates outside of this rate will be dropped. So regular non-realtime tags will never be dropped, but realtime tags like bpm etc will be dropped if they are sent too often.

So he will use a new tag that floors the bpm while the old tag will still exist for people who want the added resolution. Then anybody can set their max update rate in the config file, pending some kind of UI update to support the input.

Wietjuh commented 7 months ago

I completely understand, I would also solve these problems as I go, and my setup such a specific use case that I understand the chose to do a new tag for this. I'll test the build this afternoon btw.

To give you a little insight to it, the updates that I send are to for example a Strobe overlay in obs that does 2* the bpm frequency and if it gets updated during the Strobe fx input bpm cycle (150ms for writing the value to obs ) it crashes. That happened a lot. After the previous test that didn't happen at all. Also I have an on screen bpm display textGDI in obs that doesn't like updating more than once or twice a second.

So it's working around the shortcomings of other software. Therefor I'm really happy with these additions

Unreal-Dan commented 7 months ago

interesting, which program crashes? OBS?

Wietjuh commented 7 months ago

My bot, because it writes the value to obs and while it sends it it gets an update, so the bot crashes.

Haven't had it since the second 'update'

erikrichardlarson commented 7 months ago

Nice @Wietjuh so the latest build prevented the excessive updates on your end it seems?

Wietjuh commented 7 months ago

It didn't freeze or hang anymore, but I must see what Dan's update is up to, seems his approach makes I can tweak it a little bit more.

Unreal-Dan commented 7 months ago

okay I have for you an experimental version, to be honest I'm both lazy and busy so I've implemented what I think should solve the problem but I haven't really bothered to test it much.

I have implemented two things:

  1. new tags: %rt_deck1_rounded_bpm% %rt_deck2_rounded_bpm% %rt_deck3_rounded_bpm% %rt_deck4_rounded_bpm% %rt_master_rounded_bpm%

  2. Secret config in the config.ini: update_rate=

You can set the update_rate to any integer value in the range 1-4billion and it should cap the server updates to that amount per second.

I don't know what a good value to pick is, I'd love if you could find out some good values that would make sense.

RekordBoxSongExporter3.8.4-TEST.zip

Unreal-Dan commented 6 months ago

bump @Wietjuh sorry for the delays on my end, give that test build a try next time you can :)

Wietjuh commented 6 months ago

I've already tried that build, it works! Haven't played around with the rate limit to much, but at least it behaves like I expected.

Also running on the latest RB version.

Also are there any plans to make it also usable through ProDJ link

Guili123 commented 6 months ago

Also running on the latest RB version.

Does is tworks with 6.8.2? How you do this?

EDIT: Found it....

Wietjuh commented 6 months ago

I can assure you it works, but it's not really tested, I just keep a copy of 6.7.5 available/installed for when its not working and I need to a/b test if it's RB that's not a problem.

Wietjuh commented 6 months ago

For now I think we got where we wanted to @Unreal-Dan so we could close this one I guess. Or do you want to keep it open untill it's pushed to master?

Guili123 commented 6 months ago

I used it with 6.8.1, just changed the RB path, started it and worked perfectly!

Unreal-Dan commented 6 months ago

Are you guys able to confirm whether the update_rate= option keeps the logging working as intended while reducing network traffic?

If the new features are working good I'll post it as a public release, just want to be completely certain first.

Wietjuh commented 6 months ago

Are you guys able to confirm whether the update_rate= option keeps the logging working as intended while reducing network traffic?

If the new features are working good I'll post it as a public release, just want to be completely certain first.

I got it on set to 1, but i do see more updates than once a second when i am on 139.9/140.1 (with a little bit of flutter) it keeps switching, havent seen nearly as much traffic, but if i where to guess its still more than once a second. Ive added the rate limit in the config file as you suggested.

Will test this wednesday more acuratly.

Wietjuh commented 6 months ago

After testing and setting the limit to the update rate I can confirm normal operations on rb 3.8.2, so I conclude this is working as expected.

Every once in a while it still throws me an unexpected error when the first track loads up, but that has been an ongoing issue non related to this I hope. (the same behaviour on 3.7.5, in case you wondered. And I've tested without the bpm stuff and it still happens, but not predictably)

BTW if you need debug logs from RB when that happens let me know, Ill save them anyway

Unreal-Dan commented 6 months ago

Only the rt_ (realtime) updates will be ratelimited by the update_rate.

So other updates like regular %artist% or %track% do not obey the update_rate.

My thought process is: those updates like %track% don't happen often and you never want to skip them, but the rt_ updates can be very frequent and it's okay to drop them sometimes.

So I wrote the logic to only ratelimit the rt_ updates, so if you have the update_rate set to 1 then you should only see 1 rt update per second -- but you'll still see the other non-rt updates happening as normal.

What do you think about this approach? I'm happy to take input and revise this more before releasing.

Also, yeah, if you're still crashing then could you send me the dmp file? Make sure it's produced by the most recent TEST version I've posted above.

I thought I fixed that but it seems maybe I've still overlooked something.

Wietjuh commented 6 months ago

The ratelimit is good this way, it doesn't need to wait with the other info, it's just the livebpm.

Ill test if I can replicate the RB crash. I've been in a car accident yesterday so I got some stuff to arrange today, I'll test after this weekend

Unreal-Dan commented 6 months ago

oh jeez hope you're doing okay, take it easy

Wietjuh commented 6 months ago

Yeah, we are OK, we had a good car and it wasn't that hard, but the car is total I guess 😂