Closed jtackaberry closed 2 years ago
StephanRoemer
commented
4 years ago Hey Jason, just read thru it and I really like the idea. As for "Create a project-specific materialized reabank": you say it's a bit ugly and will slowdown project loading. But... wouldn't this only be the case for the first loading? I mean, we would be able to save the project and then everything is adjusted to our machine. I don't see a huge problem there!
StephanRoemer
commented
4 years ago Hmm, now that I think of it... I think I would prefer solution 1. Maybe a popup to inform the user that the project will be altered should be shown? Like: "please make a backup of this project, before you apply the changes.
jtackaberry
commented
4 years ago you say it's a bit ugly and will slowdown project loading. But... wouldn't this only be the case for the first loading?
The idea is that Reaticulate would generate the project-specific reabank (and updating Reaper's global configuration to point to it) each time the project is opened or switched to when you have multiple tabs.
I've been working this idea more and I think I have the performance down to acceptable levels. When you open a project (or switch to a subproject) where the MIDI editor is open, it adds about 100ms to the project load time. If MIDI editor is closed, it's about 30ms. (The reason is that I need to do some extra beat-Reaper-into-submission stuff when the editor is open for it to notice the reabank has changed.) This is often dwarfed by other work that's done when you switch (sub)projects, so it's not a significant overhead based on percentages.
That's on my system anyway, although my processor (Threadripper 2950X) is a bit run-of-the-mill when it comes to single core performance so I think it should be reasonably representative.
So I think that's not too bad, and per-project reabank files actually is much cleaner in terms of design.
Hmm, now that I think of it... I think I would prefer solution 1.
Interesting. Can you elaborate on why that is?
Thanks for giving this a read over. :)
StephanRoemer
commented
4 years ago You're welcome, took me a bit more time to get back to this, sorry!
If I get this right, then solution would be some kind of wrapper working in realtime, right?
Couldn't we actually have both solutions? Solution 1 for if we are definitely sure that the project won't be sent back to the original user and Solution 2 for the case, that other people will work on it.
IIRC, you mentioned on the cockus forums, that you might add a check that maps can be updated, when a project is loaded. Like: a different version of the map has been found on your computer, would you like to apply it and replace the existing one? Basically, this would already be solution 1, right? Just in a different context.
The reason why I would be in favour of solution 1: if I know, I'm the only person working on that project file, I would like to make the changes permanent.
jtackaberry
commented
4 years ago A goal is that this is as transparent to the user as it can be. As much as possible, things should always Just Work without the need to bother the user.
This is why I'm favoring option 2, because it works in both cases, whether local-only or loading projects that were saved on other systems. The main stumbling block there was the performance concerns, but I think I have those down to manageable levels.
(It's not just on project load or switching subprojects, but any time the current project's Reabank needs to be generated and Reaper suitably kicked to notice it. Another scenario is adding a new bank to a track for the first time, which means the bank needs to be added to the Reabank file Reaper is currently using.)
There are some more optimizations I can imagine to further reduce the lag in this case. I really want to avoid Reaticulate needing to start mangling MIDI events in projects. I feel like that opens the door for a lot of things to go wrong.
If your preference for option 1 was just purely based on performance concerns, I think I've got that worked out. Did you have some other reasons for preferring to alter the program changes in the project?
IIRC, you mentioned on the cockus forums, that you might add a check that maps can be updated, when a project is loaded. Like: a different version of the map has been found on your computer, would you like to apply it and replace the existing one? Basically, this would already be solution 1, right? Just in a different context.
Hm, well I think that scenario is agnostic to either of these option.
The two options deal with the case of how to handle a different MSB/LSB for a bank in the project versus the local system-wide bank. Once Reaticulate assigns the MSB/LSB for imported banks, it can ensure things are neatly consistent on the local system. So this is really around importing projects that were created on a system (or Reaper installation) that has a different idea of which MSB/LSB belongs to which bank.
If I'm importing a new version of a bank, Reaticulate can just reassign it the same MSB/LSB it had before. So neither option 1 nor option 2 is relevant there, I think.
StephanRoemer
commented
4 years ago Okay, thanks for that detailed insight! Also for the bank updating explantion, makes a lot of sense to me.
Yes, I can see why you prefer solution 1, especially in terms of things going wrong when re-writing the PCs of multiple items. The only other reason why I would prefer solution 2 is OCD. But you can safely ignore that one, LOL 😄
Go for solution 1!
jtackaberry
commented
4 years ago I can respect OCD here, I guess I just don't know why option 2 is friendlier to your OCD. What do you find triggering about option 1? :)
StephanRoemer
commented
4 years ago Haha! Thanks for that! 😄
It's the fact, that nothing needs to be "converted on the fly". BUT, you might have way more insights in the anatomy than me. So, I fully trust you on in that regard!
jtackaberry
commented
4 years ago Ah, I see. Yeah, we're comparing a less invasive, collaboration-compatible local-only action you do quite frequently (regenerating the Reabank file Reaper points to), with an fairly invasive mutation on the project you do once (rewrite the PCs in the project).
The former should be transparent to the user, while the latter definitely isn't. If the overhead of that frequent action is sufficiently low, I think that's a good enough reason to pursue it.
So that's what I'll be working on now. I may yet change my mind once I see how things work in practice, who knows. :)
Great to bounce these ideas off you!
StephanRoemer
commented
4 years ago Yes, exactly, you hit the nail 😄
Yeah, that's the only downside with coding. Usually you cannot plan the practical part of the code. You have to code it first in order to see what works best. Anyway, totally looking forward to whatever you come up with!
And I concur, it's great to exchange ideas/views. Thanks for that, Jason!
jtackaberry
commented
2 years ago Implemented in 0.5.0-pre1
Motivation
Anticipating a need for users to be able to share banks without a centralized IANA-like bank numbering authority, we should abstract the MSB/LSB detail from the definition of Reaticulate banks. Otherwise, conflicts resulting from user-chosen MSB/LSB are inevitable, requiring users to pick their own non-conflicting values, and detracting from the Just Works goal of Reaticulate.
Overview
Instead, Reaticulate banks will identify themselves via a uuid. If the bank editor produces a legitimate 128-bit uuid with sufficient entropy, it will be improbable for uuids to collide, allowing the bank editor to use user-generated uuids for redistribution.
The goal here is to make MSB/LSB as unimportant as possible. Of course, MSB/LSB will need to ultimately be used. Reaticulate itself can program the RFX according to the uuid -- it's otherwise entirely agnostic about MSB/LSB relying only on program numbers (except for articulation feedback to control surface (see later)) -- but Reaper's MIDI items will contain program events specific to MSB/LSB that need to resolve to the right bank in the materialized reabank file.
The idea is that some MSB/LSB will be generated on the user's system when a bank is imported via the bank editor (which has the effect of adding the bank to the user's reabank file), or when the user manually adds a bank lacking an MSB/LSB and triggers the reabank refresh action.
When this occurs:
Data/Reaticulate-tmp\d+.reabank)SetExtStateor as part of the user's reabank file, likely the latter)From that point, the MSB/LSB for that bank will be constant for the user's system.
Chosen MSB/LSB values are arbitrary, but for now we will prefer MSB < 64 so as not to conflict with factory banks that might be in use (discussed more below).
Loading foreign projects
When the user assigns the bank to a track in a project (which will mean it must have been imported), the uuid -> MSB/LSB mapping will also be stored in the project file. This is needed in case the project is loaded by a different user, whose MSB/LSB assignment for that bank uuid is different (or non-existent).
There are a couple cases to consider when the user opens a foreign project:
If all referenced banks fall under 1 but not 2, we can offer the user to import the missing banks. In that case, they can be imported from distributed factory banks, or from the yet-built online database.
If there are cases of 2, we need some form of conflict resolution. I can think of two options:
1 is ugly, and invasive to the project (consider the case of saving it out and sending back to the original user).
2 sounds cleaner but has some complications. One is that updating the active reabank is ugly and slow in Reaper (~100-300ms). It would have to be refreshed each time the project is loaded, which isn't a huge problem (project load times are dwarfed by other things), except it would slow down project tab changes by a lot. Likewise if a new bank is assigned to some track in the project, it will need to be saved and refreshed.
But if that can be optimized (which may be possible if there are no bank->MSB/LSB conflicts between the projects), project-specific materialized reabanks as the default are elegant, and in line with the philosophy of "MSB/LSB doesn't matter."
Backward compatibility
There are number of obstacles with the current in-field approach:
This implementation should suffice for the version of Reaticulate that transitions to uuids:
Control Surface
Articulation details can be sent to a control surface. Currently MSB/LSB is sent prior to the program change. This could still be done, but if the MSB/LSB is not predictable or user-controlled then it's not very useful. The bank uuid can't be sent as a system text event, because the RFX doesn't actually know it (it would be stored as app-level userdata opaque to the RFX).
This is a pretty esoteric feature -- sending bank details isn't even something I use (relying instead on just standardized program numbers) -- so my inclination for now is not to worry about it, and if the need arises, provide the user some way to discover and/or override the bank's MSB/LSB for purposes of configuring their control surface. However this may be addressed with #70.
Other Considerations
This will depend on #62 to allow storing a full uuid for each bank.
Theoretically, we could store full reabanks in the project too. This might be a nice option to ensure bank updates can't break existing projects. Sometimes the user would want to pick up changes to the bank across all their projects though, so the UX would need to be considered.