sensorium / Mozzi

sound synthesis library for Arduino
https://sensorium.github.io/Mozzi/
GNU Lesser General Public License v2.1
1.06k stars 185 forks source link

RFC: Mozzi 2.0 part 1: Single compilation unit #204

Closed tfry-git closed 2 months ago

tfry-git commented 10 months ago

This is a write-up of something I have been pondering for quite some time, now: One of the larger problems with Mozzi is that in order to change something in the config (and the need to do so is not uncommon at all), you have to modify config files in the library itself. To spell out some of the problems with that:

That problem has been felt, before, so why didn't we fix it, yet?

Why can't we have the latter? Compilation units. For a very rough summary, every .cpp-file is compiled, separately, and only in a final stage everything is linked together. One such .cpp-file is the user's sketch. Another, for example is MozziGuts.cpp. Both also include several common .h-files, but importantly, where they make use of defines such as AUDIO_RATE, this has to match in all compilation units. If AUDIO_RATE was defined at the top of the user-sketch, the MozziGuts.cpp compilation unit would not "see" that define. The current way to make things work, is thus to have both MozziGuts.cpp and the user sketch include a common configuration file. Due to the way the arduino build system works (I won't go into details, mostly, because I don't have them at the top of my head, but I did investigate, before), this common configuration file can only reside in the library folder itself. -> The situation we are in.

So what's the alternative? Putting everything into a single compilation unit. Note this does not mean a single code file, but rather than having MozziGuts.cpp include MozziGuts.h, we'd have MozziGuts.h include (a renamed) MozziGuts.hpp (and the same for all other .cpp files inside Mozzi). In effect, starting with the user sketch as the first file the preprocessor will lump all #included code files together into one big file, and compile that in a single go. Thus a #define near the top of the user sketch will affect all Mozzi code files, too.

What are the drawbacks of that plan?

What are some side-benefits?

What else should we worry about?

So what will this mean for user sketches? Sketches using the default config will remain 100% unchanged. Others might look like:

#include <MozziCustomConfig.h>
#define AUDIO_RATE 32768
#define AUDIO_CHANNELS 2
#define AUDIO_CHANNEL_1_PIN 5

#include <MozziGuts.h>  // this line and everything below unchanged

So, as indicated, I'll write up some further details over the coming days, but of course I'd also like to hear your thoughts on the general plan. Also, what else has always annoyed you about Mozzi, but wasn't quite fixable in a backwards compatible way? Now might be the time to plan that, too.

tfry-git commented 10 months ago

Alright, so what about user sketches which have several compilation units (i.e. several .cpp files) themselves? As long as only one of those files includes Mozzi headers, that's not actually a problem. If several do, we run into two issues:

  1. We'll end up with several implementations (one for each unit) of the Mozzi functions. The linker will complain about that.
  2. We again need to make sure that each unit uses identical configuration parameters (unless they are at default values, of course; see above).

To address this, the recommendation will be to create a common header file inside the user sketch, containing the configuration:

#include <MozziCommonConfig.h>
#define AUDIO_RATE 32768
// etc.

One source file would then start like this:

#include "MyCustomConfig.h"  // i.e. the file, above
#include <MozziGuts.h>

All other source files would start with:

#include "MyCustomConfig.h"
#define MOZZI_HEADERS_ONLY  // this signifies that Mozzi headers shall only include their declaration, not their implementation
#include <MozziGuts.h>

I guess you can see why this is my biggest worry, here. On the other hand, I believe that most sketches will only ever contain one .cpp file, and users creating multi-compilation-unit sketches will be more experienced, too, on average.

tomcombriat commented 10 months ago

Hi,

Indeed, getting custom configs to work without having to edit the source would be a great improvement! I personally run Mozzi on a lot of different configs, also for testing purposes and, even thought I have no problem changing the config, I inevitably run into errors by forgetting to do so. I remember reading that it is a limitation from the Arduino IDE, that has been asked to be removed to work in a similar way than PlatformIO (I think) by other library dev, with no success. I do not think I have the in-depth knowledge of the compilation process to be of much help in conceptualizing this but I am happy to help where I can, in particular in testing.

I have one question though (might be very naive, bare with me ;)), what happens to the #include "MyCustomConfig.h" for the default config? Basically, if there is no custom configuration, like for some of the examples the compiler will complain that this file does not exist, is there a way to branch that out in that case? Maybe with ifincluded or so?

As per other big changes, nothing comes to mind at the moment that would break backward compatibility. I had the thinking for some time to improve further the fixmath provided by Mozzi, which I think can be a bit hard to grasp, for instance, when multiplying one Q8n8 with a Q16n16, what should be the shift to end up with a correct Q16n16? Indeed, that could can be inferred by: (QAnB) = (QCnD * QEnF) >> (F+D-B) but I wonder if that could be templated in some way to add some fixmath operations that gives straight the result in an expected fixmath type. If only we could template the strings of the typenames ;)…

sensorium commented 10 months ago

I'm mostly a spectator these days, but am enjoying the show...

Being able to config within sketches would be great, and your plans sound well considered, Thomas.

And Tom, I agree that the fixed maths is really inelegant at the moment, unfamiliar and confusing for most people! It's in lots of the example sketches and I imagine it's a barrier for many users. Hard to do much synthesis on AVRs without it, though. Any improvements it would be welcome.

Well, since you ask:

what else has always annoyed you about Mozzi

I have a list! Feel free to ignore these, they are just personal gripes...

So, no actual practical help at the moment here, but just my 2 cents...

tfry-git commented 9 months ago

Sorry for the long delay (I'm afraid, it won't be the last time):

@tomcombriat

what happens to the #include "MyCustomConfig.h" for the default config?

That #include is at the level of the user sketch, not Mozzi itself (as an aside, one day I had the idea, that the library could include a config file from inside the sketch directory, but alas, this turned out to be positively impossible). Therefore, if it is not needed, one can simply omit it.

The idea is simply to document this as recommended practice, in case that more than one cpp file exists in the user sketch. We can, but do not have to follow the same in our examples (which are all single-cpp-file, as far as I am aware).

I think I also have some ideas on how to detect (most) cases, where users might have an inconsistent config across two cpp files, and show a clear error message in that case.

@sensorium All of that makes sense. In this particular context, however, I think "naming conventions" is the only thing that would really need to be addressed for a 2.0, as everything else has no risk of breaking backwards compatibility. Arguably, of course the line-segment control may fall into the same category, as it might involve small API changes in ADSR and/or other classes.


Anyway, I'm thrilled to see you're both generally positive towards this slightly megalomaniac plan. At the same time it's beginning to dawn on me that I should try to actually start implementing it :-o. To make the overall work appear less daunting, here's an additional suggestion for getting (some) concepts to work, in a step-by-step fashion:

Would you think an extra folder "playground" (inside the master branch) could be a good idea? It would be excluded from API-docs, and it would carry a Readme that warns against using the code for real, but might be an area, where we can more easily share concepts under development (such as fixedMaths++, but I'm also thinking about some macros that should help with config checking, for instance), without actually having to merge all of Mozzi 2.0 in a single huge MR.

tomcombriat commented 9 months ago

That #include is at the level of the user sketch, not Mozzi itself

Indeed, I must have skipped the part where you said:

the recommendation will be to create a common header file inside the user sketch

I think (but might not be aware of all the implications) that a branch might be more suited than a playground. Of course having a separated branch might lead to one huge MR but at the same time moving this folder to the root will lead to one huge commit. With separate branch(es?) we can consider merging things which do not break compatibility on the go, and also between devel branches (for instance fixMath2 into Mozzi2) and might allow us to be a bit more modular in the approach like branching out from Mozzi2 for testing purposes or new and independent features. It might also give us "clean" sketches for testing instead of importing everything as "#include<playgroud/whatever.h" (I might be completely wrong here…). At the same time it makes these new features less accessible for anyone else who want to test or use nightly built features, and the partial merges from devel to master might give us quite a few headaches so after writing this I am not really sure…

In all cases, I think there is a good set of potential improvements of I am really looking forward to them and to help where I can! Not having to change mozzi_config.h between different setups would really be something…

tfry-git commented 9 months ago

Good points. Let's make it so.

tfry-git commented 9 months ago

As a small update, I'll have to admit that one drawback I failed to foresee for the "single compilation unit", is that add our internal #defines (and we have a lot of those), are now potentially spilling out, i.e. once MozziGuts has been included, internal stuff like "LOOP_YIELD" would now be "available" in user code, and possibly cause clashes, there. Contrary to regular functions and variables, these cannot simply be confined into a namespace, either, as they exist purely in the preprocessor.

I have two-and-a-half ideas to mitigate this:

  1. Quite a few of these can probably be converted to C-level consts (or constexprs or inline functions), with - theoretically - no impact on the assembly produced (a theory, we should test, carefully, though).
  2. All others should be prefixed with MOZZI__ (two underscores), in order to avoid name clashes, and document that they are meant for internal use.
  3. It would also be possible to undef them after use, but I'm a little concerned that this would make the code yet harder to read.