rafalcieslak / vmodsynth

a modular software synthesizer
GNU General Public License v3.0
6 stars 3 forks source link

file I/O #19

Closed robertgyllenberg closed 8 years ago

robertgyllenberg commented 8 years ago

Hi,

I hereby submit the suggested code for file I/O in XML-format. Cheers,

/Robert

rafalcieslak commented 8 years ago

Thanks for your contribution Robert. I've examined your patch, and I have some comments to share.

Before reviewing this branch I pushed some changes to my master branch - remember to git rebase your branch onto my master, in order to import commits from master and automatically re-apply your changes after what was added to the main branch.

There are several details in your patch that will require fixing before I can merge your branch, I'll be describing them below. Keep in mind that when fixing them you can just simply add new commits to your branch, and they will be automatically considered as a part of this pull request - github will even generate an updated diff.


Since vmodsynth is versioned using git, there is really no need to mark one's contributions with a comment on every line they wrote. Git stores commit history, and can easily find the author of any source line, should we need to track/blame them. Similarly, when you remove parts of source, go ahead and remove them entirely. Leaving them as comments brings no benefit (on can always see the previous, deleted version of any file in the changes history), and severely clutters the source in the long run. Same goes for CMakeLists.txt_backup file, it is unnecessary.

You do, however, deserve an entry in copyright notices, if you feel like it. For example, let's have a look at src/modules/v212.cpp. Clearly you wrote this file from scratch, therefore I suggest to change the copyright notice in that file from:

By Robert Gyllenberg 2016 contributing to vModsynt

This file is part of vModSynth.
Copyright (C) 2012, 2013 Rafał Cieślak

to:

This file is part of vModSynth.
Copyright (C) 2016 Robert Gyllenberg

If you are concerned that you basically copied a module template from a file I created, feel free to add one more line:

Based on vModSynth module template by Rafał Cieślak 2012, 2013

In files like src/Engine.cpp you've marked your contribution in a comment in the middle of the source file. Such section-marking style eventually leads to scattered source code. Instead, state your contribution in the copyright notice at the beginning of the file. For example, in Engine.cpp you might want to add a new line in the notice, similar to this one:

Modified 2016 Robert Gyllenbert - implemented file saving and loading procedures.

Keep in mind that there it is never necessary to explicitly mark your copyrights in a source file - you own those rights anyway, so there is never any harm in under-using copyright notices. So in files where you've barely changed anything it's totally fine if you don't leave your signature anywhere.


I see that some parts of the source code are oddly formatted, while others look good. In particular, in Engine.cpp the new source code uses 1-space indentation. Can we please stick to 4-spaces uniformly throughout the entire source?


It appears that the FindXml++.cmake script you're using is indeed having some trouble detecting libxml++ correctly. However, hard-coding the paths in CMakeLists is a very bad idea, because on various systems the libraries and header files may be installed in different directories. For example, the ubuntu variant that I use stores some libxml++'s headers in /usr/lib/x86_64-linux-gnu/libxml++-2.6/include/. The main task of cmake is to dynamically detect these paths on any system, so writing them by hand kills cmake's purpose. By the way, libxml++ documentation states that the only correct way of including it's headers is to #include <libxml++/libxml++>, otherwise it suffers from some weird internal problems.

I suppose writing cmake scripts can be quite tedious if one is not experienced with them. I'll swiftly prepare a variant that detects libxml++ correctly and will send it to you as a pull-request into your branch :-)


There is a place in your code where you need to determine whether a knob is a selector or not, and for that you introduced an extra subtype member field. That's unnecessary! You can determine the type with any of these methods:

if(typeid(*knobs[i]) == typeid(Selector)){
    // This is a selector!
}

or

if(dynamic_cast<Selector*>(knobs[i]) != nullptr){
    // This is a selector!
}

The changes you've made to v100 module are invalid, I suspect. There are two reasons why that if statement is important:


The toolbar save/open icons are nice, but their placement confuses me a little. I suppose the most used buttons on the toolbar are the "+" and edit mode switch. Let's move save/open buttons to between "delete" and "zoom in" - they are far less frequently used, so they don't deserve the left-most spot.

Finally, about the save files. Firstly, I find it peculiar that you use an elegant, full-featured xml library for reading xml files, but you make no use of it when writing them. I suppose writing xml files manually is indeed a simple task, so it's a good starting point, but it would make much more sense (and would be easier to maintain in the future) to use libxml++ for writing as well!

Secondly, I see you chose to gather all knobs globally, and save the state of all knobs mixed together (so the std::vector you switched too actually works as a map from knob id into knob pointer). I guess this works okay, apparently. However, I believe a more natural approach would be to store knob values per each module separately. So instead of

<module ... >
<module ... >
<module ... >
<knob ... >
<knob ... >
<knob ... >
<knob ... >

the save file would have a hierarchy:

<module ...>
    <knob ... >
    <knob ... >
    <knob ... >
</module>
<module ...>
    <knob ...>
</module>

Why do I feel it is a better idea? Generally, a knob number within a module is always fixed. So, for example, the 3rd knob of v100 is always the pulse width knob, regardless of the patch state - while, on the contrary, in the solution you implemented the 9th global knob may be the knob of any module, and resolving which knob it actually is depends on the order of modules. This has various potentially bug-prone side-effects, for example, one cannot change an order of nodes in the xml file without recalculating knob ids (in my proposed solution the order of xml nodes does not affect knob ids). Similarly, if a future vmodsynth release introduces a new knob onto a module, then all save files become instantly corrupted, as they now use misaligned knob numbering (the alternative solution is immune to this as well)! Also, the hierarchical solution makes it easier to add extra parameters to modules in the future. It is also much easier to read such xml-files by hand, as I can easily see which knob value belongs to which module.

Finally, your trick to save wire parameters by storing their coordinates is... unnecessarily complicated. Wouldn't it be much much easier to just store that "there is a wire from module 5 knob 1 to module 2 knob 3" ? Like this:

<wire src="5" src_knob="1" dst="2" knob="3"/>

Not only this solution is simpler to implement and easier to maintain, but it is also more forward compatible (suppose a module in a future release is 10 px wider - again, all save files become invalid), and keeps the GUI information away from the save file, which should only describe the layout of the patch.

By the way, why are module names stored in the save file? I don't see them being used for anything.


Lastly, I am quite worried about that std::setlocale function you use at some point. You never revert the locale to it's previous setting, so it ends up permanently switched to "C", which is definitely undesirable!


Whoa, I guess that's all I have to say. Sorry for such a long reply, but I wanted to go into details in order to make sure that all my points are explained well. In case you need some help fixing any of above or wish to discuss some details, I'll be happy to help you!

robertgyllenberg commented 8 years ago

Hello,

I have now made most of the suggested changes and corrections and I fully agree with your criticism on most points. I have tried to tidy things up and removed dead code and unnecessary comments etc. And yes, let's indeed stick to the 4 space indentation.

When it comes to the structure and semantics of the XML-code, I also agree on most points. Storing the logical numbers of inlets and outlets instead of their coordinates indeed makes better sense and preserves future compatibility if one should move things around. I have made the necessary changes for this. (In fact, keeping the logic and functionality totally separate from user interface would be a noteworthy milestone of some future release.)

Your further suggestion about structuring the XML by aggregating the knob (/selector/switch) setting within each -section in favor of the current model is fine with me, but i also find good arguments of keeping it in its current form, since it resembles the workflow of a manual setup:

  1. Install the modules in the rack, 2. Do the wiring 3. Turn the knobs into position (and preferably the master volume last in line in order to avoid noise pollution.)

I did move the "Open" and "Save" widgets away from the pole position as suggested, but I still wonder if there is some (written or de Facto-) standard stating where those buttons should reside? (Maybe a separate file-menu would be a better choice in order not to clutter the menu bar too much.) There are still use cases that I have not investigated, e.g. when a user tries loading a setup with some modules already residing in the rack. (Yes, my current implementation will mess things up.) I shall add some "New"-button that clears the rack up (preferably asking the user to save any unsaved configuration etc...).

I have committed the changes to my branch, but since I'm kind of new to this Github, i still have to ask. Is the current Pull Request still valid for my updates or do I need to close it and open a new one?

Best Regards, /Robert

robertgyllenberg commented 8 years ago

One more detail: The XML-code also contains metadata that is not needed by the application, such as the names of the modules listed. This is only an attempt to improve human readability of the XML-file itself. One could also add names to the setups or even think extend the functionality to include whole sets of setups in one file and a means of changing the synth patch in run time via keyboard or MIDI-input. Human readability is also (in my opinion) a strong benefit of using XML. (In the 1980's, when storage was expensive, we used to squeeze data into bit vectors and a typical maze game would take only 32 bytes to store the configuration including geometry of an entire level.) I agree, the XML++-library should indeed be utilized for both writing and reading the XML-file, but as I'm used to both reading and hand writing a lot of structural documents, the quick-and-dirty way of getting things saved was outputting preudo-XML to std:cerr before even installing the xml++ -library on the machine. Writing an XML-parser from scratch would be a waste of time, so I made a compromise here. Utilizing the library for creating the document as well is on my ToDo-list. /Robert

rafalcieslak commented 8 years ago

Nice progress! I'm happy to see this branch shaping up nicely, I should be ready for merging really soon.

Here are some minor notes:

Please don't change --std=c++11 to --std=c++0x. The c++0x flag was used by compilers before they fully supported C++ 11 standard, and is now deprecated. The reason why --std=c++11 won't work on Ubuntu 13.04 is because that version of ubuntu shipped with an older GCC, which did not yet support the newer c++ standard. If one wishes to install vmodsynth on such an old linux distro version, they have to upgrade their compiler to a newer version that supports c++11 features. By the way, Ubuntu 13.04 and 13.10 have both reached their end-of-life and are no longer supported releases.

I see you forgot to delete CMakeLists.txt_backup.

Now that you've added get_outlet_n_at and get_inlet_n_at, the original functions get_outlet_at and get_inlet_at are no longer used, and can be safely removed.

I also find it odd that you search for outlet/inlet id's querying them by their position. But you already have a pointer to such outlet/inlet, I suppose it should be enough to search for their id by searching the vector of outlets for that pointer. Generally, I am very bothered to see any GUI features, like knob placement, within save routines, because the structure of the path is independent from what it looks like.

Also, because the save file format has changed, the demos you provided do not open for me anymore :( (by the way, I really liked some of them!). If you wish, you might correct them by hand, but I suggest we remove them for now, and re-create them after this branch is merged.

I am not familiar with any standard that defines a recommended layout of toolbar buttons, though I suspect http://freedesktop.org may have some suggestions. And I totally agree that since we have more global actions, grouping them in a File menu totally makes sense. Though let's implement it as a separate feature.

About save file format: I agree with your point about readability. Let's keep module names in the file, but please mark the point where the names are saved with a short comment explaining that the names are intended to be unused and are there just for humans reading the file - so that we won't delete them in a future source code cleaning.

To be honest, my workflow with modular synthesizers that I personally use is actually different to what you described. I add a module, tune it's knobs (possibly listening to its output, if applicable), then add a next module, connect it with with previously tuned modules, tune it's knobs, add another module, etc. Furthermore, I don't feel like we need the XML file to resemble any workflow. It's good if a human is capable of reading the graph that a save file represents, but it's not like they are going to simulate the path in their minds.

Also, for the sake of forward-compatibility, please include some kind of save file format version information, maybe like this: <patch name="synthesize" savefile_version="1">, this way, when loading a file, we could check if this file uses an older (or newer) format.

Finally, to clarify how pull-requests work: There is no need to create a new pull request. You have update this one with some changes, the discussion continues, you may add even more changes, repeat until merged. This is a typical workflow with PR's. However, once this one gets merged, it it gets closed and archived, so to add another feature you will then need to open a new PR. It is generally recommended to open one PR per each feature you are working on. Normally you would create multiple branches within your fork, work on them independently, and create separate PRs for each of them. This way the changes in each branch may be reviewed, discussed or rejected independently from others.

robertgyllenberg commented 8 years ago

Right,

The position-sensitive in- and outlet-queries have now been substituted with a more robust solution and the unused functions have also been removed accordingly.

I put the folder back for now, but we can, indeed leave it out from the official release.

I also have a couple of specific questions about the naming convention of modules:

  1. As I have been implementing some modules of my own, I have simply numbered them as 211, 212, 213 (because these numbers were simply not in use) without any second thought. Both numbers and functionality are "pulled out of my hat", because there are no corresponding archetypes for these. Is there some naming or numbering convention, that I should be aware of? Would it be better to use another number space, say, v1xxx or v2xxx for my own modules?
  2. Would it make sense to refactor the workflow of introducing new modules to the application, either by compile time text substitution or maybe in the future allowing run-time linking of separately compiled module binaries? I realize this will involve a lot of work or at worst have a degrading impact on the real-time performance, so for now on, I will be happy just adding the modules "by hand" to the source tree.

BR /Robert

rafalcieslak commented 8 years ago

I like that you're working on new modules as well, but I recommend moving them to a separate branch and opening another pull request for them. The "one feature per one banch / pull request" strategy really works best, because the file I/O features you've implemented are almost ready to be merged and it would be great to be done with that already, but as this branch has grown in new features such as experimental modules, I'll need to review it again (Similarly, there are some details that I would like to work on but I have to wait until file saving features land in master branch). Also I suppose there may be a lot to discuss about new module ideas, keeping that discussion separate from file saving features would totally make things clearer.

I'll answer your questions about modules briefly here, and we can discuss them in greater detail either as a separate issue (on Github, issues are fine for both reporting bugs and discussing ideas).

First, the naming is not entirely arbitrary - my intention is to mimic real analog module synthesizers, and they frequently have numbered hierarchical names. Initial idea was to use 1xx for signal sources, 2xx for signal control, 3xx for filters, 4xx for mixing and processing, 7xx for effects, 8xx for sequencers, and 10xx for system special modules (application input/output). I should write that up somewhere as a reference.

Second, such a dynamic plug-in system that uses run-time linking is generally a great idea, and actually it happens that I do have a fair experience implementing that kind of module support. Careful implementation can have zero performance impact. However, the amount of work and design needed for this feature is far greater then the total effort put into developing vmodsynth until now. Also, this feature is only beneficial if we intend to have a huge collection of modules (which would be very unlike an analog synthesizer), or if we want third-parties to develop modules independently from us, which - given the current userbase of vmodsynth - is unlikely. Therefore while I consider this feature to be a nice idea, it is not the right time yet to work on it.

On the other hand, compile-time macros that simplify creating modules (like a template for module class declaration) would be fine.

robertgyllenberg commented 8 years ago

OK, I get the point. I have removed the experimental modules and demos from this fork. BR /Robert

robertgyllenberg commented 8 years ago

Hi,

More precisely, the fork is still the same, but the experimental modules and demos now reside in a new branch of mine called features and the master branch only contains the File I/O -specific implementations reflecting the name of my original pull request. Regarding the structure of the XML-file, i will continue to implement the parsing in such a way, that it allows both save formats (structures) that we discussed before. In the future, we can have the knob etc. settings aggregated within -sections as well, without invalidating the current model. Hope this is OK?

Cheers, /Robert

rafalcieslak commented 8 years ago

This seems like a good idea.

robertgyllenberg commented 8 years ago

Changed the file output format to reflect your idea about aggregating module specific settings as nested within each -node. The load function is able to parse both formats, but all output is saved in the new format. Also added an attribute reflecting the format version. This attribute is for reference only.

rafalcieslak commented 8 years ago

Sorry for the long delay, been very occupied recently.

Nice work! This branch is looking very well now, and I don't see any more issues. Unless there is anything else you would like to include with this feature, this branch is ready for merging - as soon as I give it some final thorough testing :-)

robertgyllenberg commented 8 years ago

Hello,

Unless there is anything else you would like to include with this feature, this branch is ready for merging.

At some earlier point, I was thinking about adding some functionality to make the save/load -functionality more "fool proof", such as displaying warning messages about unsaved changes at different levels (changed modules / changed patch / changed knob settings etc). On the other hand, I believe we should still leave some responsibility to the end user, whether teacher, student, hobbyist or professional.

In the present implementation, the program will crash, if an attempt is made to load some type of module that is not implemented or hasn't been compiled in. Since i moved both my experimental modules and the demo-patches that are using them to another branch, this situation should normally not come up.

I would like to have your opinion: Would it be necessary to sanity-check each file for non-existent modules, non-existent jacks, controls etc. at runtime?

Sorry for the long delay, been very occupied recently.

No need for apologies. Take your time.

Cheers, /Robert