Rework embedded banks storage

[Wohlstand]

Current embedded banks system is very limited: it allows to store GM-only banks and doesn't allows to store complete AIL and WOPL banks are promising a multi-bank support. As custom banks, WOPL banks are serving the good job: multi-bank system works perfectly. When the bank was imported as embedded, it's getting be stripped to GM (128:128 set) and no way to use multi-bank except of mapping builder usage.

The work will contains next parts:

• [x] Re-design the ADLDATA storage to be more compact and more flexible.
• [x] Completely rewrite the Gen-AdlData algorithm to fit the new storage format.
• [ ] #272

[jpcima]

I think, the original code of genadldata is sadly lacking of adequate structuring. A shown by the omnipresent imbrications of std::map. Such arrangement does not make ontological sense, although algorithmically it may. In a perspective of rewrite, I would think of favoring a more flat arrangement, but to have such mappings as just side-tools of things like duplicate finding.

Feature request: I would like if embedded banks can be output as collection of WOPL files. I would like to ship this collection along with VST. (VST's realtime loading does not use such a mecanism as adl_openBank)

[jpcima]

I've already make some work in this direction, as now the VST has the collection of embedded banks as WOPL data. One thing of this conversion is, as you have noted, it's only GM. (this work was made by dumping the instruments out of libADLMIDI by bank API)

For storage I have used a custom compressed pak format. The WOPL collection was a 1.5MB set, it was ~140k single-file compressed, vs ~180kB raw ADLdata. As observation, I take it that redundancy elimination is important to keep library footprint small. A direct use of WOPL structures as storage probably would not be a good fit for a thing like this.

I would think of this as a decent strategy: take FileFormats source out of OPL-BE and make it de-Qt-ified, give it banks.ini inputs to convert as wopl, and do the export wopl->adldata.cpp in a final step (includes redundancy detection and eliminating).

[Wohlstand]

Anyway, on a first run I trying to upgrade the working structure (to don't use ADLDATA's structures directly, but use alt that is more friendly for a work, the similar that I made on OPNMIDI). Then, ADLDATA would have absolutely any custom structure, but the working internal format will use own thing.

[jpcima]

Currently I've made a Genwopl program for the needs of VST. It takes inputs with the ini file and using the FileFormats of OPL3-BE.

I took the WOPL banks from its output and I embedded them in latest ADLplug. As result, there are now multi-banks, names, and I think some volume models which would not be set before. Byte comparisons find much differences with banks from libADLMIDI's embeds, but the Genwopl results seemed to match OPL3-BE's own.

The things still not made:

• a cache of measurer results on disk (but OMP parallel has been done)
• the EA and AdLib Gold formats: these, I took a WOPL of them from my old conversion of ADLMIDI's embeds
• and exporting to new adldata, so libADLMIDI can get a use from this

[jpcima]

About the Adlib Gold banks, I must note: Master of Magic 53 and 54 claim 4op instruments in their titles but don't contain any :thinking: (and does not come as too surprising after a look at LoadBNK2 code, forcing the 4op flags at false)

[Wohlstand]

Since some time ago, here is a "wip-new-embedded-banks" branch where work on a new banks database is going

[Wohlstand]

It's almost complete, there are next cases left unfinished:

• [x] get-bank/set-bank are now broken, needs a fix, keep GM banks only for compatibility, for extended banks, make a new separated call
• [x] banks mapper is now broken, nees a fix (same, use GM banks only for compatibility)
• [x] instruments iterator is broken, it's needed for an instrument testing tool

[Wohlstand]

Done a while ago. Speaking about console BANK2WOPN tool, it's a new issue.