Adaptation Generator

[markusschloesser]

How about a simple form based adaptation .py generator? A user would need to

1. name the synth / adaptation = generates file names and identity
2. storage (number of banks etc)
3. all the rest, divided into mandatory and optional. Sysex strings should be copy/pastable, tool should handle hex to python /0x00 conversion. The tool itself in knobkraft (or website) would spit out a .py file so that users don't need to actually mess around with code. I guess that would lower the barrier for users to actually do Adaptation.

[Andy2No]

Good idea. I expect there are some synths that won't work for, and some .py code would be needed to handle the special cases, but that could probably cover a lot of them.

How about a generic .py adaptation with just some commented constants and string constants at the top for people to change, to make a first draft? That's basically the same idea, but without a form to fill in - just some simple editing to do without needing to know anything about Python.

[christofmuc]

Interesting! I believe it would only work for the simple cases, but even of these are many. You could also make the structure confiom to the fields from the SoundDiver Adaption editor, so if you open an ADA file in SoundDiver, all you need to do is to copy over the form data from SoundDiver. Just an idea.

You wouldn't necesarily need a generator, you could just format the Python file in that there is a section at the top which defines the data fields, and code below uses that data. Later you could then load the data from a file generated by some tool etc.

Originally I though you would just start with a rather similar synth in the create new Adaptation button function, and then just edit the file which has enough comments to get you going. But even finding out which one is similar can be hard task (we had that over there with the idea of the table showing all the features of the adaptations).

I had a long discussion of GS about code-driven Adaptations vs. data-driven Adaptations, and while I think the code-driven approach of KnobKraft is far more future-proof than all the data-driven Librarians, it clearly requires a programmers mind. Recent example pro code-driven method: To implement the renamePatch() function for the Electra One adaptation, you have to turn sysex binary data into a JSON string, parse the JSON, set the attribute, convert new JSON back to binary. A few lines of Python, and importing the json library, and you're done without any change to the KnobKraft. In data-driven Librarians - impossible.

Another way to make it easier is to implement higher-level abstractions in Python, e.g. a Sequential implementation (they are all very much alike) which you import and just provide some data points for the generic Sequential module.

Then a Roland base module, a Yamaha base module etc. that at least takes away code copying between adaptations and would provide more tricky stuff like the escapeSysex/denibble functions.

[christofmuc]

Sorry when I repeated AndyNo's comment, I answered Markus question first before getting into his answer ;-)

[christofmuc]

So this was interesting. I took the SoundDiver Adaptation manual, and went through the Tutorial "3.1 Bank Manager Kawai K1". I distilled the data entered into the diverse dialog boxes of SoundDiver to the following data structure (short of Cartridge banks, the complication with the Edit Buffer and the missing Multi-Bank Dumps, for the sake of brevity):

    adaptation = {
        "Manufacturer Name": "Kawai",
        "Manufacturer MIDI ID": 0x40,
        "Model": "K1",
        "Device ID": (2, 1, 16),  # sysex offset, min value, max value
        "Default Timeout": 200,  # milliseconds to wait during Scan
        "Default Send Pause": 80,  # milliseconds to wait during sending multiple messages
        "Scan with Universal Device Inquiry": False,
        "Scan request": [0xf0, 0x40, 0x00, 0b01100000, 0xf7],
        "Scan reply": [0xf0, 0x40, 0x00, 0b01100001, 0x00, 0x03, 0xf7],
        "Data Types": [{"Name": "Single", "Size": 87, "Name Size": 10, "Name Offset": 0, "Name format": "Ascii"}],
        "Bank Driver": [{"Bank Name": "Int-Singles I/1", "Data Type": "Single", "# of Entries": 32,
                         "Transmission Format": "7bit", "Checksum Type": "Kawai K1/K4",
                         "Memory Bank": True,
                         "Offsets": (0, 0, 0),  # These are the program offsets for single, bank, program change
                         "Single Request": [0xf0, 0x40, 0x00, 0x00, 0x00, 0x03, 0x00, "EN#", 0xf7],
                         "Single Reply": [0xf0, 0x40, 0x00, 0x20, 0x00, 0x03, 0x00, "EN#", "SUM", "SIN", "CHK", 0xf7],
                         "Bank Request": [0xf0, 0x40, 0x00, 0x01, 0x00, 0x03, 0x00, 0x00, 0xf7],
                         "Bank Reply": [0xf0, 0x40, 0x00, 0x21, 0x00, 0x03, 0x00, 0x00, "[", "SUM", "SIN", "CHK", "]",
                                        0xf7]},
                        {"Bank Name": "Int-Singles i/2", "Data Type": "Single", "# of Entries": 32,
                         "Transmission Format": "7bit", "Checksum Type": "Kawai K1/K4",
                         "Memory Bank": True,
                         "Offsets": (32, 32, 32),  # These are the program offsets for single, bank, program change
                         "Single Request": [0xf0, 0x40, 0x00, 0x00, 0x00, 0x03, 0x00, "EN#", 0xf7],
                         "Single Reply": [0xf0, 0x40, 0x00, 0x20, 0x00, 0x03, 0x00, "EN#", "SUM", "SIN", "CHK", 0xf7],
                         "Bank Request": [0xf0, 0x40, 0x00, 0x01, 0x00, 0x03, 0x20, 0x00, 0xf7],
                         "Bank Reply": [0xf0, 0x40, 0x00, 0x21, 0x00, 0x03, 0x20, 0x00, "[", "SUM", "SIN", "CHK", "]",
                                        0xf7]}
                        ],
    }

So yes, this contains everything that is needed to work with the Kawai K1. We could make a "SoundDiver" adaptation type that reads this data and implements the functions required by the Orm, and I think it would work.

There is a bit of black magic going on here however, typical for data-driven approaches:

• The name format "Ascii" is easy to understand, but what happens with synths having a completely different character set like the Access Virus or the Roland D-50, or a non-unique character set like the Matrix 6 where two different numbers could create the same character?
• The "Transmission Format" parameter also is a fixed list of character encodings, like nibbled, 7bit, MSB byte, ... I could see that this list is more limited than the list of character sets, but still it is unlikely that you'll catch all formats.
• The checksum type is quite hilarious, as it is called only "Kawai K1/K4", so that is certainly something that should be code and not data, imho,

I do like the "pseudo bytes" they use like "EN#" or even the repeat-loop with the brackets "[" "]", this is pretty powerful.

You can also see the limitation - while they have abstracted the entry number EN# for a program place, they failed to abstract the bank number but rather want a copy of the whole "bank driver" data structure for each bank, and then you need to figure out again where the differences between two banks are, and you could have a bug that one bank works but the other doesn't.

Still, it could be a worthwhile exercise to implement this, as it could make porting SoundDiver ADA files much easier. You just have to figure out all the various special cases...

[christofmuc]

It's pretty straightforward, I implemented a SoundDiver driver for the code above in about an hour, and it seems to work. A bit more is missing, but you get the idea. The full file is referenced in the commit above.

You can also see in the file how I add tests to the python module to check if my implementation is correct, without even needing to fire up KnobKraft.

[markusschloesser]

I expect there are some synths that won't work for, and some .py code would be needed to handle the special cases, but that could probably cover a lot of them.

MS: that's always the case anyway, isn't it? But a 80/20 approach should work. :-)

[markusschloesser]

I had a long discussion of GS about code-driven Adaptations vs. data-driven Adaptations, and while I think the code-driven approach of KnobKraft is far more future-proof than all the data-driven Librarians, it clearly requires a programmers mind. Recent example pro code-driven method: To implement the renamePatch() function for the Electra One adaptation, you have to turn sysex binary data into a JSON string, parse the JSON, set the attribute, convert new JSON back to binary. A few lines of Python, and importing the json library, and you're done without any change to the KnobKraft. In data-driven Librarians - impossible.

MS: I tested the adaptation and managed to crash knobkraft succcessfully :-) Do you automatically get the crash dmp files?

Another way to make it easier is to implement higher-level abstractions in Python, e.g. a Sequential implementation (they are all very much alike) which you import and just provide some data points for the generic Sequential module.

Then a Roland base module, a Yamaha base module etc. that at least takes away code copying between adaptations and would provide more tricky stuff like the escapeSysex/denibble functions.

MS: That's why I had the idea, I thought manually typing the sysex manufacturer ID 5 times can be done more efficiently (I'm lazy) :-) MS: A manufacturer based template would ease writing adaptations significantly!

[christofmuc]

MS: I tested the adaptation and managed to crash knobkraft succcessfully :-) Do you automatically get the crash dmp files?

ha ha. Not that I have seen, strange. Can you crash it again, please :-) ?

MS: That's why I had the idea, I thought manually typing the sysex manufacturer ID 5 times can be done more efficiently (I'm lazy) :-) MS: A manufacturer based template would ease writing adaptations significantly!

I will check. The DSI/Sequential stuff naturallly lends to it, as they use the same software in everything since the Evolver, it's mostly copy and paste from there. For Roland same, but for that I need to support the handshake stuff properly.

How are you getting along with making your own adaptation? Let me know if I can be of help.

[markusschloesser]

It's pretty straightforward, I implemented a SoundDiver driver for the code above in about an hour, and it seems to work. A bit more is missing, but you get the idea. The full file is referenced in the commit above.

You can also see in the file how I add tests to the python module to check if my implementation is correct, without even needing to fire up KnobKraft.

re your K1 sounddiver adaptation: I'm not sure I fully understand.

1. How do I open an existing diver ada in sounddiver to be able to copy stuff out of it and into a knobkraft adaptation?
2. is your sounddiver driver a generic one which, presuming I got 1. done, I can use use to get all existing sounddiver supported synths into knobkraft? And how would I do that? Use you're new k1.py file as a template?

[christofmuc]

1. How do I open an existing diver ada in sounddiver to be able to copy stuff out of it and into a knobkraft adaptation?

Hm, I have to look it up, in the SoundDiver application there is an Adaptation editor window that let's you see all these dialog boxes with the MIDI messages. Let me try to dig it out, last time it didn't want to start on my box.

2. is your sounddiver driver a generic one which, presuming I got 1. done, I can use use to get all existing sounddiver supported synths into knobkraft? And how would I do that? Use you're new k1.py file as a template?

Ah, yes, just not today! I am still working on it, and the goal would be that there is a file sounddiver.py or something which then get's imported into a KawaiK1.py, and the KawaiK1.py basically just contains the data definition and maybe one line of instantiation. That's a bit of more work though, as it is kind of duplicating the dynamic profile implementation - you should only write a function createBankDump if the synth supports BankDumps etc.

[markusschloesser]

MS: I tested the adaptation and managed to crash knobkraft succcessfully :-) Do you automatically get the crash dmp files?

ha ha. Not that I have seen, strange. Can you crash it again, please :-) ?

MS: That's why I had the idea, I thought manually typing the sysex manufacturer ID 5 times can be done more efficiently (I'm lazy) :-) MS: A manufacturer based template would ease writing adaptations significantly!

I will check. The DSI/Sequential stuff naturallly lends to it, as they use the same software in everything since the Evolver, it's mostly copy and paste from there. For Roland same, but for that I need to support the handshake stuff properly.

How are you getting along with making your own adaptation? Let me know if I can be of help.

re crash: how can I upload the dmp file? GitHub won't let me re adaptations: have started 5 (!), none work properly 🙄😂.

1. Andromeda: already written about, Currently only gets detected. It is supposed to have an edit buffer, but doesn't really. Can't read the edit buffer for "programs" only for "Mixes". Don't know why. Some things not really clear in the (A6) doc. Will need to continue.
2. K5000s: no edit buffer at all
3. Jomox Alphabase: insufficient sysex documentation, will need to write to Jürgen Michaelis to find out if there's more sysex supported but not documented
4. Yamaha FS1R: when I tried yesterday, I had to wrap my head around the whole performance, parts, voices, fseq thing. Will try later with the new knowledge from today
5. Quasimidi Cyber-6: insufficient sysex support, no device detection, only "Request Data from device" and "Dump Data to device" plus an "Address Map" So if I can get 1,2,4 for free by using ada, I'll save a lot of time. Plus, some concepts are hard for me to understand. I am good at inspiring, not so much at "doing" when it comes to code.

btw if you ever find that there's enough "ideas" from my side, please let me know :-)

[markusschloesser]

as for the other synths:

1. Mks80: waiting for your adaptation ☺😇
2. Prophet 12: very happily using Andy's
3. Kijimi: I have supported the kickstarter for babufrik, which is a librarian and editor for the kijimi (and now open source, might be worth checking out, also Juice based iirc)
4. RD-8: waiting for your adaptation, but as there's not presets anyway, doesn't really matter
5. SE ATC-X: Have the official editor for it, which is ok-ish, will be maybe look at it later.

[markusschloesser]

1. How do I open an existing diver ada in sounddiver to be able to copy stuff out of it and into a knobkraft adaptation?

Hm, I have to look it up, in the SoundDiver application there is an Adaptation editor window that let's you see all these dialog boxes with the MIDI messages. Let me try to dig it out, last time it didn't want to start on my box.

2. is your sounddiver driver a generic one which, presuming I got 1. done, I can use use to get all existing sounddiver supported synths into knobkraft? And how would I do that? Use you're new k1.py file as a template?

Ah, yes, just not today! I am still working on it, and the goal would be that there is a file sounddiver.py or something which then get's imported into a KawaiK1.py, and the KawaiK1.py basically just contains the data definition and maybe one line of instantiation. That's a bit of more work though, as it is kind of duplicating the dynamic profile implementation - you should only write a function createBankDump if the synth supports BankDumps etc.

Adaptation editor: there's a menu entry in the "Install" windows, but greyed for me for all the synths (uni and specific) ada/knobkraft: got it 😊 Thanks!

[christofmuc]

Ok, I found it in SoundDiver - you need go to the Install window, select a synth of type "UNI" (universal module, its listed in column 3 which by default is not shown) and add it to your setup. Then, in the Setup window you can do an "Open Device" double click on the device (first abort all the scan and autodetection messages). When you're in the Device Window, you suddenly get an Adaption menu in the main menu bar. First menu item is Edit Adaptation... which will open the Adaptation Editor.

I am just looking at the Novation Supernova, but it looks insanely complex with 19 different data types, I am not sure this is what I'd like to do...

[markusschloesser]

worked for the Andromeda!

[Andy2No]

as for the other synths:

2. Prophet 12: very happily using Andy's

That's a lot more credit than I deserve - I only helped a bit with bug finding. Christofmuc did the rest.

[christofmuc]

@markusschloesser Regarding the crash dump - if you use the Windows version from the installer here, and have consented to upload a crash, it should automatically be uploaded. You can give consent in the Help menu item again.

[markusschloesser]

I did consent in the past, now again. Let me know if you received it. Otherwise I can provide in another way. btw opened it, it says: CONTEXT: (.ecxr) rax=00007ffaee9d9eb8 rbx=00007ff673f2bf60 rcx=000000000000000c rdx=00007ffaee9d9eb8 rsi=0000000000000000 rdi=000000514a78ebf0 rip=00007ffb0b25d759 rsp=000000514a78ea80 rbp=000000514a78ebc0 r8=0000000000000008 r9=0000000000000000 r10=000000514a78e469 r11=000001dd292a0000 r12=000001dd2cff4e90 r13=000001dd379d86b0 r14=000001dd32233fb0 r15=000001dd2cff4e80 iopl=0 nv up ei pl nz na pe nc cs=0033 ss=0000 ds=0000 es=0000 fs=0053 gs=002b efl=00000202 KERNELBASE!RaiseException+0x69: 00007ffb0b25d759 0f1f440000 nop dword ptr [rax+rax] Resetting default scope

EXCEPTION_RECORD: (.exr -1) ExceptionAddress: 00007ffb0b25d759 (KERNELBASE!RaiseException+0x0000000000000069) ExceptionCode: e06d7363 (C++ EH exception) ExceptionFlags: 00000001 NumberParameters: 4 Parameter[0]: 0000000019930520 Parameter[1]: 000000514a78ebf0 Parameter[2]: 00007ff673f2bf60 Parameter[3]: 00007ff673550000

PROCESS_NAME: KnobKraftOrm.exe

ERROR_CODE: (NTSTATUS) 0xe06d7363 -

EXCEPTION_CODE_STR: e06d7363

EXCEPTION_PARAMETER1: 0000000019930520

EXCEPTION_PARAMETER2: 000000514a78ebf0

EXCEPTION_PARAMETER3: 00007ff673f2bf60

EXCEPTION_PARAMETER4: 7ff673550000

STACK_TEXT:
000000514a78ea80 00007ffb0841486d : 000000514a78ebf0 0000000000000000 0000000000000000 0000000000000000 : KERNELBASE!RaiseException+0x69 000000514a78eb60 00007ff673772a03 : 00007ff673550000 000000514a78ec80 000001dd2cff4e70 00007ffa00000028 : VCRUNTIME140!_CxxThrowException+0xad 000000514a78ebd0 00007ff673550000 : 000000514a78ec80 000001dd2cff4e70 00007ffa00000028 00007ff673c6da60 : KnobKraftOrm+0x222a03 000000514a78ebd8 000000514a78ec80 : 000001dd2cff4e70 00007ffa00000028 00007ff673c6da60 000001dd3230b5c0 : KnobKraftOrm 000000514a78ebe0 000001dd2cff4e70 : 00007ffa00000028 00007ff673c6da60 000001dd3230b5c0 0000000000000001 : 0x000000514a78ec80 000000514a78ebe8 00007ffa00000028 : 00007ff673c6da60 000001dd3230b5c0 0000000000000001 000001dd32233fb0 : 0x000001dd2cff4e70 000000514a78ebf0 00007ff673c6da60 : 000001dd3230b5c0 0000000000000001 000001dd32233fb0 000000514a78ec80 : 0x00007ffa00000028 000000514a78ebf8 000001dd3230b5c0 : 0000000000000001 000001dd32233fb0 000000514a78ec80 00007ff6737729a4 : KnobKraftOrm+0x71da60 000000514a78ec00 0000000000000001 : 000001dd32233fb0 000000514a78ec80 00007ff6737729a4 000000000000001f : 0x000001dd3230b5c0 000000514a78ec08 000001dd32233fb0 : 000000514a78ec80 00007ff6737729a4 000000000000001f 0000000000000019 : 0x1 000000514a78ec10 000000514a78ec80 : 00007ff6737729a4 000000000000001f 0000000000000019 000000000000000f : 0x000001dd32233fb0 000000514a78ec18 00007ff6737729a4 : 000000000000001f 0000000000000019 000000000000000f 00007ffb0b66f706 : 0x000000514a78ec80 000000514a78ec20 000000000000001f : 0000000000000019 000000000000000f 00007ffb0b66f706 000001dd00000001 : KnobKraftOrm+0x2229a4 000000514a78ec28 0000000000000019 : 000000000000000f 00007ffb0b66f706 000001dd00000001 fffffffffffffffe : 0x1f 000000514a78ec30 000000000000000f : 00007ffb0b66f706 000001dd00000001 fffffffffffffffe 000000514a78ed18 : 0x19 000000514a78ec38 00007ffb0b66f706 : 000001dd00000001 fffffffffffffffe 000000514a78ed18 00007ff673772117 : 0xf 000000514a78ec40 00007ffb0d7a5d21 : 0000000000000020 000001dd292a0000 0000000000000000 0000000000000000 : ucrtbase!_malloc_base+0x36 000000514a78ec70 00007ff6737721e4 : 000000514a78ed18 0000000000000000 0000000000000000 000000514a78efb0 : ntdll!RtlFreeHeap+0x51 000000514a78ecb0 000000514a78ed18 : 0000000000000000 0000000000000000 000000514a78efb0 000000514a78eef0 : KnobKraftOrm+0x2221e4 000000514a78ecb8 0000000000000000 : 0000000000000000 000000514a78efb0 000000514a78eef0 00007ff67376ba4b : 0x00000051`4a78ed18

SYMBOL_NAME: knobkraftorm+222a03

MODULE_NAME: KnobKraftOrm

IMAGE_NAME: KnobKraftOrm.exe

STACK_COMMAND: ~0s ; .ecxr ; kb

FAILURE_BUCKET_ID: CPP_EXCEPTION_e06d7363_KnobKraftOrm.exe!Unknown

OSPLATFORM_TYPE: x64

OSNAME: Windows 10

FAILURE_ID_HASH: {b1d1d561-675e-cd2c-4dd5-ebc98e81fcdf} `

[christofmuc]

@markusschloesser Hm, no, strangly nothing to see in the Sentry,io log. Normally I get crashes in there. The stack trace above shows an unhandled exception, not sure if in the C++ part or the Python part. Do you see a directory %APPDATA%\KnobKraftOrm\sentry? The content would be interesting. Else, you can attach stuff here or better open a new issue with the means to reproduce the crash.

[markusschloesser]

that's where I got the dmp file from. new issues #60

[christofmuc]

Marked the crash dump discussion as off-topic, as we now have #60.

Back on topic - I will look at the Andromeda ADA and see how much is missing over the K1 I did yesterday, I think the K5000 is epitome of Sysex, you will not find a synth with more different message types. For that, we definitely need the multi-data type function ready. I'd suggest to focus on one device and get that working first before trying others, the FS1R is also a beast, and the Yamaha format is also special, I don't think you can already do it with an adaptation. The Cyber-6 (never heard of that before, looks cool) should be doable, I'll give it a go.

[markusschloesser]

Ok, I found it in SoundDiver - you need go to the Install window, select a synth of type "UNI" (universal module, its listed in column 3 which by default is not shown) and add it to your setup. Then, in the Setup window you can do an "Open Device" double click on the device (first abort all the scan and autodetection messages). When you're in the Device Window, you suddenly get an Adaption menu in the main menu bar. First menu item is Edit Adaptation... which will open the Adaptation Editor.

I am just looking at the Novation Supernova, but it looks insanely complex with 19 different data types, I am not sure this is what I'd like to do...

How did you export the data in the "Edit Adaptation" window? Asking because Sounddiver does have an existing Waldorf Pulse UNI, which I could then put into the KawaiK1 adaptation?

[christofmuc]

@markusschloesser You can't export :-/ This is why I didn't continue that path, the file format is binary obscure, so you basically have to open SoundDiver and best case copy/paste the info. Note that I have not implemented all the different bits and pieces and details, so taking no a new synth probably needs to extend the implementation of the KnobDiver SoundDiver emulation. Not sure if this is really better than doing a clean implementation...

It certainly helps for synths where the documentation might be missing or unclear, becaues the SoundDiver hex strings will work!

[markusschloesser]

@christofmuc ah shit, what a pity! :-) my sysex is already valid, as in identical to the strings in SD, but I thought I could skip the implementation part. Will continue my questions in the correct issue, thanks! BTW U at Superbooth?

[christofmuc]

Can't make Superbooth (again :-/), though I'd love to. But from Munich even with the sprinter train you need to full weekend, and we have other plans. I will try to put it more strict into the calendar for next year!

[markusschloesser]

that's a pity! 😕