Update patch file format not use .fxp

[mkruselj]

Yup, exactly. So that we don't have to faff around with FXP header and whatnot. Allows regular text editing, proper XML (or JSON or whatever else) formatting, etc.

Encode wavetables as base64.

[mkruselj]

From Paul:

"So the FXP parser does these things: it checks the binary header, unpacks the XML and unpacks the wavetables. If instead the wavetables were in the XML, we could skip the outer binary. Hence .surgepatch as an XML file is a really easy addition, because it is load_xml() detecting a WT field. And then .surgepatch file vectoring there directly, rather than through load_patch(). Add then format to the scanner and drag and drop. And add a "Save as .surgepatch" option to user defaults."

To which I add "not a fan of superlong extensions, we should either use bog-standard .xml as extension, or just .srg.

[turian]

I would still be interested in this.

One difficulty I've found is that the XML in the FXPs has a lot of inconsistencies, so it's hard to verify programmatically that the XML was indeed fully and correctly parsed. You can't simply XML to JSON to XML and compare the roundtrip.

[baconpaul]

what sort of inconsistencies? it's a pretty simple model I thought.

[turian]

@baconpaul If you like, I can share some code. Check out https://github.com/turian/fxp2json

I have tried using a variety of libraries to roundtrip from FXP XML to JSON and back to XML:

    "bs4_json",
    "lxml_etree_iterparse_json",
    "lxml_etree_json",
    "lxml_etree_sax_json",
    "pytinyxml2_json",
    "xmltodict_json",

I have gotten them so they all create the same JSON. However, when I convert to XML and save the FXP, surge just crashes on loading the FXP.

I have tried super super hacky ways of creating the same XML, (see branch lxml) including:

• Stripping whitespace between elements

• Making surge that empty elements are written like this: <tags />

• Making sure that strings always use double quotes, EXCEPT the initial XML tag which should have single quotes

• Making sure certain special characters are written like this:

  def handle_special_characters(xml_str: str) -> str:
  # Replace specific characters with their XML entities
  xml_str = xml_str.replace("
", "
")
  xml_str = xml_str.replace("
", "
")
  return xml_str

• Adding trailing space:

  # Add space before the trailing slash for self-closing tags
  xml_str = xml_str.replace("/>", " />")
  xml_str = xml_str.replace("?>", " ?>")
  xml_str = xml_str.replace("\n", "")

With that said, I get 22% of the way through factory patches without being able to 100% round-trip the XML, in this file I break: /Library/Application Support/Surge XT/patches_3rdparty/Luna/Plucks/Acidic Feedback Plonk.fxp

Because this is one of the XMLs in which the element order differs. I tried a separate branch where I tried to maintain the element order but that also broke.

[turian]

@baconpaul Is the XML or patch format documented somewhere? Even the JSON output is inconsistent, sometimes things have "children" with "modrouting" (can they always?), etc.

I want to be able to generate random FXPs that are valid AND fully grok what is the fully expressive XML

[mkruselj]

The format is not really documented anywhere, apart from just having the source code at hand (relevant code is here). And yes things can be conditional especially regarding modulation assignments.

[baconpaul]

The format is not only not documented it’s also not something we support as a public api. So even if you make this work no guarantee it works at 1.4. The xml being human readable is deceptive at best

[baconpaul]

Also with xt2 we will definitely not be using human readable xml as a patch format. It will be more like short circuit which is msgpack serializarion of c++ objects. I think this issue is probably a “won’t do” from surge perspective (but we can keep it open for discussion if you want)

[baconpaul]

Oh also if you are getting crashes with your xml, do a debug build and run under the debugger to see which element blows up is my advice. We are defensive against optionals missing but not against required fields missing

[baconpaul]

One last thought - the format does have a streaming version. Older streaming versions are very different than new ones and we have code which translates

So like at version 14 the constant for every filter changed and we gave a big map in softeware m

this is also not documented except in code (although things like that are tested)

[turian]

@baconpaul @mkruselj I have been able to successfully generate XML and squeeze it into an FXP that doesn't crash Surge.

Nonetheless, I would be interested in understanding what is a complete (nothing missing, nothing additonal) format for the patches, and maybe even contributing docs. Are you open to helping me? I've been making progress on inverse synthesis (recovering the parameters for a particular sound) on synths that DO have clearly defined patch formats. However, I can't attack Surge until I know that I have a full definition of the patch format.

You can email me (lastname at gmail's email) or we can chat on discord?