Break buttplug device config into directory tree with smaller files

[qdot]

Right now, the Buttplug Device Config system consists of 2 monolithic files:

• The Core Device Config file, which is maintained by the core team and contains all of the base device configs
• The User Device Config file, which is maintained either by hand or via Intiface Central

Both of these files consist of a single, large tree of either protocols and their various communication configurations and device definitions, or user configurations (which are defined under the same tree type and schema as the core).

The solution was going to be moving to sqlite, but that is turning out to be way more of a beast to implement than I planned on. Distributing updates of the DB to the user will be very difficult, it won't be easy to pack into the library, it makes WASM compilation extremely difficult, etc... It also still gives us a single file to write to and possibly overwrite, which is our whole problem in the first place. We also don't have THAT many keys/identifiers/foreign relations, and data replication here barely costs kilobytes, so an RDBMS is ending up being overkill.

Therefore, I'd like to break both of these files up into a tree of directories/files, for separate reasons.

Things I'd like to fix

• User config data may get edited by hand or programmatically, so making files have the smallest units of data possible would be nice so we aren't overwriting everything like we do when we have one file.
• The core device file is getting really big, and is difficult to build parsers for in the GUI.
• I'd like to keep text files around because they're easily hand editable both by devs and users without me having to build more UI.
• The RDBMS solution was also a try at fixing our index collision issue for #552 but was overly complex for what we need. This bug is still a step toward fixing that issue, but we're still going to have to change formats later to really fix it.

User Device Config File

The user device config file is the most important part of this. Right now, if the user makes hand edits to the device config file, Intiface Central will stomp on it when it saves other user data, because I haven't built in all of the proper reserialization yet. Not to mention, it'd be nice if we could just change as little as possible within files when writing out data, as most of our keys are static anyways.

I'd like to break this out so that keys are directories, and the leaf data is files. This directory tree would look something like:

- user_config
  - lovense
    - comms
      - websocket.json
      - serial.json
    - configs
      - [identifier name, see potential issues below for how this might go badly]
        - [device address, somehow sanitized for directory naming]
          - user_config.json

This would allow us to save only the files we need to change when making GUI edits to anything, which will still allow hand editing while we wait on me to actually build the UI out for full configuration.

For user_config.json, I'd like to be able to store the device limits in that file. I'm tempted to maybe just store the device configuration as a whole and use that as our source of truth for a device with the (protocol, identifier, address) tuple match.

Core Device Config

As far as we're concerned, the core device config file is ours to do with what we want. We can stomp it at any time, and do not expect users to change it.

There are still problems with it though. The file itself is getting huge, and it's hard to reason about things quickly within it. It's also difficult to build parsers (lots of nodes 'til we hit leaves in the graph, all of which have to be defined), and the json schema for it is quite complicated.

We could take a similar approach to the user device config, with a directory tree that looks something like this:

- protocols
  - aneros
    - comms 
      - btle.json
    - configs
      - default.json
      - configs.json  [See potential issues for why there isn't an identifier directory here like in user config]
  - nintendo-joycon
    - comms
      - hid.json 
  - etc

Potential Issues

• Right now it's pretty easy to ship the single json device config file built into the library itself. Changing to a directory based system for the core device config makes that far more difficult, so that may not be worth it for the core device config.
• Not all device identifiers make good directory names. There's a few that have whitespace or other weirdness that we'd need to figure out, and I'm not quite sure how to do that. We could also convert to something like hashes of names, but that severely reduces human readability.
• Making a directory-per-device-identifier means a LOT of duplication that we currently handle in the core device config file. Brands like foreo and satisfyer would have tons of repetition.
• I'm still not quite sure how I feel about the inheritance of specific configuration to default in the core device file when we want to change a name but inherit device message attributes. It's handy for saving repetition in the file but it's difficult to model in software. This was a big part of why I was trying the RDBMS solution in the first place, since it was easier to model there with foreign relations. This could be overcome by just treating the user device config file generated on connection (the thing that holds the user device name, device limits, etc) as the device config itself, and having the base config in there.
  • If I save device config data into a user device config file, how often do we change base data? The only issue I could see here is possibly coherence: if we make a mistake in the base config (wrong limit or something) and update that, how do we know to update (or delete) the user's corresponding config file as well?

[qdot]

@blackspherefollower pinging you for thoughts.

[qdot]

Ok, having had a day to mull this over: Still on board with my ideas about user config, base device config I think we can punt on for a while longer. We can start with the file/directory tree for user config, and work from there. The user config and internal representations within the system are really all we're worried about anyways, dividing up the base config was more of a "it might be easier in the long run" but I'm not really convinced that yak needs immediate shaving.

[qdot]

My notes from trying to come up with a design for this.

Device Config Outline

Goals

• Stop stomping on user configs
• Allow more variability for saving user configs

  Communication Specifiers

• btle
• serial
• websocket
• usb
• hid

Device Configurations

• Name
• Identifier - string - Optional - None means default
• Address - string - Optional - User devices only
• Features
  • Description - String
  • Type - String
    • Maps to list of feature types, which is combined actuator/sensor list from Buttplug v3
  • Step Range - int tuple - Optional - For actuators
  • Step Limit - int - Optional - User devices only
  • Value Range - array of int tuples - Optional - For sensors
  • Messages
    • List of Message Types
    • Currently just name of message type, will iterate on per-message capabilities in next version.
• Display Name - Optional - User devices only
• Device Index - Optional - User devices only
• Allow - Optional - User devices only
• Deny - Optional - User devices only

Types of Device Configurations

• Core Device Config
  • Distributed by core team, included with library, contains no optional user fields
• User Created Device Config
  • For variable config devices, like OSR-2 with serial port and per-build feature requirements. Contains no user fields, but is also not in core device config.
• User Generated Device Config
  • Generated on device connect

Where things are saved

• Core Device Config
  • Same as usual, single monolithic config file
  • in Intiface - [IntifaceDir]/config/buttplug-device-config.json
• User Created Communication Specifiers:
  • Needs to be file-per-protocol so we're rewriting as little as possible from upper layers
  • in Intiface - [IntifaceDir]/config/user/comm/[ProtocolName]/[CommSpecName].json
    • i.e. [IntifaceDir]/config/user/comm/tcode/serial.json
    • This still means we may have lists per file, in case someone has multiple serial ports, websocket connectors that use name lists for identification, etc.
• User Created Device Config
  • Needs to be file-per-protocol so we're rewriting as little as possible from upper layers
  • in Intiface - [IntifaceDir]/config/user/config/whatever-the-user-names-it.json
    • We'll keep the protocol/config format from the core device config in this file, so filename doesn't really matter here, as we'll get the protocol/identifier part of the full identifier tuple from there.
• User Generated Device Config
  • Needs to be file-per-protocol so we're rewriting as little as possible from upper layers
  • in Intiface - [IntifaceDir]/config/user/device/[DeviceIdentifierHash].json
    • While it'd be nice to have a human readable version of the (protocol, identifier, address) identifier, and I may try a formatter for that, making it so it'll work with paths is annoying (having to replace colons in bt addresses, etc).

Buttplug Interface Changes Needed

• Buttplug/Intiface Engine should specify a user config directory, rather than a file

[qdot]

Actually, nah. Didn't need any of this. Just rebuilding the configs to actually consider ease to serialization and user configurability.