Refactor to use a single Config Datastore for all clients

[thelontx]

Make.com has limits on the number of datastores per Organization tied directly to the number of operations allowed by the subscription. 1 datastore for every 1k of monthly operations. This is true even if the amount of data stored inside the datastore is minimal. Each data store requires a minimum of 1MB of storage space to be allocated.

Currently, the plan is to use NPC's Make.com Org for all future clients as separate Teams. Without other changes, the subscription will need to be increased to allow more operations (and datastores).

Each WAQM client requires 1 Config datastore. Quickbooks Desktop clients require an additional 1 datastore for the IIF TMP processing data store.

Idea: Use 1 central datastore to hold the Config for all clients. Use Webhooks/HTTP Send to access the appropriate Config record for each client.

Factors to consider:

• Changes to all WAQM scenarios
• Security/protection to ensure the Config does not get mixed up between clients - this could result in disclosing financial info to the wrong parties.
• any impact of parallel processing on reading Config from 1 datastore?
• Nice to have: better UI or technique to view/filter/edit records in a single Datastore for all clients and versions of their records

[thelontx]

New Make.com feature of custom variables may make this easier? Currently, it supports long text strings (for JSON), but does not support arrays or collections (specification). Need to think through and see if this saves effort/operations/complexity or not.

Examples:
a) When a core scenario runs, should it always check and get the latest version of the config? If yes, it effectively needs to read the central datastore (via HTTP/webhook) every run anyway. Storing the JSON config in the client's environment variable might be a redundant step.

b) Conversely, if the central repository (in NPC) is only checked periodically, how does someone trigger it to use new config from NPC?