Create config store.

[kmonson]

The basic solution is that we should offer a platform configuration file store that is accessible to all agents. The configurations do NOT represent the configurations of currently running agents or necessarily imply a relationship with specific agents (running or not).

For each configuration file the platform store the following: Name - Configuration file name. All names must be unique. Category - Category of the configuration file. Defaults to "generic". Configuration - A string with the contents of the configuration file.

Interface: add_configuration(name, content, category="generic")

• adds/updates a configuration in the file store

get_configuration(name)

• gets the config string for the named config

get_configurations_for_category(category="generic")

• gets the list of configuration names for a category

get_categories()

• gets the current list of category names

remove_configuration(name)

• drops a configuration from the store

Notifications of changes and additions to the configuration store are published to the /config topic on the pubsub channel in the form: /config/[add|remove|update]/<category>/<name>

How configuration are named, formatted, and their contents are all considered conventions that are not enforced by the store.

[craig8]

The Platform (services/core/platform) agent has the capability to store a setting. The setting store is generic enough to do most of what you have in this spec. The main difference that I see is that name would be "key" (a unique key for the configuration). The content (value in the setting file) is free form as long as it can be serialized.

The main changes would be to add update and publish capabilities to the current agent.

[kmonson]

Ideally we would like to add support to dynamically reconfigure any agent on the fly after it has been started up. The problem is that an agent must be changed to support this feature. There is no reasonable way to reconfigure existing agents or have the agents retrieve their configuration from a store on the platform.

We should add a configuration subsystem to the base agent. This system will manage registering callbacks that are called in response to changes to changes to a configuration file.

A common pattern is to reference one configuration file from another. This causes issues with ordering agent callbacks and unneeded reconfiguration. We will need to add specific support for one configuration to reference another. The config store will then flag every configuration in a reference chain and signal the change up the chain. When the configuration is provided by the store the contents of the file will replace the reference. In most cases the agent will only need to listen for changes to the top of the chain.

Only json based config files can have references. When referencing another config file you must specify the type as either json or raw.

[kmonson]

The config store will examine json files recursively looking for file references. A reference will be a value string (not a key) that will probably take the form "json://" or "raw://<path in store".

If a file does not exist it will be returned as a None. Agents will have to know how to cope with this and signal an error condition via the heartbeat subsystem as needed.

We will use an simple event framework to trigger publishes after a file changes.

[kmonson]

After more thinking about this we need to detect/specify the config type (raw vs json) when we ingest the file into the store. Otherwise we don't know the type of file at the start of a chain of configurations. It is certainly possible to have agents that are configured with non-json files event though it is uncommon.

This changes the file reference to not include the file type as it's not needed. File references will take the form "file://".

The remaining question is how to make an agent aware of configurations available to it that are not necessarily referenced by another configuration. Previously I had addressed this with category option for the uploaded file. I'm not sure if this is the way to go.

Also, how does the Agent know where to start? Do we specify a canonical name for the main configuration or is it in specified some other way? Do we want to just let the agents work it out itself?

[kmonson]

Another problem:

How do we associate a configuration or a group of configurations with a specific agent?

The original idea was to store the configurations with the other platform agent data or in some other storage that associated the configurations with the uuid of the agent.

One major problem. What happens when the agents uuid changes? The agent is repackaged or is being run in developer mode from the command line by hand (or via the launch scripts in 'scripts/'). In both cases the uuid changes and the relation to the configuration in the store is lost.

Whatever new mechanism we come up with must work with multiple instances of the same agent.

[craig8]

So I have dealt with this issue with the VolttronCentralPlatform (vcp) and VolttronCentral (VC) agents. What I have come up with is a persistent piece on volttron central where there is a cache of uuid->vip addresses. Then when the VCP agent in the field connects to VC the VCP is assigned that uuid (e.g. there is an rpc method on VCP that is called set_platform_uuid). So the VCP agent uuid (on the field platform) is not used when communicating with VC.

For agents with known identities that shouldn't be an issue as they can have a unique store ala $VOLTTRON_HOME/data/master.driver/ or whatever that would be their location for putting data. This could be specified through an environmental variable before the agent process is started. As far as transient agent identities this does pose an issue.

[kmonson]

@craig8: @jhaack and I went over this on Friday and came up with a different solution: Bring back the Agent ID or possibly a configuration ID that is either specified by the agent or assigned by the platform if the Agent does not specify it. Platform assigned IDs could be set during installation (and used at runtime) or derived at run time if no installation or agent ID was provided.

Automatically derived IDs follow a predictable pattern that would allow multiple instances of the same agent to be configured and replaced without issue. The pattern is this:

<package>-<version>-<instance number>

If more than one instance is packaged and started up they will be assigned instance numbers based on the order they are started in.

Also agents could specify an ID which will allow for custom incrementing values. Something like this:

"listener-{n}"

"{n}" would then be automatically be replaced by an increasing value. Basically a custom version version of the automatically derived IDs above that are not tied to the version of the agent.

More than one instance of an agent with an id would cause the second instance to be rejected by the platform.

This ID would then be used to specify which configs are available in the config store.

[kmonson]

After more discussion with @craig8 we realized that we can use the VIP identity for this. We already assign the identity for agents that don't provide it themselves.

This already has the benefit of prevented duplicate identities and we already have a mechanism for assigning them at install and run time.

The only two hang ups are how we come up with default VIP identity and the order we consults our sources for the identity.

Currently we do it in this order:

Agent run time Install time file loaded at run time (IDENTITY file in installed agent directory)

Ideally we should do this:

Install time file loaded at run time (IDENTITY file in installed agent directory) Agent run time (for agents not installed but run from the command line during testing) Platform run time (ditto)

A default IDENTITY file will be created at install time or a VIP IDENTITY chosen at run time using the methods I suggested in my previous comment.

Agents that want/need to specify a VIP address at install time can supply an IDENTITY file (or the contents of said file) that is used instead of one generated by the platform at install time. The contents of the IDENTITY file can have the {n} string replacement if needed.

[jhaack]

Part of 4.0 #815 #779 #736 #218