Create MongoDB-based MSOutput configuration to replace the Unified configuration

[amaltaro]

Impact of the new feature ReqMgr2MS, MSOutput

Is your feature request related to a problem? Please describe. All our MicroServices are currently relying on a few parameters from the Unified configuration, which means we need to keep fetching the Unified configuration such that Unified and MicroServices are in sync with the same configuration. This is somehow fragile though, someone can change the Unified configuration parameters and/or data structure, and it could possibly break MicroServices.

Describe the solution you'd like We need to investigate which parameters are actually used from the Unified configuration (and find other parameters that would be useful to allow Ops to tweak them) and make our own service configuration (replacing part of those parameters already available in the python-based service configuration).

This involves many items though:

• finding out all the required parameters from Unified and copy their values over to the WMCore-based configuration;
• identify other service parameters that should be exposed to Ops, such that they can change them if needed;
• persist this configuration in the database (for MSOutput case, it's likely MongoDB);
• expose this configuration document through REST APIs (CRUD model)
• apply some validation on the field names and values.

Describe alternatives you've considered We could also create one single configuration file for all the MicroServices. Makes it simpler, but likely more error prone.

Additional context none

[todor-ivanov]

@amaltaro Alan, a general question here: Does this change relate only to MSoutput? I was having the impression that the Unified Config is used in other places too. So when it comes to the following:

persist this configuration in the database (for MSOutput case, it's likely MongoDB);

Should I understand that we will be using a separate (or at least more than one) database for every other service to use the unified config? Can you give some more details on that.

[amaltaro]

That's actually a good question! I wanted to stop adding things on top of CouchDB and start adopting more MongoDB instead. However, in this case, since all Microservices (MSOutput and MSTransferor) already have an interface to the CouchDB reqmgr_aux db, it might be easier for us to replace the current UNIFIED_CONFIG - which is a sync from Unified repository - to keeping our own set of parameters (and allowing some users to read/write/update it).

Another alternative would be to add it to MongoDB, add this dependency in MSTransferor, and then both services can read it from there instead. Of course, we would have to implement the REST CRUD APIs for it. I guess it should be discussed in our next chat?

[todor-ivanov]

Thanks @amaltaro ! Just to mention here, we may then consider a centralized instance of MongoDB.