Impact of the new feature
WMAgent and management of jobs
Is your feature request related to a problem? Please describe.
Working on https://github.com/dmwm/WMCore/pull/11884 issue we faced large complexity to adjust pileup configuration files located in job's tarball. In order to make any change each individual pilupe configuration file should be extracted, changed and written back to tarball. This procedure can be significantly simplified if we'll have configuration file service.
Describe the solution you'd like
It seems to me it would be worth to explore pileup configuration service which will store pileup configuration files and provides API to change them. Here is I envision it:
</create>/path POST HTTP api to store pileup conf file, where path represents dataset (pileup) path
</delete>/path DELETE HTTP api to delete configuraiton file
</search>/path GET HTTP api to search for configuration file
NOTE: here I used optional </action> end-point (such as /<search>) for demonstration purposes but they are not strictly required if we'll follow HTTP method since each method define the action.
There are different ways to organize backend, e.g.
use key-value store, e.g. Redis. It will only store dataset-pileup-path vs pileup json file pairs
NoSQL database, e.g. MongoDB, if we want to have more comprehensive search over content of pileupconf.hson content
traditional RDBMS, e.g. ORACLE, if we want additional relationship stored in DB backend
But all of these solution can be wrapped into simple HTTP based service which can be installed on CMSWEB and used by different parts, from WMAgent to data-ops queries/requests, to query, update, create, and delete pileup configuration files.
The benefits of such service is many-fold;
Provide common place to store configuration files
Provide common place to query and modify configuration files
Independent service from WMAgent machinery
Easy integration in different worklfows machineries, e.g. data-ops can change configuration on a fly, i.e. before or after workflow is placed into WMA system
Describe alternatives you've considered
Do nothing and keep using sandbox tar ball which currently holds configuration files
Additional context
This is a nice example of pure micro-service architecture where a single piece, i.e. handling configuration files, is delegated to the service. Such service has no connection with other WMCore Micro-services or WMAgent components but will be used by later comprehensively everywhere.
Impact of the new feature WMAgent and management of jobs
Is your feature request related to a problem? Please describe. Working on https://github.com/dmwm/WMCore/pull/11884 issue we faced large complexity to adjust pileup configuration files located in job's tarball. In order to make any change each individual pilupe configuration file should be extracted, changed and written back to tarball. This procedure can be significantly simplified if we'll have configuration file service.
Describe the solution you'd like It seems to me it would be worth to explore pileup configuration service which will store pileup configuration files and provides API to change them. Here is I envision it:
</create>/path
POST HTTP api to store pileup conf file, where path represents dataset (pileup) path</delete>/path
DELETE HTTP api to delete configuraiton file</search>/path
GET HTTP api to search for configuration file NOTE: here I used optional</action>
end-point (such as/<search>
) for demonstration purposes but they are not strictly required if we'll follow HTTP method since each method define the action.There are different ways to organize backend, e.g.
But all of these solution can be wrapped into simple HTTP based service which can be installed on CMSWEB and used by different parts, from WMAgent to data-ops queries/requests, to query, update, create, and delete pileup configuration files.
The benefits of such service is many-fold;
Describe alternatives you've considered Do nothing and keep using sandbox tar ball which currently holds configuration files
Additional context This is a nice example of pure micro-service architecture where a single piece, i.e. handling configuration files, is delegated to the service. Such service has no connection with other WMCore Micro-services or WMAgent components but will be used by later comprehensively everywhere.