Core functionality for WM microservices v2

[amaltaro]

Impact of the new feature Meta issue for the next generation of WM microservices

Is your feature request related to a problem? Please describe. Yes, we want to modernize the microservices stack, make them independent of WMCore, more modular, maintainable and friendly to core and non-core developers.

Describe the solution you'd like Provide a new repository with the core functionality for a new generation of WM microservices. Some of the requirements for this new project/repository are:

• not to depend on WMCore (if needed, we should replicate module(s) over here while being mindful with real core modules)
• use a modern software stack (e.g. adopt Flask instead of CherryPy)
• support any Python version greater or equal than 3.8
• unit tests must be an integral part of this project

A list of sub-tasks will follow and get added to this ticket.

• [ ] create new github repository (which name? WMMSCore, WMMS, WMMicro, ?) with a basic structure https://github.com/dmwm/WMCore/issues/11823
• [ ] create a CI pipeline based on GitHub actions (unit tests, code coverage, static linting, code style) https://github.com/dmwm/WMCore/issues/11824
• [ ] create a CD pipeline based on GitHub actions (create setup.py, package and upload to PyPi) https://github.com/dmwm/WMCore/issues/11825
• [ ] create HTTP core functionality module (http methods, http header parsing - CRIC, JSON/NJSON output, standard output structure) https://github.com/dmwm/WMCore/issues/11826
• [ ] create core/common "/status" endpoint to return a common status information of the microservice https://github.com/dmwm/WMCore/issues/11827
• [ ] create core/common "/metrics" endpoint to return a common metrics of the microservice (based on psutils?) https://github.com/dmwm/WMCore/issues/11828
• [ ] provide a base class to manage connections to MongoDB and perform CRUD operations https://github.com/dmwm/WMCore/issues/11829

On the tooling scope, it's mostly about providing tools/hooks/scripts to auto-generate templates/skeletons, allowing us to easily bootstrap a brand new WM microservice. Required functionalities are:

• [ ] create a GH repository and its basic content structure, including a basic and working web-server inheriting the common /metrics and /status REST APIs https://github.com/dmwm/WMCore/issues/11830
• [ ] basic CI and CD pipelines https://github.com/dmwm/WMCore/issues/11831
• [ ] auto-generation of the Dockerfile https://github.com/dmwm/WMCore/issues/11832
• [ ] auto-generation of a basic configuration/secrets file https://github.com/dmwm/WMCore/issues/11833
• [ ] auto-generation of the required frontend/ingress rules https://github.com/dmwm/WMCore/issues/11834
• [ ] auto-generation of the deployment files (kubernetes manifests) https://github.com/dmwm/WMCore/issues/11835
• [ ] documentation and examples for the creation of a new microservice and CERN monitoring setup https://github.com/dmwm/WMCore/issues/11836

Describe alternatives you've considered Glad to hear of any alternatives

Additional context None

[amaltaro]

I didn't see we already had a meta issue for this: https://github.com/dmwm/WMCore/issues/11815 but given that this one is now much more advanced in content than 11815, I am closing that one out.