search

cms-PdmV / cmsPdmV

CERN CMS McM repository
4 stars 10 forks source link

Migrate McM to Python 3 #1124

Open ggonzr opened 9 months ago

ggonzr commented 9 months ago

Update all the McM source code to drop the support for Python 2.

Is your feature related to a problem?

This application still uses Python 2 as the main reference version for its code base. This version has been deprecated since January 2020 and it is required to update it. Also, another motivator is the ongoing OS migration to AlmaLinux 9, newer distributions don’t provide this old package anymore.

Describe the solution you'd like

Initial plan

  1. Build an environment (using containers) to deploy automatically the application as it currently is with all its dependencies.
  2. Include CI actions to build automatically the environment and execute test cases.
  3. Start coding a test suite to test all the API endpoints offered by the application.
  4. Iterate between 2 and 3 until a high test coverage has been reached.
  5. Use the Python 2 to 3 transpiler to migrate as much as possible code.
  6. Perform the required corrections so that no test fails.
  7. Format all the source code using black
  8. Include pylint as a static code analyzer.
  9. Improve the log rotation.
  10. Upgrade all the current packages to its latest version.
  11. Change the container base image to the latest Python 3.11 version
  12. Deploy to production.

More details will be included in the comments below.

Linked ticket: PDMVDEV-5

ggonzr commented 3 months ago

Related to the next tests cases, I think we should continue with coding more scenarios similar to #1137 focusing on other MC samples that include not frequent steps like DIGI, RECO or others that split the GEN and SIM steps like: B2G-chain_RunIISummer19UL16wmLHEGEN_flowRunIISummer19UL16SIM_flowRunIISummer19UL16DIGIEpsilonPU_flowRunIISummer19UL16HLT_flowRunIISummer19UL16RECO_flowRunIISummer19UL16MiniAOD_flowRunIISummer19UL16JMENtuplesNanoAOD-00002. Then, we could also start coding other tests for checking endpoints that are not directly related with the execution of the previous cases like: gen_logs. @lmoureaux, what do you think about this approach?

lmoureaux commented 3 months ago

The approach sounds reasonable. Do we already test endpoints retrieving information about existing requests like the inject/test commands?

ggonzr commented 3 months ago

No, not yet. The only group of tests that exists is for the campaign entity