Identify and adopt concurrent IO whenever beneficial

[amaltaro]

Impact of the new feature WMCore in general

Is your feature request related to a problem? Please describe. Many components and services in WMCore rely on sequential data processing and network-based requests. In order to make the system more performant and scalable, we better adopt parallel/concurrent programming in critical sections of our code base, especially when it involves I/O operations (like HTTP requests).

A concurrent pattern usually requires more lines of code, with smarter error handling, timeout control and the use of the RESTful endpoints instead of client wrappers. So questions like: a) is concurrency required in the current scenario/load? b) is concurrency going to be required in 5 years from now? c) does the end user rely on the execution of this sequential/concurrent implementation (thus blocking an HTTP thread/request)?

Describe the solution you'd like This is a meta-issue and will eventually require sub-tasks to be created.

Some of the required tasks are: a) identify components within WM that need to provide a better performance b) within those components, can we benefit from concurrent I/O c) create common concurrent implementations to be shared among the project

Describe alternatives you've considered Remain with the sequential implementation

Additional context None

[vkuznet]

+1, to keep myself notified about this issue.