search

ORNL / DataFed

A Federated Scientific Data Management System
https://ornl.github.io/DataFed/
Other
18 stars 14 forks source link

Redesign Core Service for Fail-over and Load Balancing #769

Open dvstans opened 3 years ago

dvstans commented 3 years ago

Need to separate global orchestration functions from regular client processing in core, then allow multiple core client worker processes to be run concurrently for load balancing and fail-over. The task processing should also be changed to permit concurrent processes but will required a shared queue architecture to ensure tasks are fed to workers without collisions (some tasks can run concurrently, others cannot). This work cannot be started until the core server communication is refactored to use TLS instead of zeromq.

dvstans commented 1 year ago

This sounds like DataFed v2 - probably should close this ticket for v1.