Open clauded opened 4 years ago
HA XLoader is certainly possible. However, there is an increased risk of slowdowns and database transaction deadlocks when datasets have a high number of resources and different XLoader instances compete over them.
We have a setup that puts XLoader on its own server(s), and no special configuration is needed; it just needs the CKAN configuration file, and access to the database and Redis queue, just like CKAN would.
We're planning to deploy our CKAN setup with high availability (two CKAN servers, one db container, one solr container and one redis container). We would like to have xloader highly available as well.