ckan / ckanext-xloader

Express Loader - quickly load data into DataStore. A replacement for DataPusher.
GNU Affero General Public License v3.0
46 stars 50 forks source link

High availability setup #108

Open clauded opened 4 years ago

clauded commented 4 years ago

We're planning to deploy our CKAN setup with high availability (two CKAN servers, one db container, one solr container and one redis container). We would like to have xloader highly available as well.

ThrawnCA commented 2 years ago

HA XLoader is certainly possible. However, there is an increased risk of slowdowns and database transaction deadlocks when datasets have a high number of resources and different XLoader instances compete over them.

We have a setup that puts XLoader on its own server(s), and no special configuration is needed; it just needs the CKAN configuration file, and access to the database and Redis queue, just like CKAN would.