search

ckan / ckanapi

A command line interface and Python module for accessing the CKAN Action API
Other
178 stars 74 forks source link

SOLR error: exceeded limit of maxWarmingSearchers=2 #118

Open jakubklimek opened 7 years ago

jakubklimek commented 7 years ago

When I load a large number of datasets with multiple processes using ckanapi like this: ckanapi load datasets -I /data/ckan.jsonl.gz -z -p 4 -c /etc/ckan/testing/development.ini

Eventually, a Solr error is thrown: ERROR [ckan.model.modification] Solr returned an error: (u'Solr responded with an error (HTTP 503): [Reason: Error opening new searcher. exceeded limit of maxWarmingSearchers=2, try again later.]',)

I could increase the number of WarmingSearchers in Solr or decrease the number of workers for the load, but IMHO the bigger issue here is whether it could be possible for bulk loading to have an option to commit to Solr only after loading all datasets, not one by one, as it still takes a long time this way (9 hours for 130k datasets to load using single worker)

torfsen commented 7 years ago

Similar issue w.r.t. reindexing: ckan/ckan#3482.