SuLab / Krusty

neo4j <-> wikibase import/export tool
1 stars 2 forks source link

wikibase dump out problem due to wdqs crashes #9

Open NuriaQueralt opened 5 years ago

NuriaQueralt commented 5 years ago

WDQS CRASHES

look at the wdqs log: $ docker-compose logs -f --tail 200 wdqs

show what is running in the container: $ docker ps

The dumping out of wikibase by Krusty::wd_to_neo4j.py makes wdqs crash when executed from aws. Consequently the dump out is unsuccessful. No problem when executed from our servers. Looking at the wdqs log it shows something about memory errors:

wdqs_1 | Caused by: com.bigdata.rwstore.sector.MemoryManagerClosedException: null

WDQS RE-START

solution 1: restart the query service via docker-composer

cd /home/ubuntu/wikibase docker-compose restart wdqs docker-compose restart wdqs-updater

solution 2 (harder crash): restart docker and then the query service via

docker-compose down service docker restart docker-compose up -d

THE PROBLEM

It seems that we are suffering from RAM problems. The docker-compose setup requires more than 2GB of available RAM to start. While being developed the dev machine has 4GB of RAM.' according to: https://github.com/wmde/wikibase-docker/blob/master/README-compose.md

aws RAM:
$ free -mh total used free shared buff/cache available Mem: 7.8G 5.7G 589M 121M 1.5G 1.6G Swap: 0B 0B 0B

our servers RAM: $ free -mh total used free shared buff/cache available Mem: 62G 5.9G 9.3G 2.2M 47G 56G Swap: 63G 0B 63G

Possible solutions: