search

MI-DPLA / combine

Combine /kämˌbīn/ - Metadata Aggregator Platform
MIT License
26 stars 11 forks source link

StateIO export not working for Docker deploy #390

Closed ghukill closed 5 years ago

ghukill commented 5 years ago

Handful of errors, likely a combination of hosts configurations and/or OS packages that need to exist:

combine-celery_1      | [DEBUG] 2019-04-01 17:29:49,075 core.models._collect_related_components (8417): mongoexport cmd: mongoexport --db combine --collection record --out /home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7/record_exports/j22_mongo_records.json --type=json -v --query '{"job_id":22}'
combine-celery_1      | [2019-04-01 17:29:49,075: DEBUG/ForkPoolWorker-1] mongoexport cmd: mongoexport --db combine --collection record --out /home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7/record_exports/j22_mongo_records.json --type=json -v --query '{"job_id":22}'
combine-celery_1      | sh: 1: mongoexport: not found
combine-django_1      | [DEBUG] 2019-04-01 17:29:49,086 core.models.refresh_from_livy (149): querying Livy for session status
combine-celery_1      | [DEBUG] 2019-04-01 17:29:49,085 core.models._collect_related_components (8440): mongoexport cmd: mongoexport --db combine --collection record_validation --out /home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7/validation_exports/j22_mongo_validations.json --type=json -v --query '{"job_id":22}'
combine-celery_1      | [2019-04-01 17:29:49,085: DEBUG/ForkPoolWorker-1] mongoexport cmd: mongoexport --db combine --collection record_validation --out /home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7/validation_exports/j22_mongo_validations.json --type=json -v --query '{"job_id":22}'
combine-django_1      | [DEBUG] 2019-04-01 17:29:49,091 core.models.refresh_from_livy (177): session found, updating status
combine-celery_1      | sh: 1: mongoexport: not found
combine-celery_1      | [DEBUG] 2019-04-01 17:29:49,099 core.models._collect_related_components (8471): elasticdump cmd: ['elasticdump', '--input=http://localhost:9200/j22', '--output=/home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7/mapped_fields_exports/j22_mapped_fields.json', '--type=data', '--sourceOnly', '--ignore-errors', '--noRefresh']
combine-celery_1      | [2019-04-01 17:29:49,099: DEBUG/ForkPoolWorker-1] elasticdump cmd: ['elasticdump', '--input=http://localhost:9200/j22', '--output=/home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7/mapped_fields_exports/j22_mapped_fields.json', '--type=data', '--sourceOnly', '--ignore-errors', '--noRefresh']
combine-django_1      | [DEBUG] 2019-04-01 17:29:49,102 core.models.get_application_id (4383): found application matching Livy session id: local-1554119262088
combine-django_1      | [DEBUG] 2019-04-01 17:29:49,312 core.models.refresh_from_livy (149): querying Livy for session status
combine-django_1      | [DEBUG] 2019-04-01 17:29:49,317 core.models.refresh_from_livy (177): session found, updating status
combine-django_1      | [DEBUG] 2019-04-01 17:29:49,326 core.models.get_application_id (4383): found application matching Livy session id: local-1554119262088
combine-celery_1      | Mon, 01 Apr 2019 17:29:49 GMT | starting dump
combine-django_1      | [DEBUG] 2019-04-01 17:29:56,731 core.models.refresh_from_livy (149): querying Livy for session status
combine-django_1      | [DEBUG] 2019-04-01 17:29:56,737 core.models.refresh_from_livy (177): session found, updating status
combine-django_1      | [DEBUG] 2019-04-01 17:29:56,747 core.models.get_application_id (4383): found application matching Livy session id: local-1554119262088
combine-django_1      | [DEBUG] 2019-04-01 17:29:56,906 core.models.refresh_from_livy (149): querying Livy for session status
combine-django_1      | [DEBUG] 2019-04-01 17:29:56,913 core.models.refresh_from_livy (177): session found, updating status
combine-django_1      | [DEBUG] 2019-04-01 17:29:56,920 core.models.get_application_id (4383): found application matching Livy session id: local-1554119262088
combine-django_1      | [DEBUG] 2019-04-01 17:29:58,295 core.models.delete_gm (7934): removing gm: 462552c1524f4c259fc9ee7d336a1a1b
combine-django_1      | [DEBUG] 2019-04-01 17:30:09,556 core.models.refresh_from_livy (149): querying Livy for session status
combine-django_1      | [DEBUG] 2019-04-01 17:30:09,563 core.models.refresh_from_livy (177): session found, updating status
combine-django_1      | [DEBUG] 2019-04-01 17:30:09,568 core.models.get_application_id (4383): found application matching Livy session id: local-1554119262088
combine-celery_1      | Mon, 01 Apr 2019 17:30:09 GMT | Error Emitted => connect ECONNREFUSED 127.0.0.1:9200
combine-celery_1      | /usr/lib/node_modules/elasticdump/elasticdump.js:116
combine-celery_1      |           self.log('got ' + data.length + ' objects from source ' + self.inputType + ' (offset: ' + offset + ')')
combine-celery_1      |                                  ^
combine-celery_1      | 
combine-celery_1      | TypeError: Cannot read property 'length' of undefined
combine-celery_1      |     at self.input.get (/usr/lib/node_modules/elasticdump/elasticdump.js:116:34)
combine-celery_1      |     at version.err (/usr/lib/node_modules/elasticdump/lib/transports/elasticsearch.js:92:25)
combine-celery_1      |     at Request.baseRequest (/usr/lib/node_modules/elasticdump/lib/transports/elasticsearch.js:118:25)
combine-celery_1      |     at Request._callback (/usr/lib/node_modules/elasticdump/node_modules/lodash/lodash.js:10052:25)
combine-celery_1      |     at Request.requestRetryReply [as reply] (/usr/lib/node_modules/elasticdump/node_modules/requestretry/index.js:105:19)
combine-celery_1      |     at Request.<anonymous> (/usr/lib/node_modules/elasticdump/node_modules/requestretry/index.js:146:10)
combine-celery_1      |     at self.callback (/usr/lib/node_modules/elasticdump/node_modules/request/request.js:185:22)
combine-celery_1      |     at Request.emit (events.js:189:13)
combine-celery_1      |     at Request.EventEmitter.emit (domain.js:441:20)
combine-celery_1      |     at Request.onRequestError (/usr/lib/node_modules/elasticdump/node_modules/request/request.js:881:8)
combine-celery_1      | [DEBUG] 2019-04-01 17:30:09,651 core.models._finalize_export_manifest (8616): writing Record Group: sandbox to export_manifest
combine-celery_1      | [2019-04-01 17:30:09,651: DEBUG/ForkPoolWorker-1] writing Record Group: sandbox to export_manifest
combine-celery_1      | [DEBUG] 2019-04-01 17:30:09,652 core.models._finalize_export_manifest (8616): writing Organization: oai handling to export_manifest
combine-celery_1      | [2019-04-01 17:30:09,652: DEBUG/ForkPoolWorker-1] writing Organization: oai handling to export_manifest
combine-celery_1      | [DEBUG] 2019-04-01 17:30:09,653 core.models._finalize_export_manifest (8616): writing OAI endpoint: Ouroboros to export_manifest
combine-celery_1      | [2019-04-01 17:30:09,653: DEBUG/ForkPoolWorker-1] writing OAI endpoint: Ouroboros to export_manifest
combine-celery_1      | [DEBUG] 2019-04-01 17:30:09,654 core.models._finalize_export_manifest (8616): writing Ramsey 1, Job #22 to export_manifest
combine-celery_1      | [2019-04-01 17:30:09,654: DEBUG/ForkPoolWorker-1] writing Ramsey 1, Job #22 to export_manifest
combine-celery_1      | [DEBUG] 2019-04-01 17:30:09,655 core.models.package_export (8564): compressiong exported state at /home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7
combine-celery_1      | [2019-04-01 17:30:09,655: DEBUG/ForkPoolWorker-1] compressiong exported state at /home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7
combine-celery_1      | [DEBUG] 2019-04-01 17:30:09,661 core.models.package_export (8577): archive /home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7.zip created in 0.005181312561035156s
combine-celery_1      | [2019-04-01 17:30:09,661: DEBUG/ForkPoolWorker-1] archive /home/combine/data/combine/stateio/exports/3f99f2d36c7340ca90db0c95e32989b7.zip created in 0.005181312561035156s
combine-celery_1      | [DEBUG] 2019-04-01 17:30:09,667 core.models.export_state (8147): total time for state export: 20.63993263244629
combine-celery_1      | [2019-04-01 17:30:09,667: DEBUG/ForkPoolWorker-1] total time for state export: 20.63993263244629
combine-celery_1      | [2019-04-01 17:30:09,668: INFO/ForkPoolWorker-1] Task core.tasks.stateio_export[f55c81b1-b52e-4bf4-9319-2682e002e72e] succeeded in 20.65282006491907s: None
ghukill commented 5 years ago

Fixed.