search

bcgov / BCHeritage

Branch level repository for documentation and product issues.
Apache License 2.0
4 stars 0 forks source link

Import real dataset to Legacy #109

Closed bferguso closed 2 years ago

bferguso commented 2 years ago

This is still a manual process, not part of the ansible scripts. Started the import at approximately 21:30 on May 4th

bferguso commented 2 years ago

Still running at 08:30 May 5th. It may be better to run import on a dev machine and then export/import the data (or entire database) on target server.

bferguso commented 2 years ago

Performed the following steps:

  1. imported CSV data on QED dev machine
  2. exported data as JSON.
  3. transferred JSON files to Legacy
  4. Imported JSON data into Legacy project Import failed on BC Fossil Sample after approx. 59617 samples with the errors below. There was still significant space on all disks. We don't have access to look deeper into the issue on LEGACY as we don't have the necessary rights. Need to determine whether we should try to resolve this on LEGACY or wait until the trial GCP or Tourism target enviornment.
2022-05-07 15:01:31,166 arches.app.search.search WARNING  2022-05-07 15:01:31.163850: WARNING: failed to index document: {'value': 'Spray', 'nodeid': '477535ec-9fb1-11ec-9db6-5254008afee6', 'nodegroupid': UUID('47753100-9fb1-11ec-9db6-5254008afee6'), 'tileid': UUID('341199ba-a631-4015-96be-7afa2b90003f'), 'resourceinstanceid': UUID('5dc21f96-fcdf-4747-93f1-20640481a1aa'), 'provisional': False}
Exception detail: TransportError(429, 'cluster_block_exception', 'index [arches_fossils_terms] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];')

2022-05-07 15:01:43,303 arches.app.search.search WARNING  2022-05-07 15:01:43.303279: WARNING: failed to index document: {'resourcexid': UUID('12cf1136-cd66-11ec-9de7-0050568377a0'), 'resourceinstanceidfrom': UUID('aeae0d43-275a-4483-8117-b3ec549c5259'), 'resourceinstancefrom_graphid': UUID('df3ee1ae-9c1c-11ec-964d-5254008afee6'), 'resourceinstanceidto': UUID('f9d0e49f-e694-4c10-8649-dfeddbf9be00'), 'resourceinstanceto_graphid': None, 'notes': '', 'relationshiptype': 'http://www.cidoc-crm.org/cidoc-crm/L54i_is_same-as', 'inverserelationshiptype': 'http://www.cidoc-crm.org/cidoc-crm/L54i_is_same-as', 'tileid': UUID('8dccf7d4-5504-41d3-9118-a7809a4df8e8'), 'nodeid': UUID('5e4b75ba-a079-11ec-bc6e-5254008afee6'), 'datestarted': None, 'dateended': None, 'created': datetime.datetime(2022, 5, 6, 12, 58, 2, 759170), 'modified': datetime.datetime(2022, 5, 7, 15, 1, 43, 277488)}
Exception detail: TransportError(429, 'cluster_block_exception', 'index [arches_fossils_resource_relations] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];')

2022-05-07 15:02:00,715 arches.app.search.search WARNING  2022-05-07 15:02:00.715859: WARNING: failed to index document: {'resourcexid': UUID('12cf1136-cd66-11ec-9de7-0050568377a0'), 'resourceinstanceidfrom': UUID('aeae0d43-275a-4483-8117-b3ec549c5259'), 'resourceinstancefrom_graphid': UUID('df3ee1ae-9c1c-11ec-964d-5254008afee6'), 'resourceinstanceidto': UUID('f9d0e49f-e694-4c10-8649-dfeddbf9be00'), 'resourceinstanceto_graphid': None, 'notes': '', 'relationshiptype': 'http://www.cidoc-crm.org/cidoc-crm/L54i_is_same-as', 'inverserelationshiptype': 'http://www.cidoc-crm.org/cidoc-crm/L54i_is_same-as', 'tileid': UUID('8dccf7d4-5504-41d3-9118-a7809a4df8e8'), 'nodeid': UUID('5e4b75ba-a079-11ec-bc6e-5254008afee6'), 'datestarted': None, 'dateended': None, 'created': datetime.datetime(2022, 5, 6, 12, 58, 2, 759170), 'modified': datetime.datetime(2022, 5, 7, 15, 2, 0, 685736)}
Exception detail: TransportError(429, 'cluster_block_exception', 'index [arches_fossils_resource_relations] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];')

2022-05-07 15:02:09,989 arches.app.search.search WARNING  2022-05-07 15:02:09.989406: WARNING: failed to index document: {'resourcexid': UUID('12cf1136-cd66-11ec-9de7-0050568377a0'), 'resourceinstanceidfrom': UUID('aeae0d43-275a-4483-8117-b3ec549c5259'), 'resourceinstancefrom_graphid': UUID('df3ee1ae-9c1c-11ec-964d-5254008afee6'), 'resourceinstanceidto': UUID('f9d0e49f-e694-4c10-8649-dfeddbf9be00'), 'resourceinstanceto_graphid': None, 'notes': '', 'relationshiptype': 'http://www.cidoc-crm.org/cidoc-crm/L54i_is_same-as', 'inverserelationshiptype': 'http://www.cidoc-crm.org/cidoc-crm/L54i_is_same-as', 'tileid': UUID('8dccf7d4-5504-41d3-9118-a7809a4df8e8'), 'nodeid': UUID('5e4b75ba-a079-11ec-bc6e-5254008afee6'), 'datestarted': None, 'dateended': None, 'created': datetime.datetime(2022, 5, 6, 12, 58, 2, 759170), 'modified': datetime.datetime(2022, 5, 7, 15, 2, 9, 979111)}
Exception detail: TransportError(429, 'cluster_block_exception', 'index [arches_fossils_resource_relations] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];')

2022-05-07 15:02:18,843 arches.app.search.search WARNING  2022-05-07 15:02:18.842947: WARNING: failed to index document: {'resourcexid': UUID('12cf1136-cd66-11ec-9de7-0050568377a0'), 'resourceinstanceidfrom': UUID('aeae0d43-275a-4483-8117-b3ec549c5259'), 'resourceinstancefrom_graphid': UUID('df3ee1ae-9c1c-11ec-964d-5254008afee6'), 'resourceinstanceidto': UUID('f9d0e49f-e694-4c10-8649-dfeddbf9be00'), 'resourceinstanceto_graphid': None, 'notes': '', 'relationshiptype': 'http://www.cidoc-crm.org/cidoc-crm/L54i_is_same-as', 'inverserelationshiptype': 'http://www.cidoc-crm.org/cidoc-crm/L54i_is_same-as', 'tileid': UUID('8dccf7d4-5504-41d3-9118-a7809a4df8e8'), 'nodeid': UUID('5e4b75ba-a079-11ec-bc6e-5254008afee6'), 'datestarted': None, 'dateended': None, 'created': datetime.datetime(2022, 5, 6, 12, 58, 2, 759170), 'modified': datetime.datetime(2022, 5, 7, 15, 2, 18, 833563)}
Exception detail: TransportError(429, 'cluster_block_exception', 'index [arches_fossils_resource_relations] blocked by: [TOO_MANY_REQUESTS/12/disk usage exceeded flood-stage watermark, index has read-only-allow-delete block];')