azerbini / eamena_dev

Arches is a web-based, geospatial information system for cultural heritage inventory and management.
GNU Affero General Public License v3.0
1 stars 2 forks source link

Upload speed+eamena v1 v2 migration #136

Closed mradamcox closed 5 years ago

mradamcox commented 5 years ago

Description of Change

This pulls all v1-v2 migration logic into the eamena core branch. Additionally, it pulls in the upload_speed branch that @TeriForey developed, which was hugely helpful during the v1-v2 data migration.

v1-v2 Migration Process

The following is a basic breakdown of the v1-v2 data conversion and loading process that this PR adds.

Preparation

  1. Export all resources from the v1 database, either into json or jsonl format. They should be split into separate files by resource type.

  2. python manage.py packages -o install --eamena

    Create a fresh (empty) installation of whichever v2 package the data will be migrated to.

  3. python manage.py packages convert_v1_data --prepare

    Add the extra EFFECT_TYPE_x and SITE_FUNCTION_TYPE nodes and sets up the corresponding dropdown lists.

Loading E27 Resources

  1. python manage.py convert_v1_data -s <path_to_exported_json_or_jsonl_file>

    Use this command for E27 resources that have been exported from the v1 database. A lot of node-type-specific logic is implemented to conform to the E27 v2 graph. The result will be a .arches file.

  2. packages -o load_resources -s <path_to_arches_file>

    The file from step 1 is loaded just like a normal .arches file.

Loading E73 Resources

  1. python manage.py update_concept_uuids -s <path_to_exported_json_or_jsonl_file>

    Use this command for non-E27 resources that have been exported from the v1 database.

  2. packages -o load_resources -s <path_to_revised_json_or_jsonl_file>

    The file from step 1 is loaded just like a normal .json or .jsonl file.

Types of changes