Documenting Good Practices for "Arches Data Lifecyle" Operations

[ekansa]

I'd love to compile some of your collective experience about good practices in curating data with Arches over different time scales. It would be great to share knowledge about “gotchas” (where things can go wrong, and how to avoid problems), tips, and other good practices people have learned. Here are some issues that I'd like to cover:

• Maintaining data integrity
• Good data practices relating to upgrading Arches (especially major version upgrades)
• Security, managing sensitive information
• Database backup strategies and approaches
• Database archiving (with outside digital repositories)
• Useful command line utilities
• PostgreSQL utilities
• Use of cloud computing database services (Amazon RDS, etc.)

I’d like to gather those good practices for the Arches documentation so we can make Arches managed data easier to maintain. Here's a link to the documentation branch I'm using to compile this guidance: https://github.com/archesproject/arches-docs/blob/data_life_7_5/docs/administering/data-life-cycle.rst

You're welcome to contribute directly to that branch. Or if it is easier, please respond to this thread in the Arches forum: https://community.archesproject.org/t/documentation-request-good-data-lifecycle-practices-with-arches/2387

I'd also be happy to arrange Zoom meetings to spare you from writing if that helps!

[zoometh]

@ekansa this is a brief description of how our 2 projects (with @ads04r, and only 1 database) address your questions on data lifecycle

EAMENA-MaREA

https://eamena.org/home https://marea.soton.ac.uk/

Maintaining data integrity

• data types: free text (string) fields have been reduce to the minimum, and most of the EAMENA fields are concepts (i.e. controlled vocabularies)

datatypes of the fields of the Heritage Places resource model

• keys: we are making the correspondances between the UUIDs and their semantic content publicly available and interoperable:
  • browse
  • describe here and there
• storage: an AWS snapshot is created daily (cloud), and a Postgres backup is created daily (local server),
• auditing: the package eamenaR and Python functions help to audit the database

Good data practices relating to upgrading Arches (especially major version upgrades)

• see: https://github.com/eamena-project/eamena-arches-dev/tree/main/dbs/database.eamena/install

Security, managing sensitive information

• SSH and Postgres access use AWS firewalls
• User access read vs edit are managed through Django Admin
• 'managing sensitive information' is THE real issue we are facing (see here), and we are struggling limit the access to some information (ex: exact coordinates of the Heritage Places, collaborator names) in a region (Middle East and North Africa) often politically shattered

Database backup strategies and approaches

Described previously (see storage)

Database archiving (with outside digital repositories)

We have a workflow (and an Arches plugin) to deposit EAMENA datasets in Zenodo (and have DOIs, a OAI-PMH compliant deposit, etc.)

Useful command line utilities

Documented here and there, and at many places in our GitHub

PostgreSQL utilities

Same as before, here and there.

Use of cloud computing database services (Amazon RDS, etc.)

None