Backup of data on kg-hub s3 bucket

Knowledge-Graph-Hub / knowledge-graph-hub-support

Issues, support, and discussion for KG-Hub. Covers tools, infrastructure, and graph projects.

BSD 3-Clause "New" or "Revised" License

5 stars 2 forks source link

Backup of data on kg-hub s3 bucket #3

Open justaddcoffee opened 2 years ago

justaddcoffee commented 2 years ago

We have enough data and projects to where we should strongly consider a backup solution.

Describe the desired behavior

Backup existing data on s3://kg-hub-public-data/ to a non-AWS location on an ongoing basis - monthly?

Additional context

Google Storage might be a good solution: https://cloud.google.com/storage/docs/storage-classes

justaddcoffee commented 2 years ago

@caufieldjh @kltm

caufieldjh commented 2 years ago

One strategy for GCP: https://cloud.google.com/architecture/transferring-data-from-amazon-s3-to-cloud-storage-using-vpc-service-controls-and-storage-transfer-service

caufieldjh commented 11 months ago

Discussion in BBOP meeting today (Nov 11 2023):

Set up KG-Hub community on Zenodo
Automate upload to Zenodo (try sandbox.zenodo.org) - see also https://github.com/cthoyt/zenodo-client and the GO's client
Check with @kltm for any assistance

Note that we're trying to ensure both data security/integrity (i.e., backups) and accessibility/findability (i.e., can users download graph releases and have specific DOIs)

justaddcoffee commented 11 months ago

Sorry to have missed the discussion at group meeting (I was traveling)

@caufieldjh should we hack on this on our KG construction calls, and/or our M/W/F hackathons?

caufieldjh commented 11 months ago

@caufieldjh should we hack on this on our KG construction calls, and/or our M/W/F hackathons?

Probably a good subject for the KG construction calls, especially if we can record the details in a best practices doc

justaddcoffee commented 11 months ago

+1 @caufieldjh sounds good!