aicoe-aiops / ocp-ci-analysis

Developing AI tools for developers by leveraging the data made openly available by OpenShift and Kubernetes CI platforms.
https://old.operate-first.cloud/data-science/ai4ci/
GNU General Public License v3.0
33 stars 72 forks source link

Use MOC OpenStack for long term data storage [EPIC] #442

Open MichaelClifford opened 2 years ago

MichaelClifford commented 2 years ago

We have been requesting a location to host public data for some time now (https://github.com/operate-first/support/issues/23) and the consensuses decision has been to use MOC Openstack accounts for storage rather than something on one of the opf clusters as they are more ephemeral (https://github.com/open-infrastructure-labs/ops-issues/issues/33).

To that end, I have created a public bucket specifically for AI4CI project here: https://kzn-swift.massopen.cloud/swift/v1/ai4ci/
This bucket should allow for users to read data without restriction, but only users with credentials can write/delete data.

Here are the docs for interacting with the bucket: https://docs.massopen.cloud/en/latest/openstack/Object-Storage.html

Please reach out to me if you are working on this and I can provide credentials or add you to the openstack project managing the bucket.

We should migrate the existing ai4ci work such that all data is stored here, rather than on the opf-datacatalog bucket on smaug.

Acceptance Criteria:

antter commented 2 years ago

Working on this first point:

https://github.com/aicoe-aiops/ocp-ci-analysis/issues/443