Investigate using pushmi_pullyu with OLRC

[pgwillia]

Investigate using pushmi_pullyu with OLRC

I think we will likely need to re-configure PMPY to point to OLRC this fall. With that we might take several approaches - simple configuration of existing PMPY against a new swift container using the current API approach, or a more complex project of integrating Duracloud into our PMPY pipeline. - Kenton

First steps

1. Meet with Kenton for context and make this a ticket.
2. Ask for credentials that we'll need and the container we'll write to for this test.

Maybe the first step is seeing if our development environment will write to OLRC.

[jefferya]

Here are the major differences I've experienced when converting the CWRC perseveration tooling to use OLRC Swift as a drop-in replacement for UAL Swift.

The main difference, OLRC requires an additional property, a domain as part of the authentication. As of 2023-08-15, I'm testing to see if further changes are required: https://github.com/ualbertalib/cwrc_preservation/tree/jefferya/olrc_compatibility

• OLRC OpenStack RC file env var OS_USER_DOMAIN_NAME to OpenStack Gem connection property user_domain
• OLRC OpenStack RC file env var OS_PROJECT_DOMAIN_ID to OpenStack Gem connection property project_domain_id

The ruby-openstack connection is defined here: https://github.com/ruby-openstack/ruby-openstack/blob/master/lib/openstack/connection.rb

• tenant is not required with OLRC
• project_domain_name: I used the project_domain_id instead as there was no project domain name in the OLRC OpenStack RC file

The aforementioned OpenStack RC file can be downloaded from the Horizon UI via the upper right-hand corner user account menu.

If you want to test the Swift authentication, one can use the OpenStack Swift command-line tool. The following is a command-line OpenStack Swift example that can be used to test (replace values with items from the OpenStack RC file) the individual parameters.

swift --os-auth-url 'a' \
  --os-username 'b' \
  --os-user-domain-name 'c' \
  --os-password  'd' \
  --os-project-name 'e' \
  --os-project-domain-id 'f' \
  --auth-version 3 \
  stat

Note: running source on the OpenStack RC file will populate environment variables so one doesn't need to populate the swift command-line arguments as on the command line (e.g., don't need to add a visible password on the command-line).

The install process for the Swift client is:

pip3 install python-swiftclient --user

One might also need sudo apt install python-openstackclient but I'm not certain.

[lagoan]

Thanks to @jefferya work the process of redirecting PMPY to OLRC has been straightforward.

The following shell command works for me to test authentication:

swift --os-auth-url 'AUTH_URL' \
      --os-username 'USER_NAME' \
      --os-user-domain-name 'USER_DOMAIN_NAME' \
      --os-password  'PASSWORD' \
      --os-project-name 'PROJECT_NAME' \
      --os-project-domain-name 'PROJECT_DOMAIN_NAME' \
      --auth-version 3 \
      stat

The changes in the codebase to redirect pushing the items to OLRC Swift have been mirroring the configuration changes used to authenticate.

I've successfully pushed synthetic data to a new container in OLRC.

My next steps include:

• [x] Testing pushing real data from a database dump in my development environment
• [x] Test with real data, including files, from our staging environment
• [x] Configuring the docker image of OpenStack Swift to use v3 for authentication
• [x] Once our OpenStack Swift instance is configured, I will be able to create the tests for development