search

usegalaxy-au / infrastructure

Galaxy Australia's Ansible scripts
MIT License
7 stars 18 forks source link

aws bucket and nectar in object store #2220

Closed cat-bro closed 1 month ago

cat-bro commented 1 month ago

Add test backends to galaxy object store conf: "nectar_test" and "aws_bucket_test". These are configured more or less exactly like "minio_test".

The cache directories are all on volA which is where Galaxy's output data is going at the moment. This is going to change to volD very soon, so the cache directories should probably all go on volD and this should not be merged until that change has been made.

cat-bro commented 1 month ago

The sample object store recommends boto3 for everything (well sort of, it also says aws_s3 and generic_s3 are more battle-tested). It uses boto3 and generic_s3 for its swift examples and there is no example of type: swift in the sample document at all. The only reason I know type: swift is possible is that Galaxy Main uses it. It looks similar to generic_s3 in its object store configuration and we could try this instead - do you know why people choose one over the other?

nuwang commented 1 month ago

The s3 compatibility layer in swift is sometimes not available, in which case direct swift becomes the only option. Agree that we are not buying much by testing direct swift, because we have the more battle tested boto3 layer available.