search

usegalaxy-eu / infrastructure-playbook

Ansible playbook for managing UseGalaxy.eu infrastructure.
MIT License
16 stars 91 forks source link

Enable and add user file sources and object store features to EU #1260

Closed sanjaysrikakulam closed 1 month ago

sanjaysrikakulam commented 1 month ago

This PR:

  1. Enables the user file source and object store features in usegalaxy.eu
  2. Add the respective template catalogs
  3. Configures the object store cache path and the size. The cache path is set to /data/jwd02f/s3_object_store_cache, and the size (in GiB) is set to 10000. Both values are based on the object store conf we have here.
  4. sn06 group_vars is updated to copy the catalog files to the galaxy config directory

Important

  1. We use the templates that are defined as "production" in the Galaxy codebase
  2. Only non-legacy (boto3) production templates are included in the catalog files (for an available list of templates, refer: file source templates, and object store templates)
  3. We do not copy or maintain a copy of these templates in our Galaxy conf; rather, in the catalog, we refer to/include the ones in the Galaxy code base (in EU /opt/galaxy/server/).

To do on the venv: I did the following on the ESG instance to get the non-legacy templates to work. The non-legacy templates depend on the boto3 package, whereas the legacy templates depend on boto version 2.

  1. To get the non-legacy templates to work, we must modify Galaxy's venv (ping @bgruening). Non-legacy templates need boto3 to work, and boto3 is installed as a conditional dependency and gets installed only when a boto3 type object store is defined in the object_store_conf or when AWSBatchJobRunner is used in the job_conf. We need to modify the dependencies in our fork. The below is just a couple of manual things I did for testing.
# Activate the venv
. /opt/galaxy/venv/bin/activate

# Uninstall boto related packages
pip uninstall boto botocore aiobotocore

# Install boto3
pip install boto3 aiobotocore

# Restart handlers and gunicorns

Once this is done, we should be able to see two new features (like in the image below) in the User -> Preferences

image

and for the user object store, the non-legacy templates (ignore the legacy templates in the image below; this is from the ESG instance)

image

and for the user file sources,

image