GAT-next Improvements

[hexylena]

• [ ] ~Fully automated videos~ Not yet :( 2023.
  • [x] @mvdbeek suggests that the job_conf.xml file can be a job_conf.yaml, and the dependency resolver related stuff can be YAML, even within galaxy.yml, doesn't have to be a separate file. This would trigger re-recording of like 10 videos, we'll wait until helena's automated videos work.
• [x] add cleanup cron job role
• [ ] move dependency resolvers and container resolvers to embedded
• [ ] move pulsar config to support the same.
• [x] Create a dbservers group and putting postgres stuff in group_vars/dbservers.yml and connect over TCP, show modification of pg_hba.conf to support.
• [x] Maybe move job_conf.xml and job_metrics_conf.xml to their own "adding additional configs" tutorial that gets run right after ansible-galaxy?
• [x] Use "local" tools: Update job conf/pulsar tutorial for this amazing PR from @jmchilton that will save so many admins so much random pain. https://github.com/galaxyproject/galaxy/pull/12258 looks like we just need <tool class="local" destination="local" />

These require nginx package changes:

• [x] add mod_zip, thanks @mvdbeek @lldelisle https://docs.galaxyproject.org/en/latest/admin/nginx.html#creating-archives-with-mod-zip ~- [ ] maybe add https://docs.galaxyproject.org/en/latest/admin/nginx.html#receiving-files-with-nginx too? it's better for performance~ TUS!

Re-record next time

everything, tbh.

[natefoo]

Regarding the job conf, yes, we have converted it to a (standalone) YAML file on .org, this works now. For the admin training, putting it directly in galaxy.yml may be preferable as it's fewer steps.

I tested dependency resolvers and container resolvers as standalone YAMLs and added support for those in the galaxy role.

Dependencies are also supported under dependency_resolvers or dependency_resolution.resolvers in galaxy.yml. Pulsar supports dependency_resolvers.resolvers as well (I dropped support for the standalone file in the pulsar role since you can just specify it directly in the app YAML), although we don't use it in the training.

[hexylena]

awesome!!

[natefoo]

Couple more things:

• Maybe should consider creating a dbservers group and putting postgres stuff in group_vars/dbservers.yml. It might be confusing up front but it will help people who later deploy on their own infra and have a separate DB server.
• Should we create galaxy_user_name and galaxy_db_name vars to avoid duplication in postrgresql_objects_* and database_connection?

[natefoo]

Maybe move job_conf.xml and job_metrics_conf.xml to their own "adding additional configs" tutorial that gets run right after ansible-galaxy?

[hexylena]

Should we create galaxy_user_name and galaxy_db_name vars to avoid duplication in postrgresqlobjects* and database_connection?

it's so early in the training, later it makes sense, but I worry about defining a variable in step like 1, and students thinking they have to define variables for everything (even if that's often best practice.)

[hexylena]

Maybe move job_conf.xml and job_metrics_conf.xml to their own "adding additional configs" tutorial that gets run right after ansible-galaxy?

I especially support that if we also mention tool data tables since I forgot that and needed to load a private genome and it took a while to re-figure out.

Maybe user_prefs too, that's pretty key these days.

[natefoo]

I worry about defining a variable in step like 1, and students thinking they have to define variables for everything (even if that's often best practice.)

There's definitely a balance. This mostly came up because I was recording the new video and realized how many times I pointed out having one canonical source of truth for everything, and then we don't do it in that case, heh.

[hexylena]

@natefoo and I discussed, we should aggressively cut down on the slide decks, at least on day 1. Some of them are just... not necessary, all the important/good information is in the tutorials after all.

slides	action	comments
DB	cut	we don't need to discuss sqlite or connection string formats, we have a table for that in the tutorial (or if we don't, we can), same with tuning parameters, this is documentation which isn't necessary for understanding "galaxy connects to a DB." and since we started including the good defaults, it's even less relevant. Install DB we mention and never touch again, not great.
systemd	merge	we should merge this into a tip box.
uwsgi	cut	:(
ephemeris	keep	they're good
cvmfs	keep	same
users groups quotes	keep	idk, it's on thin ice.
cluster computing	merge
DB querying	meh	these are now basically gxadmin advertising. I don't find them very useful, and it could be covered in a tuto. That said they're already an auto-video, so, meh, keep?
monitoring	idk	there is so little information there, it's basically a showcase. Is that useful? Is that the best format to talk about monitoring?

[natefoo]

Because the + and - prefixes are hidden in diffs, they might be impossible to read for people with red/green colorblindness. In addition, at least one non-r/g-colorblind student mentioned difficulty seeing the color difference on their display. Can we add a switchable style for higher contrast and forchanging the diff colors entirely?

[hexylena]

Update job conf/pulsar tutorial for this amazing PR from @jmchilton that will save so many admins so much random pain. https://github.com/galaxyproject/galaxy/pull/12258

looks like we just need <tool class="local" destination="local" /> and then figure out what's needed for requires_galaxy

[hexylena]

We should add tusd, that's def going to be an improvement

• https://github.com/galaxyproject/galaxy-helm/pull/341
• https://github.com/galaxyproject/galaxy/pull/12656/files#diff-27f31d370acb827ff20e93b891a4fceedee04b8aad03f0e36461489e8ce16956

[hexylena]

Do we want to teach admins to use git this year @natefoo @Slugger70 @martenson? \

• [ ] if we do, teach them gitattributes for automatically decrypting vaults in git log -p output

  $ cat .gitattributes
  secret_group_vars/*.yml diff=ansible-vault merge=binary
  secret_host_vars/*.yml diff=ansible-vault merge=binary

[natefoo]

:+1: to git, also wow that git log -p magic, I was not even aware of. Seems worth setting up?

[Slugger70]

I think that's a great idea. Especially the vault merging stuff which is always confusing..😁

On Tue, 30 Nov 2021, 04:43 Nate Coraor, @.***> wrote:

👍 to git, also wow that git log -p magic, I was not even aware of. Seems worth setting up?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/galaxyproject/training-material/issues/2583#issuecomment-981865199, or unsubscribe https://github.com/notifications/unsubscribe-auth/AANI3XUC3I5WHIMZBK4SSLDUOO3UBANCNFSM457AQARA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[hexylena]

ok, time for a super quick git lesson then, that's a good improvement/addition for sure.

[hexylena]

More TUS: https://github.com/galaxyproject/galaxy/pull/13313

[mvdbeek]

Is it too late to switch to Gunicorn for web serving ? The mile-high view for the changes is in https://docs.galaxyproject.org/en/latest/admin/migrating_to_gunicorn.html#how-do-i-upgrade-my-instance-to-use-gunicorn. The individual documentation PRs are collected in https://github.com/galaxyproject/galaxy/issues/12257.

It shouldn't be a large change for production setups, you'd likely only need to change the actual python entry point for the web process and replace uwsgi_pass with proxy_pass.

[hexylena]

I'll start by saying I would love that! It would be really fantastic.

However. Given that:

• the two admins I usually rely on to help me do the updates have zero bandwidth and are pretty burnt out currently (like me) leaving any updates I had the energy to push through to me (which was essentially none, and TUS only happened with your help / copy pasting another tutorial.)
• The incredible person who did all of the project management left, and leadership hasn't made huge strides to replace them, leaving that all the project management to me

I'm going to say it's too late :disappointed:

However: If you are willing to make a PR for it? I will absolutely consider re-recording that video. That's a lower bandwidth activity for me that might be possible.

It's great that the updates are easy from the Galaxy side, but it looks like ansible-galaxy is not updated at all, I still see references to mules and uwsgi which needs to be fixed first I guess. (here for example and in the tasks)

[mvdbeek]

That's fair, and I totally understand that you're strapped for time and energy. I'll think about it and check with @natefoo if it's realistic to get this done in the short term.

[hexylena]

Update miniconda role with galaxy one

[hexylena]

To summarise for others passing through: testing.xml, my_rules.py and map_resources.py all have to go under files/galaxy/... rather than templates/galaxy/...

[hexylena]

Track minimum ansible version, note it in metadata of tutorials. Currently ansible-galaxy says >=2.7 which is hilariously outdated.

[natefoo]

I wish the linter could determine the minimum ansible version, because I always think of it as I'm updating the role and then forget by the time I go to update the meta.

[hexylena]

from @fubar2

With cvmfs reference data, doesn't singularity need /cvmfs added as a path in job_conf.xml.j2 ? :

<param id="singularity_volumes">$defaults,/cvmfs:ro</param>

It works then, but before I added that, mappers always failed - unable to access the /cvmfs reference data It may be that I've broken something in removing lots of extra stuff - or nobody got around to testing with cvmfs reference data?

A more generic issue about singularity and cvmfs is noted at https://cvmfs.readthedocs.io/en/stable/cpt-configure.html#remounting-and-namespaces-containers, suggesting that -p is recommended for singularity so I also added:

<param id="singularity_run_extra_arguments"> -p </param>

Dunno if it makes any difference but...doesn't seem to be harmful.

Currently this is a bit of a sequencing issue, we cover singularity, then later cvmfs. I think that means cvmfs should come before singularity? maybe?

[hexylena]

from gtn feedback from

it would be useful if the hands-on gxadmin commands were in a copyable code block like they are in other tutorials – it really helps to be able to copy them with the single click. Also, the 'admin favourites' section could include the arguments for the functions listed, e.g. , for a bit more clarity

and

There is an "error" in the example code for the Galaxy/FTP portion of galaxyservers.yml. if you intend to use separate upload directories (with email addresses as the folder name) for each user - as the tutorial indicates will happen - then the current code (below) will result in the uploads going into the parent directory "/uploads" instead of "uploads/user_email" . # FTP ftp_upload_dir: /data/uploads ftp_upload_site: "{{ inventory_hostname }}" Instead the code should be (add a "/"): # FTP ftp_upload_dir: /data/uploads/ ftp_upload_site: "{{ inventory_hostname }}" Took me a while to work out why it was going into the wrong directory. Also - it should be stated that there may be a newer version of the playbook that could solve errors/conflicts with newer operating system versions than those used in the tutorial. Also suggest highlighting parts in the code examples which should be customized to your server environment if you are using this as a guide to setup your own installation (primarily directories where stuff is located).

[lldelisle]

I don't understand this ftp error. On my instance, I set it up without / (https://github.com/lldelisle/galaxyduboule-infrastructure/blob/17378097331db5cb6275ac906b47ac10c28bcccd/group_vars/galaxyservers.yml#L97-L98) (I am using release_21.01) and

$ ls -d /data/galaxy/uploads*/
/data/galaxy/uploads/
$ lsb_release -d
Description:    Ubuntu 20.04.3 LTS

Do you still have access to VMs from GAT so I can check how it is on one of them? However, if I remember corrrectly we already had an issue on this no? So maybe it is my instance which is weird.

[hexylena]

I haven't checked it in detail, just copied from the feedback. I vaguely remember a similar ftp complaint that we determined was a non-issue, but I didn't have time to dig out that issue. I'll look into it eventually :)

[hexylena]

https://github.com/galaxyproject/galaxy/issues/13687#issuecomment-1093020466 cover changing object store by user/group info in the mapping jobs to destinations (doesn't have to be a worked example, a tip box would be sufficient.)

[hexylena]

@Slugger70 @natefoo and I had a short discussion ahead of GCC planning

• we'll move cvmfs ahead of singularity
• and enable singularity to use cvmfs containers?
  • No, we still can't do fall through dependency resolution :sob:
• DBA ends up being gxadmin, I've got a collection of links but not sure how to turn that into a good tutorial

[mvdbeek]

• No, we still can't do fall through dependency resolution 😭

What's that about ? Do we have an issue for that ?

[hexylena]

@natefoo can provide some more details there but in general it seems everyone using singularity is manually choosing which tools/versions are going to a singularity destination

And for most smaller galaxies we need something more like "run in the right container, if available, and if not run in the default container and pull in the conda dependencies"

[mvdbeek]

And that's not how it works ? I had it configured that way back when I was admin for a small instance and I switched to singularity (lot's of resolver related things changed since then, so it may not work anymore ...).

The order should be

• resolve conda
• resolve container

If a container is resolved it'll take over, else conda is getting used.

[mvdbeek]

Oh, default container + conda dependencies, not just conda. I wasn't aware that's a thing. Is that going to build an image every time you run a tool, or is that mounting an existing conda env on disk and resolving dependencies in the image ?

[hexylena]

Sorry @mvdbeek I feel like we have this conversation once a year and every year I'm confused.

That sounds great if it's something we can use, that's going to work for local admins with their custom tools that may not have pre-mulled dependencies.

Or if it falls back to conda properly (even without the container) that's probably fine too. but I think in @natefoo and my discussion mounting was mentioned.

[natefoo]

It wouldn't build every time. I have an image I use for this here: https://github.com/natefoo/usegalaxy.org-legacy-environment

I believe what would be ideal is something like:

1. Resolve a matching container and run it in that if available
2. Resolve it in conda and run in a default container with conda mounted in if the deps are already installed in conda
3. Build/mull an image if the deps are available in conda but not already installed

[hexylena]

I am sufficiently emotionally done with this issue so, we're closing it. If we really want any of the open items in the OP, we can reopen.