kysrpex
commented
1 year ago The issue is directly related to this change committed yesterday.
Closed kysrpex closed 1 year ago
kysrpex
commented
1 year ago The issue is directly related to this change committed yesterday.
hexylena
commented
1 year ago Interestingly running the git-gat playbook from roughly the same point you are, i seem to have a working celery service. I'm using https://github.com/galaxyproject/training-material/tree/63f37692cd17c3420315cbd41e3468b39b0ff3fc (I see errors for other reasons like permissions on /libraries, from the data libraries tutorial.)
kysrpex
commented
1 year ago After having a closer look at everything it seems that systemd is being used (obvious from the log I pasted :sweat_smile:) rather than supervisor, but nevertheless I believe at least an instance of the class SupervisorProcessManager is still being instantiated and that creates the directory with the wrong ownership. Issue #103 in gravity repository should not be related.
kysrpex
commented
1 year ago Interestingly running the
git-gatplaybook from roughly the same point you are, i seem to have a working celery service. I'm using https://github.com/galaxyproject/training-material/tree/63f37692cd17c3420315cbd41e3468b39b0ff3fc (I see errors for other reasons like permissions on /libraries, from the data libraries tutorial.)
You can find extra details about my playbook below.
@natefoo Do you think it makes sense to include a task that creates galaxy_gravity_state_dir and sets correct ownership beforehand given that the directory is going to be shared among processes run as root and processes run as {{ galaxy_user_name }}? Or should gravity itself be aware of the shared use? Given that gravity is choosing the path for the file causing the issue, and aware of the Galaxy configuration, personally I'd go for the second solution.
ansible.cfg
[defaults]
interpreter_python = /usr/bin/python3
inventory = hosts
stdout_callback = yaml
bin_ansible_callbacks = True
vault_password_file = .vault-password.txt
[ssh_connection]
pipelining = true
[diff]
always = Truegalaxy.yml
---
- hosts: dbservers
become: true
become_user: root
pre_tasks:
- name: Install dependencies
package:
name: 'acl'
roles:
- galaxyproject.postgresql
- role: galaxyproject.postgresql_objects
become: true
become_user: postgres
- hosts: galaxyservers
become: true
become_user: root
vars_files:
- group_vars/secret.yml
pre_tasks:
- name: Install dependencies
package:
name:
- acl
- bzip2
- git
- make
- tar
- python3-venv
- python3-setuptools
roles:
- galaxyproject.galaxy
- role: galaxyproject.miniconda
become: true
become_user: "{{ galaxy_user_name }}"hosts
[galaxyservers]
gat-2.eu.training.galaxyproject.eu ansible_connection=local ansible_user=ubuntu
[dbservers:children]
galaxyserversrequirements.yml
- src: galaxyproject.galaxy
version: 0.10.11
- src: galaxyproject.nginx
version: 0.7.1
- src: galaxyproject.postgresql
version: 1.1.2
- src: galaxyproject.postgresql_objects
version: 1.2.0
- src: galaxyproject.miniconda
version: 0.3.1
- src: usegalaxy_eu.certbot
version: 0.1.7group_vars/all.yml
---
# Python 3 support
pip_virtualenv_command: /usr/bin/python3 -m venv # usegalaxy_eu.certbot, usegalaxy_eu.tiaas2, galaxyproject.galaxy
certbot_virtualenv_package_name: python3-venv # usegalaxy_eu.certbot
# Common variables needed by all hosts
galaxy_user_name: galaxy
galaxy_db_name: galaxygroup_vars/dbservers.yml
---
# PostgreSQL
postgresql_objects_users:
- name: "{{ galaxy_user_name }}"
postgresql_objects_databases:
- name: "{{ galaxy_db_name }}"
owner: "{{ galaxy_user_name }}"group_vars/galaxyservers.yml
---
galaxy_create_user: true
galaxy_separate_privileges: true
galaxy_manage_paths: true
galaxy_layout: root-dir
galaxy_root: /srv/galaxy
galaxy_user:
name: "galaxy"
shell: /bin/bash
galaxy_commit_id: release_23.0
galaxy_force_checkout: true
miniconda_prefix: "{{ galaxy_tool_dependency_dir }}/_conda"
miniconda_version: 4.12.0
miniconda_channels:
- conda-forge
- defaults
galaxy_config:
galaxy:
admin_users:
- "dominguj@informatik.uni-freiburg.de"
brand: "Cool Galaxy 🌌"
database_connection: postgresql:///galaxy?host=/var/run/postgresql
tool_data_path: "{{ galaxy_mutable_data_dir }}/tool-data"
object_store_store_by: uuid
id_secret: "{{ vault_id_secret }}"
gravity:
process_manager: systemd
galaxy_root: "{{ galaxy_root }}/server"
galaxy_user: "{{ galaxy_user_name }}"
virtualenv: "{{ galaxy_venv_dir }}"
gunicorn:
bind: "unix:{{ galaxy_mutable_config_dir }}/gunicorn.sock"
workers: 2
extra_args: '--forwarded-allow-ips="*"'
preload: true
celery:
concurrency: 2
loglevel: DEBUG
handlers:
handler:
processes: 2
pools:
- job-handlers
- workflow-schedulers
natefoo
commented
1 year ago This is a Gravity bug, I will see if I can fix it quickly to prevent this going forward. The SupervisorProcessManager is instantiated because all ProcessManagers have to be whenever update is run, in case you changed process_manager from supervisor to systemd or vice-versa, so that the config files of the PM's you are switching from can be removed.
natefoo
commented
1 year ago This should be fixed by https://github.com/galaxyproject/gravity/pull/105
kysrpex
commented
1 year ago Interestingly running the
git-gatplaybook from roughly the same point you are, i seem to have a working celery service. I'm using https://github.com/galaxyproject/training-material/tree/63f37692cd17c3420315cbd41e3468b39b0ff3fc (I see errors for other reasons like permissions on /libraries, from the data libraries tutorial.)
I just checked out this commit from git-gat (that according to the README corresponds to 63f37692cd on training-material) on a fresh VM (gat-5.eu.training.galaxyproject.eu), ran the playbook, and could not reproduce the success.
As you say, the service seems to be working
ubuntu@gat-5:~/git-gat$ sudo galaxyctl status
UNIT LOAD ACTIVE SUB DESCRIPTION
galaxy-celery-beat.service loaded active running Galaxy celery-beat
galaxy-celery.service loaded active running Galaxy celery
galaxy-gunicorn.service loaded active running Galaxy gunicorn
galaxy-handler@0.service loaded active running Galaxy handler (process 0)
galaxy-handler@1.service loaded active running Galaxy handler (process 1)
galaxy.target loaded inactive dead Galaxy
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
6 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.but the galaxy target is not reached and the logs show a restart loop for galaxy-celery-beat.service.
kysrpex
commented
1 year ago This should be fixed by galaxyproject/gravity#105
Yes, that PR fixes it, thanks!
root@gat-5:~# su galaxy
galaxy@gat-5:/root$ source /srv/galaxy/venv/bin/activate
(venv) galaxy@gat-5:/root$ pip install git+https://github.com/galaxyproject/gravity.git@22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b
Collecting git+https://github.com/galaxyproject/gravity.git@22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b
Cloning https://github.com/galaxyproject/gravity.git (to revision 22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b) to /tmp/pip-req-build-adwr2sko
Running command git clone --filter=blob:none --quiet https://github.com/galaxyproject/gravity.git /tmp/pip-req-build-adwr2sko
Running command git rev-parse -q --verify 'sha^22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b'
Running command git fetch -q https://github.com/galaxyproject/gravity.git 22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b
Resolved https://github.com/galaxyproject/gravity.git to commit 22477189c4b2b9d8fba2b3a02a89a00a5f5dfa3b
Installing build dependencies ... done
Getting requirements to build wheel ... done
Preparing metadata (pyproject.toml) ... done
Requirement already satisfied: requests-unixsocket in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (0.3.0)
Requirement already satisfied: pyyaml in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (6.0)
Requirement already satisfied: requests in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (2.28.2)
Requirement already satisfied: pydantic in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (1.10.4)
Requirement already satisfied: jsonref in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (1.1.0)
Requirement already satisfied: supervisor in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (4.2.5)
Requirement already satisfied: Click in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (8.1.3)
Requirement already satisfied: packaging in /srv/galaxy/venv/lib/python3.10/site-packages (from gravity==1.0.2) (21.3)
Requirement already satisfied: pyparsing!=3.0.5,>=2.0.2 in /srv/galaxy/venv/lib/python3.10/site-packages (from packaging->gravity==1.0.2) (3.0.9)
Requirement already satisfied: typing-extensions>=4.2.0 in /srv/galaxy/venv/lib/python3.10/site-packages (from pydantic->gravity==1.0.2) (4.4.0)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /srv/galaxy/venv/lib/python3.10/site-packages (from requests->gravity==1.0.2) (1.26.14)
Requirement already satisfied: charset-normalizer<4,>=2 in /srv/galaxy/venv/lib/python3.10/site-packages (from requests->gravity==1.0.2) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /srv/galaxy/venv/lib/python3.10/site-packages (from requests->gravity==1.0.2) (3.4)
Requirement already satisfied: certifi>=2017.4.17 in /srv/galaxy/venv/lib/python3.10/site-packages (from requests->gravity==1.0.2) (2022.12.7)
Requirement already satisfied: setuptools in /srv/galaxy/venv/lib/python3.10/site-packages (from supervisor->gravity==1.0.2) (66.1.1)
(venv) galaxy@gat-5:/root$
exit
root@gat-5:~# ls -lah /srv/galaxy/var/gravity
ls: cannot access '/srv/galaxy/var/gravity': No such file or directory
root@gat-5:~# galaxyctl start
UNIT LOAD ACTIVE SUB DESCRIPTION
galaxy-celery-beat.service loaded active running Galaxy celery-beat
galaxy-celery.service loaded active running Galaxy celery
galaxy-gunicorn.service loaded active running Galaxy gunicorn
galaxy-handler@0.service loaded active running Galaxy handler (process 0)
galaxy-handler@1.service loaded active running Galaxy handler (process 1)
galaxy.target loaded active active Galaxy
LOAD = Reflects whether the unit definition was properly loaded.
ACTIVE = The high-level unit activation state, i.e. generalization of SUB.
SUB = The low-level unit activation state, values depend on unit type.
6 loaded units listed.
To show all installed unit files use 'systemctl list-unit-files'.
root@gat-5:~# ls -lah /srv/galaxy/var/gravity
total 12K
drwxr-xr-x 3 root root 4.0K Apr 13 08:21 .
drwxr-xr-x 14 galaxy galaxy 4.0K Apr 13 08:21 ..
drwxr-xr-x 3 root root 4.0K Apr 13 08:21 supervisor
Short version
Issue #103 in gravity repository prevents
galaxy-celery-beat.servicefrom starting because it causes the directory{{ galaxy_gravity_state_dir }}(by default/srv/galaxy/var/gravity/) to be created with the wrong ownership.Ultimately this is not a problem with the Galaxy role but Gravity, nevertheless I think it makes sense to have it reported here for anybody experiencing the same issue (even if the issue is closed inmediately, which is probably the right way to proceed, it can be read later).
Long version
I am following the tutorial "Galaxy Installation with Ansible" from the Galaxy
training-materialrepository (commit 37599dd37674e4d2679c4a5d0f2c9bb818e0006b).When I reached the part "Hands-on: Status Check", I noticed that galaxy.target is not reached.
Invoking
galaxyctl followyields the following output, that repeats in a loop.Everything seems to boil down to not having permission to write on
/srv/galaxy/var/gravity/celery-beat-schedule. The directory/srv/galaxy/var/gravityseems in fact to be owned by root instead of by the galaxy user.For the default configuration of the role,
/srv/galaxy/var/gravitycorresponds togalaxy_gravity_state_dir.I am invoking the role with the group variable
galaxy_commit_idset torelease_23.0. The checked out commit in/srv/galaxy/serveris a1bdffa43f5f23769dc32d33909cc4631aa01ca5.Based on this, I believe the reason for
/srv/galaxy/var/gravity/being owned byrootinstead ofgalaxyare the Gravity handlers for Galaxy 23.0 (see main.yml as well).In fact, the handlers call
galaxyctlwith the argumentsupdateandgraceful. Both involve calling the process manager. I assume the actual process manager being spawned isSupervisorProcessManager, because of the folder structureand the paths defined in the
__init__method.This line must be creating not only
/srv/galaxy/var/gravity/supervisor/supervisord.conf.d, but also the top-level directory/srv/galaxy/var/gravity. I assumegalaxyctlis being run then as root, which is what is expected to happen because I also setgravity/process_managertosystemdingroup_vars/galaxyservers.yml(as instructed in the tutorial) and by defaultgalaxy_systemd_rootis set totrue.This does not make much sense now, because systemd should be thus being used instead of supervisor. This may be a bug, and there is already an issue in the gravity repository.
Therefore, ultimately this is not a problem with the Galaxy role but Gravity, nevertheless I think it makes sense to have it reported here for anybody experiencing the same issue (even if the issue is closed inmediately, which is probably the right way to proceed, it can be read later).