search

bihealth / sodar-docker-compose

Setup SODAR using Docker Compose
MIT License
2 stars 3 forks source link

irods container not starting #48

Closed alabarga closed 3 months ago

alabarga commented 11 months ago

After following the setup instructions and running ./run.sh sodar-docker-compose-irods-1 is not started so celery fails

sodar-docker-compose-sodar-celeryd-default-1  | [2023-11-14 13:49:40,009: INFO/MainProcess] Task samplesheets.tasks_celery.sheet_sync_task[b6037b80-c57a-4b91-88b7-f7b29c035e11] received
sodar-docker-compose-sodar-celeryd-default-1  | [2023-11-14 13:49:40,023: INFO/ForkPoolWorker-15] Task samplesheets.tasks_celery.sheet_sync_task[b6037b80-c57a-4b91-88b7-f7b29c035e11] succeeded in 0.013157504001355846s: None
sodar-docker-compose-sodar-celerybeat-1       | [2023-11-14 13:49:46,037: INFO/MainProcess] Scheduler: Sending due task landingzones.tasks_celery.trigger_zone_move() (landingzones.tasks_celery.trigger_zone_move)
sodar-docker-compose-sodar-celeryd-default-1  | [2023-11-14 13:49:46,038: INFO/MainProcess] Task landingzones.tasks_celery.trigger_zone_move[c7972e1f-81ee-44e2-9091-536d90797761] received
sodar-docker-compose-sodar-celeryd-default-1  | 2023-11-14 13:49:46,065 [ERROR] irodsbackend.api: Unable to connect to iRODS (host=irods, port=1247): NetworkException (Could not connect to specified host and port: irods:1247)
sodar-docker-compose-sodar-celeryd-default-1  | [2023-11-14 13:49:46,065: ERROR/ForkPoolWorker-15] Unable to connect to iRODS (host=irods, port=1247): NetworkException (Could not connect to specified host and port: irods:1247)
sodar-docker-compose-sodar-celeryd-default-1  | [2023-11-14 13:49:46,067: ERROR/ForkPoolWorker-15] Task landingzones.tasks_celery.trigger_zone_move[c7972e1f-81ee-44e2-9091-536d90797761] raised unexpected: NetworkException('Could not connect to specified host and port: irods:1247')
sodar-docker-compose-sodar-celeryd-default-1  | Traceback (most recent call last):
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/session.py", line 264, in server_version
sodar-docker-compose-sodar-celeryd-default-1  |     return tuple(ast.literal_eval(reported_vsn))
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/ast.py", line 59, in literal_eval
sodar-docker-compose-sodar-celeryd-default-1  |     node_or_string = parse(node_or_string, mode='eval')
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/ast.py", line 47, in parse
sodar-docker-compose-sodar-celeryd-default-1  |     return compile(source, filename, mode, flags,
sodar-docker-compose-sodar-celeryd-default-1  |   File "<unknown>", line 0
sodar-docker-compose-sodar-celeryd-default-1  |     
sodar-docker-compose-sodar-celeryd-default-1  |     ^
sodar-docker-compose-sodar-celeryd-default-1  | SyntaxError: unexpected EOF while parsing
sodar-docker-compose-sodar-celeryd-default-1  | 
sodar-docker-compose-sodar-celeryd-default-1  | During handling of the above exception, another exception occurred:
sodar-docker-compose-sodar-celeryd-default-1  | 
sodar-docker-compose-sodar-celeryd-default-1  | Traceback (most recent call last):
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/session.py", line 270, in __server_version
sodar-docker-compose-sodar-celeryd-default-1  |     conn = next(iter(self.pool.active))
sodar-docker-compose-sodar-celeryd-default-1  | StopIteration
sodar-docker-compose-sodar-celeryd-default-1  | 
sodar-docker-compose-sodar-celeryd-default-1  | During handling of the above exception, another exception occurred:
sodar-docker-compose-sodar-celeryd-default-1  | 
sodar-docker-compose-sodar-celeryd-default-1  | Traceback (most recent call last):
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/pool.py", line 59, in get_connection
sodar-docker-compose-sodar-celeryd-default-1  |     conn = self.idle.pop()
sodar-docker-compose-sodar-celeryd-default-1  | KeyError: 'pop from an empty set'
sodar-docker-compose-sodar-celeryd-default-1  | 
sodar-docker-compose-sodar-celeryd-default-1  | During handling of the above exception, another exception occurred:
sodar-docker-compose-sodar-celeryd-default-1  | 
sodar-docker-compose-sodar-celeryd-default-1  | Traceback (most recent call last):
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/connection.py", line 224, in _connect
sodar-docker-compose-sodar-celeryd-default-1  |     s = socket.create_connection(address, timeout)
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/socket.py", line 787, in create_connection
sodar-docker-compose-sodar-celeryd-default-1  |     for res in getaddrinfo(host, port, 0, SOCK_STREAM):
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/socket.py", line 918, in getaddrinfo
sodar-docker-compose-sodar-celeryd-default-1  |     for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
sodar-docker-compose-sodar-celeryd-default-1  | socket.gaierror: [Errno -2] Name or service not known
sodar-docker-compose-sodar-celeryd-default-1  | 
sodar-docker-compose-sodar-celeryd-default-1  | During handling of the above exception, another exception occurred:
sodar-docker-compose-sodar-celeryd-default-1  | 
sodar-docker-compose-sodar-celeryd-default-1  | Traceback (most recent call last):
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 451, in trace_task
sodar-docker-compose-sodar-celeryd-default-1  |     R = retval = fun(*args, **kwargs)
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/celery/app/trace.py", line 734, in __protected_call__
sodar-docker-compose-sodar-celeryd-default-1  |     return self.run(*args, **kwargs)
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/src/app/landingzones/tasks_celery.py", line 37, in trigger_zone_move
sodar-docker-compose-sodar-celeryd-default-1  |     TriggerZoneMoveTask().run()
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/src/app/landingzones/tasks_celery.py", line 50, in run
sodar-docker-compose-sodar-celeryd-default-1  |     irods = irods_backend.get_session_obj()
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/src/app/irodsbackend/api.py", line 474, in get_session_obj
sodar-docker-compose-sodar-celeryd-default-1  |     return self._init_irods()
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/src/app/irodsbackend/api.py", line 115, in _init_irods
sodar-docker-compose-sodar-celeryd-default-1  |     raise ex
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/src/app/irodsbackend/api.py", line 102, in _init_irods
sodar-docker-compose-sodar-celeryd-default-1  |     irods.collections.exists(
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/manager/collection_manager.py", line 86, in exists
sodar-docker-compose-sodar-celeryd-default-1  |     self.get(path)
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/manager/collection_manager.py", line 21, in get
sodar-docker-compose-sodar-celeryd-default-1  |     query = self.sess.query(Collection).filter(*filters)
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/session.py", line 242, in query
sodar-docker-compose-sodar-celeryd-default-1  |     return Query(self, *args)
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/query.py", line 44, in __init__
sodar-docker-compose-sodar-celeryd-default-1  |     if self.sess.server_version >= col.min_version:
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/session.py", line 266, in server_version
sodar-docker-compose-sodar-celeryd-default-1  |     return self.__server_version()
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/session.py", line 273, in __server_version
sodar-docker-compose-sodar-celeryd-default-1  |     conn = self.pool.get_connection()
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/pool.py", line 15, in method_
sodar-docker-compose-sodar-celeryd-default-1  |     ret = method(self,*s,**kw)
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/pool.py", line 75, in get_connection
sodar-docker-compose-sodar-celeryd-default-1  |     conn = Connection(self, self.account)
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/connection.py", line 63, in __init__
sodar-docker-compose-sodar-celeryd-default-1  |     self._server_version = self._connect()
sodar-docker-compose-sodar-celeryd-default-1  |   File "/usr/local/lib/python3.8/site-packages/irods/connection.py", line 227, in _connect
sodar-docker-compose-sodar-celeryd-default-1  |     raise NetworkException(
sodar-docker-compose-sodar-celeryd-default-1  | irods.exception.NetworkException: Could not connect to specified host and port: irods:1247

Irods keeps restarting but find that ICAT database already exists

sodar-docker-compose-postgres-1               | 2023-11-14 12:52:34.622 UTC [118] LOG:  incomplete startup packet
sodar-docker-compose-irods-1                  | Password: 
sodar-docker-compose-irods-1                  | createdb: database creation failed: ERROR:  database "ICAT" already exists
sodar-docker-compose-postgres-1               | 2023-11-14 12:52:34.663 UTC [119] ERROR:  database "ICAT" already exists
sodar-docker-compose-postgres-1               | 2023-11-14 12:52:34.663 UTC [119] STATEMENT:  CREATE DATABASE "ICAT";
sodar-docker-compose-irods-1 exited with code 1

deleting the database manually will trigger a different error

sodar-docker-compose-irods-1                  | Host postgres:5432 not yet available...
sodar-docker-compose-irods-1                  | Host postgres:5432 not yet available...
sodar-docker-compose-irods-1                  | Host postgres:5432 not yet available...
sodar-docker-compose-irods-1                  | Host postgres:5432 is now available!
sodar-docker-compose-irods-1                  | --------------------------------------------------------
sodar-docker-compose-irods-1                  | docker-compose-wait - Everything's fine, the application can now start!
sodar-docker-compose-irods-1                  | --------------------------------------------------------
sodar-docker-compose-irods-1                  | Provisioning iRODS..
sodar-docker-compose-irods-1                  | Pre-create database if necessary
sodar-docker-compose-irods-1                  | Password: 
sodar-docker-compose-irods-1                  | Set up unattended configuration file
sodar-docker-compose-irods-1                  | Perform iRODS setup
sodar-docker-compose-irods-1                  | Warning: Hostname `irods` should be a fully qualified domain name.
sodar-docker-compose-irods-1                  | Error encountered running setup_irods:
sodar-docker-compose-irods-1                  | Traceback (most recent call last):
sodar-docker-compose-irods-1                  |   File "/var/lib/irods/scripts/setup_irods.py", line 445, in main
sodar-docker-compose-irods-1                  |     setup_server(irods_config, json_configuration_file=options.json_configuration_file)
sodar-docker-compose-irods-1                  |   File "/var/lib/irods/scripts/setup_irods.py", line 132, in setup_server
sodar-docker-compose-irods-1                  |     test_put(irods_config)
sodar-docker-compose-irods-1                  |   File "/var/lib/irods/scripts/setup_irods.py", line 149, in test_put
sodar-docker-compose-irods-1                  |     irods.lib.execute_command(['iput', f.name])
sodar-docker-compose-irods-1                  |   File "/var/lib/irods/scripts/irods/lib.py", line 134, in execute_command
sodar-docker-compose-irods-1                  |     check_command_return(args, out, err, returncode, **kwargs)
sodar-docker-compose-irods-1                  |   File "/var/lib/irods/scripts/irods/lib.py", line 117, in check_command_return
sodar-docker-compose-irods-1                  |     indent(err))]))
sodar-docker-compose-irods-1                  | IrodsError: Call to open process with ['iput', '/tmp/tmpyMSD6HdemoResc'] returned an error:
sodar-docker-compose-irods-1                  |   Options passed to Popen:
sodar-docker-compose-irods-1                  |   
sodar-docker-compose-irods-1                  |   Return code: 3
sodar-docker-compose-irods-1                  |   Standard output:
sodar-docker-compose-irods-1                  |     
sodar-docker-compose-irods-1                  |   Error output:
sodar-docker-compose-irods-1                  |     remote addresses: 172.19.0.5 ERROR: putUtil: put error for /sodarZone/home/rods/tmpyMSD6HdemoResc, status = -520013 status = -520013 UNIX_FILE_MKDIR_ERR, Permission denied
sodar-docker-compose-irods-1                  | iRODS server role: provider
mikkonie commented 11 months ago

Please check the permissions for the vault directory you have set for your iRODS resource. The error -520013 UNIX_FILE_MKDIR_ERR, Permission denied suggests the user has no permissions to write in that directory.

alabarga commented 11 months ago

actually that path /sodarZone/home/rods/ does not exist, should I create it?

I guess this is related to IRODS configuration in .env

IRODS_ZONE=sodarZone IRODS_ROOT_PATH=

but it is not clear for me (never used IRODS before)

alabarga commented 11 months ago

I gave write permission to all users on all folders under the volumes/ folder created with the init.sh script and that made the trick

mikkonie commented 11 months ago

Glad to hear it. I will need to review the documentation for this to make it more clear to users when encountering such issues. I'll get to it when finishing the iRODS 4.3 updates after the new year.

mikkonie commented 3 months ago

I believe this issue has been solved, closing.