Postgresql issue in Windows 10 for the Docker-Galaxy

[nmlbio]

We have developed a Docker-Galaxy based on the original docker-galaxy created by Dr. Grüning (many thanks!), our customization was mainly about installing more tools from the toolshed as well as updating the existing tools. We also added some genome data for tools such as deeptools, GATK, etc. Our docker-galaxy runs well on both Mac and Ubuntu boxes, but failed on Windows 10 computers due to its inability to start Postgresal (postgresql abnormal termination)---underlying database inaccessible. We have tried a few things but so far no luck. Any comments would be greatly appreciated!

YC

[bgruening]

:( I can not test the Windows images. But it seems to be a Windows Docker problem. Do you have the latest Docker version? Any error message?

[nmlbio]

Thanks so much for the comments, Dr. Grüning! We will try it with the latest Windows Docker and poke it around a bit more before an update!

YC

[kmshort]

This is a fairly major problem, it's a show stopper. It occurs for me as well. It works if I don't map a local drive to export. However, as soon as you map a local drive, it has issues with postgresql. This happens with current bgruening/docker-galaxy-stable, but I really just want to run the deeptools flavour.

docker run -d -p 8080:80 -p 8021:21 --privileged --pid=host -p 9002:9002 -e "GALAXY_LOGGING=full" -v d:/bioinformaticProcessing/galaxy_storage:/export/ quay.io/bgruening/galaxy-deeptools

then

Enable Galaxy reports authentification

Enable Galaxy Interactive Environments.

Starting postgres

postgresql: ERROR (abnormal termination)

Checking if database is up and running

Traceback (most recent call last):

File "/usr/local/bin/check_database.py", line 25, in

query.count()

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2980, in count

return self.from_self(col).scalar()

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2749, in scalar

ret = self.one()

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2718, in one

ret = list(self)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2761, in iter

return self._execute_and_instances(context)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2774, in _execute_and_instances

close_with_result=True)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/query.py", line 2765, in _connection_from_session

**kw)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 893, in connection

execution_options=execution_options)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/orm/session.py", line 900, in _connection_for_bind

conn = engine.contextual_connect(**kw)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2039, in contextual_connect

self._wrap_pool_connect(self.pool.connect, None),

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2078, in _wrap_pool_connect

e, dialect, self)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 1405, in _handle_dbapi_exception_noconnection

exc_info

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/util/compat.py", line 202, in raise_from_cause

reraise(type(exception), exception, tb=exc_tb, cause=cause)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/base.py", line 2074, in _wrap_pool_connect

return fn()

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 376, in connect

return _ConnectionFairy._checkout(self)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 713, in _checkout

fairy = _ConnectionRecord.checkout(pool)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 480, in checkout

rec = pool._do_get()

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1060, in _do_get

self._dec_overflow()

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/util/langhelpers.py", line 60, in exit

compat.reraise(exc_type, exc_value, exc_tb)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 1057, in _do_get

return self._create_connection()

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 323, in _create_connection

return _ConnectionRecord(self)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 449, in init

self.connection = self.__connect()

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/pool.py", line 607, in __connect

connection = self.__pool._invoke_creator(self)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/strategies.py", line 97, in connect

return dialect.connect(*cargs, **cparams)

File "/galaxy_venv/local/lib/python2.7/site-packages/sqlalchemy/engine/default.py", line 385, in connect

return self.dbapi.connect(*cargs, **cparams)

File "/galaxy_venv/local/lib/python2.7/site-packages/psycopg2/init.py", line 164, in connect

conn = _connect(dsn, connection_factory=connection_factory, async=async)

sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) could not connect to server: Connection refused

Is the server running on host "localhost" (127.0.0.1) and accepting

TCP/IP connections on port 5432?

[bgruening]

@kmshort are you using Windows? Which storage backend?

[kmshort]

Hi Björn! Thanks for getting back to me. It is the latest Windows 10, ver 2002.

The local filesystem is ntfs.

[kmshort]

@kmshort are you using Windows? Which storage backend?

Oops, that's version 1908 not 2004, sorry about that.

[bgruening]

@kmshort I don't have a Windows around. But this usually happens when you Docker installation is not totally functional. Docker has a storage-backend (this is not NFS) and if the Docker storage has some problems the PostgeSQL database can not come up. I have no idea how I can test this, unfortunately. :(

[kmshort]

@kmshort I don't have a Windows around. But this usually happens when you Docker installation is not totally functional. Docker has a storage-backend (this is not NFS) and if the Docker storage has some problems the PostgeSQL database can not come up. I have no idea how I can test this, unfortunately. :(

No, I'm 99% sure that this Docker Desktop 2.3.0.3 (v45519) is functional. This behaviour only happens once I map a local volume to /export. If I don't map the /export locally, the container runs quite happily. The container is I think trying to start the PostgreSQL using files from /export (mapped as the local volume) if it is there. If it were a pure docker problem, the container would fail when /export isn't mapped (I think).

This should be an easy thing to test, really. I'll try it on another windows box and see how I go.

[kmshort]

@bgruening Hi again Björn, I've tried again on a different Windows 10 machine (this time, confirmed 2004), and it has exactly the same behaviour. It is the same (latest) version of Docker Desktop (2.3.0.3, v45519). The container takes about 20 mins to start (I did not mention this previously) -- while I think it is copying/pulling down a lot of data... and then PostgreSQL fails to start.

Thanks for your help Björn, I really wish you could get a Win10 box, because this is super repeatable.

[bgruening]

The new 24.1 image contains a lot of changes and reflects the latest developments in Galaxy. I would like to close this PR, but please feel free to reopen and rebase against the latest version.

Please give it a try:

docker run -p 8080:80 -p 8021:21 -p 4002:4002 --privileged=true -e "GALAXY_DESTINATIONS_DEFAULT=slurm_cluster_docker"   -v /tmp/galaxy-data/:/export/ quay.io/bgruening/galaxy:24.1

... or any other combination. The readme has been updated. Please add any useful tip to it.

For a list of changes, see the Changelog.