search

galaxyFlavorsGenerators / galaxyFlavorGenerator

Configure and generate Galaxy server instances
http://galaxyflavorsgenerators.github.io/galaxyFlavorGenerator/
7 stars 2 forks source link

Cant build due to postgres issue #22

Open doomedramen opened 8 years ago

doomedramen commented 8 years ago

Dear Martin (Page),

I have tried the Galaxy Flavor Generator (using http://bit.ly/galaxy-flavor-generator)

I have selected only a few programs (see Dockerfile at the end of my email). When I try to create a docker image with sudo docker build -t galaxyexomfastqcutadapt .

(I am running docker as the galaxy user which is a sudoer)

I get the following error:

[...]

It seems to be a problem with docker and postgresql (https://github.com/nimiq/docker-postgresql93/issues/2)

Any help appreciated!

Best regards, Christian Rausch VU University Medical Center Amsterdam

Dockerfile:

FROM bgruening/galaxy-exom-seq

MAINTAINER Björn A. Grüning, bjoern.gruening@gmail.com

ENV GALAXY_CONFIG_BRAND "Galaxy"

WORKDIR /galaxy-central

RUN install-repository \
   "--url https://toolshed.g2.bx.psu.edu/ -o iuc --name fastqc" \
   "--url https://toolshed.g2.bx.psu.edu/ -o lparsons --name cutadapt" \
   "--url https://toolshed.g2.bx.psu.edu/ -o bgruening --name
trim_galore" \
   "--url https://toolshed.g2.bx.psu.edu/ -o devteam --name
varscan_version_2" \
   "--url https://toolshed.g2.bx.psu.edu/ -o geert-vandeweyer --name
varscan_wrapper" 

VOLUME ["/export/", "/data/", "/var/lib/docker"]

EXPOSE :80
EXPOSE :21
EXPOSE :8080

CMD ["/usr/bin/startup"]
doomedramen commented 8 years ago

I believe this would need to fixed at the base container level (https://hub.docker.com/r/bgruening/galaxy-stable/)

bgruening commented 8 years ago

Hi Christian,

this is probably due to a docker bug (?) with your storage backend. Please change your storage backend to aufs, I suppose you are using devicemapper?

Cheers, Bjoern

martenson commented 8 years ago

@bgruening Christian is not a part of the conversation here. @wookoouk could you please point him here?

doomedramen commented 8 years ago

Didn't I link him to it in the email?

On Wed, 4 Nov 2015, 15:11 Martin Cech notifications@github.com wrote:

@bgruening https://github.com/bgruening Christian is not a part of the conversation here. @wookoouk https://github.com/wookoouk could you please point him here?

— Reply to this email directly or view it on GitHub https://github.com/galaxyFlavorsGenerators/galaxyFlavorGenerator/issues/22#issuecomment-153756860 .

martenson commented 8 years ago

I do not see the email you sent him.

doomedramen commented 8 years ago

ah, ok @martenson, I did link him in the email (I cc'd you) so hopefully he will follow the link, if now Ill update him via email on the progress.

martenson commented 8 years ago

Inbox hiding stuff from me. I am sorry for the confusion.

chrausch commented 8 years ago

it seems that the bug is fixed when upgrading to postgresql-9.4 (Ubuntu 14.04 has version 9.3). I have reproduced the bug when running docker in interactive mode:

    docker run -i -t bgruening/galaxy-stable /bin/bash
    service postgresql start

Then I upgraded to postgresql 9.4 with:

    apt-get remove -y postgresql-9.3 postgresql-client-9.3
    apt-get update
    apt-get install -y software-properties-common
    add-apt-repository "deb https://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
    wget --quiet -O - https://postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
    apt-get update
    apt-get install -y postgresql-9.4

Then it works. I have then tried to bring this into a Dockerfile and used Björn's "galaxy-exom-seq" as starting point:

# CCA Drylab Galaxy
#
# VERSION       0.1
FROM bgruening/galaxy-stable:latest
MAINTAINER Christian Rausch, c.rausch@vumc.nl
ENV GALAXY_CONFIG_BRAND CCA Drylab Galaxy
WORKDIR /galaxy-central

# Enable TTS installation
ENV GALAXY_CONFIG_TOOL_SHEDS_CONFIG_FILE $GALAXY_HOME/tool_sheds_conf.xml

RUN apt-get remove -y postgresql-9.3 postgresql-client-9.3
RUN apt-get update
RUN sudo apt-get install -y software-properties-common
RUN add-apt-repository "deb https://apt.postgresql.org/pub/repos/apt/ trusty-pgdg main"
RUN wget --quiet -O - https://postgresql.org/media/keys/ACCC4CF8.asc | sudo apt-key add -
RUN apt-get update
RUN apt-get install -y postgresql-9.4

# Install samtools and
RUN install-repository \
    "--url http://toolshed.g2.bx.psu.edu/ -o devteam --name suite_samtools_0_1_19 --panel-section-name SAMTools" \
    "--url http://toolshed.g2.bx.psu.edu/ -o devteam --name freebayes --panel-section-name Freebayes"
# Bowtie & FASTQC
RUN install-repository \
    "--url http://toolshed.g2.bx.psu.edu/ -o devteam --name bowtie2 --panel-section-name Mapping" \
    "--url http://toolshed.g2.bx.psu.edu/ -o devteam --name fastqc --panel-section-name QualityControl"
RUN install-repository \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name package_pysam_0_7_7" \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name package_scipy_0_12" \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name package_matplotlib_1_2" \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name package_bx_python_12_2013"
RUN install-repository \
    "--url https://toolshed.g2.bx.psu.edu/ --name variant_select --owner devteam"\
    "--url https://toolshed.g2.bx.psu.edu/ --name variant_filtration --owner devteam"
RUN install-repository \
    "--url http://toolshed.g2.bx.psu.edu/ -o bgruening --name deeptools --panel-section-name deepTools" \
    "--url http://toolshed.g2.bx.psu.edu/ -o devteam --name data_manager_bwa_index_builder" \
    "--url http://toolshed.g2.bx.psu.edu/ -o devteam --name bwa_wrappers --panel-section-name Mapping" \
    "--url http://toolshed.g2.bx.psu.edu/ -o crs4 --name bwa_mem  --panel-section-name Mapping"
RUN install-repository \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name vt_variant_tools --panel-section-name VCFtools" \
    "--url http://toolshed.g2.bx.psu.edu/ -o anton --name suite_vcflib_tools_2_0 --panel-section-name VCFtools" \
    "--url https://toolshed.g2.bx.psu.edu/ -o iuc --name snpsift_genesets" \
    "--url https://toolshed.g2.bx.psu.edu/ -o iuc --name snpsift_dbnsfp" \
    "--url https://toolshed.g2.bx.psu.edu/ -o iuc --name snpsift" \
    "--url https://toolshed.g2.bx.psu.edu/ -o iuc --name snpsift_dbnsfp_datatypes" \
    "--url https://toolshed.g2.bx.psu.edu/ -o iuc --name snpsift_dbnsfp_generic"
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name snpeff --panel-section-name SNPEff"
# Install GEMINI and BED tools
RUN install-repository \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name package_tabix_0_2_6" \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name package_bedtools_2_19" \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name package_grabix_0_1_3" \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name package_gemini_0_10_0" \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name data_manager_gemini_database_downloader" \
    "--url http://toolshed.g2.bx.psu.edu/ -o iuc --name gemini --panel-section-name VariantSelection" \
    "--url http://testtoolshed.g2.bx.psu.edu/ -o iuc --name bedtools --panel-section-name BEDtools"

RUN install-repository "--url http://toolshed.g2.bx.psu.edu/ -o crs4 --name kggseq_variant_selection --panel-section-name VariantSelection"
# Mark folders as imported from the host.
VOLUME ["/export/", "/data/", "/var/lib/docker"]
# Expose port 80 (webserver), 21 (FTP server), 8800 (Proxy)
EXPOSE :80
EXPOSE :21
EXPOSE :8800
# Autostart script that is invoked during container start
CMD ["/usr/bin/startup"]

When I create an image from the Dockerfile with

sudo docker build -t chrisatdocker/galaxy-drylab .

I get:

[...]
* Starting PostgreSQL 9.4 database server
   ...done.
starting Galaxy
Galaxy could not be started.
More information about this failure may be found in the following log snippet from galaxy_install.log:
========================================
migrate.versioning.repository DEBUG 2015-11-05 11:46:41,528 Config: OrderedDict([('db_settings', OrderedDict([('__name__', 'db_settings'), ('repository_id', 'Galaxy'), ('version_table', 'migrate_version'), ('required_dbs', '[]')]))])
galaxy.model.migrate.check DEBUG 2015-11-05 11:46:41,533 psycopg2 egg successfully loaded for postgresql dialect
Traceback (most recent call last):
  File "/galaxy-central/lib/galaxy/webapps/galaxy/buildapp.py", line 64, in paste_app_factory
    app = galaxy.app.UniverseApplication( global_conf=global_conf, **kwargs )
  File "/galaxy-central/lib/galaxy/app.py", line 54, in __init__
    self._configure_models( check_migrate_databases=True, check_migrate_tools=check_migrate_tools, config_file=config_file )
  File "/galaxy-central/lib/galaxy/config.py", line 826, in _configure_models
    create_or_verify_database( db_url, config_file, self.config.database_engine_options, app=self )
  File "/galaxy-central/lib/galaxy/model/migrate/check.py", line 68, in create_or_verify_database
    dataset_table = Table( "dataset", meta, autoload=True )
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/sql/schema.py", line 416, in __new__
    metadata._remove_table(name, schema)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/sql/schema.py", line 411, in __new__
    table._init(name, metadata, *args, **kw)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/sql/schema.py", line 484, in _init
    self._autoload(metadata, autoload_with, include_columns)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/sql/schema.py", line 508, in _autoload
    self, include_columns, exclude_columns
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1958, in run_callable
    with self.contextual_connect() as conn:
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2026, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2065, in _wrap_pool_connect
    e, dialect, self)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 1395, in _handle_dbapi_exception_noconnection
    exc_info
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/compat.py", line 199, in raise_from_cause
    reraise(type(exception), exception, tb=exc_tb)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/base.py", line 2061, in _wrap_pool_connect
    return fn()
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/pool.py", line 341, in connect
    return _ConnectionFairy._checkout(self)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/pool.py", line 648, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/pool.py", line 443, in checkout
    rec = pool._do_get()
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/pool.py", line 981, in _do_get
    self._dec_overflow()
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/util/langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/pool.py", line 978, in _do_get
    return self._create_connection()
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/pool.py", line 288, in _create_connection
    return _ConnectionRecord(self)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/pool.py", line 414, in __init__
    self.connection = self.__connect()
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/pool.py", line 542, in __connect
    connection = self.__pool._creator()
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/strategies.py", line 89, in connect
    return dialect.connect(*cargs, **cparams)
  File "/galaxy-central/eggs/SQLAlchemy-1.0.1-py2.7-linux-x86_64-ucs4.egg/sqlalchemy/engine/default.py", line 377, in connect
    return self.dbapi.connect(*cargs, **cparams)
  File "/galaxy-central/eggs/psycopg2-2.5.1_9.2.4_static-py2.7-linux-x86_64-ucs4.egg/psycopg2/__init__.py", line 164, in connect
    conn = _connect(dsn, connection_factory=connection_factory, async=async)
OperationalError: (psycopg2.OperationalError) FATAL:  password authentication failed for user "galaxy"

Has anybody a clue why this fails? -- greetings, Christian

bgruening commented 8 years ago

The postgresql setup is way more complex than it seems. We are setting default users and passwords and we need to take care that it is possible to store the DB outside of the container. I think it is a bad idea to change this in a child container, as this will break everything what was build in the parents, including the ansible playbook setup. @chrausch have you tried to change your Docker backend? Does this not solve your problem?