NAL-i5K / genomics-workspace

Django website project for common sequence search tools.
http://genomics-workspace.readthedocs.io/
Other
17 stars 5 forks source link

Dockerize genomics-workspace #255

Open deming7h777 opened 6 years ago

deming7h777 commented 6 years ago

docker is very useful to rebuild the environment and whole configuration so I want to apply it to our genomics-workspace.

reference : 1. https://github.com/twtrubiks/docker-tutorial

  1. https://docs.docker.com/get-started/
deming7h777 commented 6 years ago

http://www.yamllint.com/

useful tool to check yml file

deming7h777 commented 6 years ago

image

It's docker-compose.yml I made for docker application

deming7h777 commented 6 years ago

I also create the other file name dockerfile, its function looks like vagrantfile while building the docker it will process the command in this file.

image

deming7h777 commented 6 years ago

however it shows error after running "docker-compose up" image

bradfordcondon commented 6 years ago

yaml can be annoying. Looking at yours, I see a whole bunch of problems, you have to be very careful with the indenting, new lines, -s etc. Compare it to the apollo one i wrote here: https://github.com/NAL-i5K/tripal_apollo/blob/master/docker-compose.yml

note for example

volumes:
 - .:/something

vs

volumes: pgdata:/something 

ie, different line, missing the dash. This is why its saying it isnt an array, its not defined as an indented set of -'d lines.

hsiaoyi0504 commented 6 years ago

Sorry... @deming7h777 @bradfordcondon Can I ask why we are rebuilding this ? As far as I know, there is a repo based on docker for genomics-workspace in our gitlab organization. If we want to update to newer version. That will be a good starting point.

deming7h777 commented 6 years ago

''' genomicsworkspace_workspace_1 exited with code 2 workspace_1 | python: can't open file 'manage.py': [Errno 2] No such file or directory ''' after build the four images for docker i encounter the problem that i can'y access manage.py

and i also push the branch call genomics-docker https://github.com/NAL-i5K/genomics-workspace/tree/genomics_docker

hsiaoyi0504 commented 6 years ago

Do you forget to clone the submodule ?

deming7h777 commented 6 years ago

Do you forget to clone the submodule ?

What kind of submodule you mean ?

hsiaoyi0504 commented 6 years ago

git submodule, check if your i5k directory is empty or not first.

deming7h777 commented 6 years ago

It's not empty. I also find that your i5k docker had one submodule but i don't know what's its usage.

hsiaoyi0504 commented 6 years ago

Can you provide the result when you run the tree command ? It's hard for anyone to help you without providing further detailed information and I am confused about the error you encountered here is that on your branch or the internal i5k-docker repo.

deming7h777 commented 6 years ago

Ah, I can try it myself first, thanks for your kindness ! The error I encountered was on my branch, what I want to do is updating the old version to docker-compose

hsiaoyi0504 commented 6 years ago

Got it ! I am not sure what you mean old version here. Do you mean the internal i5k-docker repo ?

deming7h777 commented 6 years ago

branch link : https://github.com/NAL-i5K/genomics-workspace/blob/genomics_docker/docker-compose.yml

deming7h777 commented 6 years ago

/bin/sh: ./configure: No such file or directory ERROR: Service 'workspace' failed to build: The command '/bin/sh -c ./configure --prefix=/usr/local --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" && make && make altinstall' returned a non-zero code: 127

now I'm trying to figure out linux problem when I try to build cutomize image for workspace

deming7h777 commented 5 years ago
configure: error: in `/genomics-workspace/Python-2.7.15':
configure: error: C compiler cannot create executables
See `config.log' for more details
ERROR: Service 'workspace' failed to build: The command '/bin/sh -c cd Python-2.7.15 &&     ./configure --prefix=/usr/lo
cal --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" &&     make &&     make altinstall' return
ed a non-zero code: 77

I change the code in Dockerfile from WORKDIR Python-2.7.15 to RUN cd Python-2.7.15 to solve the above problem.

In the current issue, it seems I need to figure out the problem of c compiler

deming7h777 commented 5 years ago
i change it to this and the build proceeds to `RUN yum groupinstall "Development tools" -y && yum install wget -y` and fails
because yum isnt installed
what exactly is your question
childers commented 5 years ago

I get an error on step 11:

$ docker-compose up
Creating network "genomics-workspace_default" with the default driver
Building workspace
Step 1/13 : FROM centos:6.10
 ---> 30e66b619e9f
Step 2/13 : LABEL maintainer Deming
 ---> Using cache
 ---> 2fda5d15f394
Step 3/13 : ENV PYTHONBUFFERED 1
 ---> Using cache
 ---> 52eba52b63c3
Step 4/13 : VOLUME /data
 ---> Using cache
 ---> ed9ad883dfbe
Step 5/13 : RUN mkdir /genomics-workspace
 ---> Using cache
 ---> 7a04d699d6fe
Step 6/13 : WORKDIR /genomics-workspace
 ---> Using cache
 ---> fb278825b72d
Step 7/13 : COPY requirements.txt /genomics-workspace/
 ---> Using cache
 ---> c42e9156239c
Step 8/13 : RUN yum groupinstall "Development tools" -y && yum install wget -y
 ---> Using cache
 ---> 94e2bc2416b5
Step 9/13 : RUN wget http://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz && tar -xf Python-2.7.15.tar.xz
 ---> Using cache
 ---> 7f26b413d79e
Step 10/13 : WORKDIR /Python-2.7.15
 ---> Using cache
 ---> a2a50cbaa3d9
Step 11/13 : RUN yum install automake && yum install autoconf && autoreconf -i && ./configure --prefix=/usr/local --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" && make && make altinstall
 ---> Running in b8fea31fcd41
Loaded plugins: fastestmirror, ovl
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirror.umd.edu
 * extras: mirror.umd.edu
 * updates: mirror.umd.edu
Package automake-1.11.1-4.el6.noarch already installed and latest version
Nothing to do
Loaded plugins: fastestmirror, ovl
Setting up Install Process
Loading mirror speeds from cached hostfile
 * base: mirror.umd.edu
 * extras: mirror.umd.edu
 * updates: mirror.umd.edu
Package autoconf-2.63-5.1.el6.noarch already installed and latest version
Nothing to do
autoreconf: `configure.ac' or `configure.in' is required
ERROR: Service 'workspace' failed to build: The command '/bin/sh -c yum install automake && yum install autoconf && autoreconf -i && ./configure --prefix=/usr/local --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" && make && make altinstall' returned a non-zero code: 1
deming7h777 commented 5 years ago
Complete!
checking build system type... x86_64-pc-linux-gnu
checking host system type... x86_64-pc-linux-gnu
checking for python2.7... no
checking for python3... no
checking for python... python
checking for --enable-universalsdk... no
checking for --with-universal-archs... no
checking MACHDEP... linux2
checking EXTRAPLATDIR...
checking for --without-gcc... no
checking for --with-icc... no
checking for gcc... gcc
checking whether the C compiler works... no
configure: error: in `/genomics-workspace/Python-2.7.15':
configure: error: C compiler cannot create executables
See `config.log' for more details
ERROR: Service 'workspace' failed to build: The command '/bin/sh -c cd Python-2.7.15 &&     yum install -y gcc gcc-c++ zlib-devel gdbm-devel readline-devel libffi-devel sqlite-devel &&    ./configure --prefix=/usr/local --enable-unicode=usc4 --enable-shared LDFLAGS="-WL,-rpath /usr/local/lib" &&     make &&     make altinstall' returned a non-zero code: 77
deming7h777 commented 5 years ago

image

image

It's great to solve the c compiler problem above then now I'm going to link

deming7h777 commented 5 years ago
FROM centos:6.10
LABEL maintainer Deming
ENV PYTHONBUFFERED 1
RUN mkdir /genomics-workspace
WORKDIR /genomics-workspace
COPY  . /genomics-workspace
RUN yum -y groupinstall "Development tools" && \
    yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel && \
    yum -y install readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel python-devel && \
    yum install wget -y
RUN wget http://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz && \
    tar -xf Python-2.7.15.tar.xz
RUN cd Python-2.7.15 && \
    ./configure --enable-optimizations  && \
    make && \
    make altinstall && \
    wget https://bootstrap.pypa.io/ez_setup.py && \
    /usr/local/bin/python2.7 ez_setup.py && \
    wget https://bootstrap.pypa.io/get-pip.py && \
    /usr/local/bin/python2.7 get-pip.py
RUN pip install -r requirements.txt
FROM python:2.7-slim

RUN mkdir /genomics-workspace

WORKDIR /genomics-workspace

COPY . /genomics-workspace/

RUN pip install -r requirements.txt

I tried these two customize dockerfile to build my own genomics-workspace image but it turn out the same error for "can't file the file name manage.py"

bradfordcondon commented 5 years ago

the docker-compose runs without error on my system after:

So I no longer get db connection errors etc from the workspace container, meaning its hooked up right... but, i get ERR_EMPTY_RESPONSE at the web address. I dont know anything about the web stack there, perhaps apache or whatever is running needs to be restarted?

bradfordcondon commented 5 years ago

warning: my knowledge of Django is nonexistant.

you need to supply the IP address in the run command, like so

command: python manage.py runserver 0.0.0.0:8000

now, instead of an empty response error, i get a routing type error from Django, yay!

screen shot 2019-01-30 at 12 39 38 pm

So from this error i can see that we only define certain paths. so if i try to visit, say, http://localhost:8000/blast/ instead, i get this error that shows me stuff is definitely happening, but perhaps not configured all the way:

screen shot 2019-01-30 at 12 41 47 pm

the admin address also doesnt work:

ProgrammingError at /admin/login/
relation "django_site" does not exist
LINE 1: ..."django_site"."domain", "django_site"."name" FROM "django_si...
                                                             ^

so looks to me like the django site isnt configured and perhaps additional setup commands or config files need to be done.

hsiaoyi0504 commented 5 years ago

@bradfordcondon You are right. We didn’t define root path and the error message means no datatable has been created. I guess python manage.py migrate hasn’t been included into the setup steps.

bradfordcondon commented 5 years ago

the only command run is the server start command specified in the docker-compose. You all probably need to run a full script that runs each command necessary to setting up and launching the django site.

deming7h777 commented 5 years ago

I didn't put migrate commend in the dockerfile because I thought we can build the container first then do the migrate commend directly in container.

bradfordcondon commented 5 years ago

in an ideal world i should just run docker compose up and have the server start and be functional. Any additional commands necessary to get the site functional should really go in teh startup script/dockerfile.

deming7h777 commented 5 years ago

however with the same repo and docker file. I got a missing file issue that Bradford didnt pop up

in an ideal world i should just run docker compose up and have the server start and be functional. Any additional commands necessary to get the site functional should really go in teh startup script/dockerfile.

yes that's true i'll add those commend

hsiaoyi0504 commented 5 years ago

@deming7h777 Can you provide more detailed information about that ? Like showing the full terminal messages including running the pwd and tree commands ?

deming7h777 commented 5 years ago

I just solve the issue "can't file the file name manage.py [no such file or directory] ", it seems like a specific issue for windows user. Docker get the "mounting the volume from local machine to container " issue itself and I'm now looking for the alternative for this . Also try to connect postgres from our genomics web

django.db.utils.OperationalError: could not connect to server: Connection refused
workspace_1  |  Is the server running on host "localhost" (127.0.0.1) and accepting
workspace_1  |  TCP/IP connections on port 5432?
workspace_1  | could not connect to server: Cannot assign requested address
workspace_1  |  Is the server running on host "localhost" (::1) and accepting
workspace_1  |  TCP/IP connections on port 5432?
bradfordcondon commented 5 years ago

I'm glad you got it working!

it wont be localhost for the hostname. See my note above:

additionally, changing the settings.py file HOST settings from localhost to postgres as docker makes the containers available under whatever you name that as.

deming7h777 commented 5 years ago

ah I see and I also try to change my postgres from port 5432 to expose 5432 but I got this kind of error

ERROR: The Compose file '.\docker-compose.yml' is invalid because:
services.postgres.expose is invalid: should be of the format 'PORT[/PROTOCOL]'
deming7h777 commented 5 years ago
 django.db.utils.OperationalError: could not translate host name "postgres" to address: Name or service not known

after changing the configuration in settings.py from host: "localhost" to host: "postgres" it pops out this error

bradfordcondon commented 5 years ago

you wrote postgrest not postgres. Doublecheck your spelling and/or share it here.

This is my functional settings.py file

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'postgres',
        'USER': 'postgres',
        'PASSWORD': 'django1234',
        'HOST': 'postgres',
        'PORT': '5432',
    }
}
deming7h777 commented 5 years ago

you wrote postgrest not postgres. Doublecheck your spelling and/or share it here.

This is my functional settings.py file

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql_psycopg2',
        'NAME': 'postgres',
        'USER': 'postgres',
        'PASSWORD': 'django1234',
        'HOST': 'postgres',
        'PORT': '5432',
    }
}

ah i just spell wrong for my comment but it is correct in my settings.py. in previous comment for error message it shows host postgrest can not be translate

bradfordcondon commented 5 years ago

ok, this is a windows bug: see https://github.com/getsentry/onpremise/issues/32

If you comment out the volumes directives it should work. (hopefully)

deming7h777 commented 5 years ago

image after adding ALLOWED_HOST=['192.168.99.100','localhost','127.0.0.1'] in settings.py

deming7h777 commented 5 years ago

![image](https://user-images.githubusercontent.com/33312760/52349204-0722f000-29f4-11e9-92e7-cfabb04879d5.png

Genomics web success to connect the db but shows the error that @bradfordcondon post before

relation "blast_blastdb" does not exist
LINE 1: ...e_type", "blast_sequencetype"."dataset_type" FROM "blast_bla..

it seems that there are some db configuration need to be modified

deming7h777 commented 5 years ago

image

it does success to work and I'm now trying to make the bash file that we can run python manage.py migrate automatically

deming7h777 commented 5 years ago

workspace_1 | standard_init_linux.go:190: exec user process caused "exec format error" now solving the error above after I made the run_web.sh which include the code

#!/usr/bin/bash
cd genomics-workspace/
python manage.py migrate
python manage.py collectstatic --noinput
python manage.py runserver 0.0.0.0:8000

and changed the command to command: ["./run_web.sh"] in docker-compose.yml for workspace container

deming7h777 commented 5 years ago

The reason that causes the above issue because the startup script is dos but it should be modified to Unix format on the linux system (docker startup the whole project via virtual machine with the linux system)

deming7h777 commented 5 years ago

The testing report from @bradfordcondon after changing bind to expose:

Starting 963fc9559b1a_postgres ... done
Recreating genomics-workspace_workspace_1 ... error

ERROR: for genomics-workspace_workspace_1  Cannot start service workspace: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"./run_web.sh\": stat ./run_web.sh: no such file or directory": unknown

ERROR: for workspace  Cannot start service workspace: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"./run_web.sh\": stat ./run_web.sh: no such file or directory": unknown
ERROR: Encountered errors while bringing up the project.
________________________________________________________________________________
| bc:~/i5k/genomics-workspace (genomics_docker)$
bradfordcondon commented 5 years ago

could this be the Python package stat dependency? import stat as Perm?

hsiaoyi0504 commented 5 years ago

In my point of view, I don’t think it’s related to Python package. The stat mentioned here should be stat linux command.

deming7h777 commented 5 years ago

In my point of view, I don’t think it’s related to Python package. The stat mentioned here should be stat linux command.

I agree of that. Is it possible because of the docker version ?

deming7h777 commented 5 years ago

docker version Client: Version: 18.03.0-ce API version: 1.37 Go version: go1.9.4 Git commit: 0520e24302 Built: Fri Mar 23 08:31:36 2018 OS/Arch: windows/amd64 Experimental: false Orchestrator: swarm

Server: Engine: Version: 18.06.1-ce API version: 1.38 (minimum version 1.12) Go version: go1.10.3 Git commit: e68fc7a Built: Tue Aug 21 17:28:38 2018 OS/Arch: linux/amd64 Experimental: false

docker-compose version docker-compose version 1.20.1, build 5d8c71b2 docker-py version: 3.1.4 CPython version: 3.6.4 OpenSSL version: OpenSSL 1.0.2k 26 Jan 2017

deming7h777 commented 5 years ago

keep some log here

workspace_1  | Traceback (most recent call last):
workspace_1  |   File "manage.py", line 8, in <module>
workspace_1  |     from django.core.management import execute_from_command_line
workspace_1  | ImportError: No module named django.core.management
workspace_1  | Traceback (most recent call last):
workspace_1  |   File "manage.py", line 8, in <module>
workspace_1  |     from django.core.management import execute_from_command_line
workspace_1  | ImportError: No module named django.core.management
genomicsworkspace_workspace_1 exited with code 1

with dockerfile

FROM centos:6.10
LABEL maintainer Deming
ENV PYTHONBUFFERED 1
RUN mkdir /genomics-workspace
WORKDIR /genomics-workspace
COPY  . /genomics-workspace
RUN yum install curl -y
RUN yum -y groupinstall "Development tools" && \
    yum -y install zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel && \
    yum -y install readline-devel tk-devel gdbm-devel db4-devel libpcap-devel xz-devel python-devel && \
    yum install wget -y
RUN wget http://www.python.org/ftp/python/2.7.15/Python-2.7.15.tar.xz && \
    tar -xf Python-2.7.15.tar.xz
RUN cd Python-2.7.15 && \
    ./configure --enable-optimizations  && \
    make && \
    make altinstall && \
    wget https://bootstrap.pypa.io/ez_setup.py && \
    /usr/local/bin/python2.7 ez_setup.py && \
    wget https://bootstrap.pypa.io/get-pip.py && \
    /usr/local/bin/python2.7 get-pip.py
WORKDIR /genomics-workspace
RUN pip install -r requirements.txt
RUN curl -sL https://rpm.nodesource.com/setup_11.x | bash - && \
    yum install -y nodejs
RUN npm run build
mpoelchau commented 5 years ago

@deming7h777 I tried setting up genomics_docker and got the following error:

lots of output, then:

Successfully built 692edce39382
Successfully tagged genomics-workspace_workspace:latest
WARNING: Image for service workspace was built because it did not already exist. To rebuild this image you must use `docker-compose build` or `docker-compose up --build`.
Creating genomics-workspace_memcache_1 ... done
Creating postgres                      ... done
Creating genomics-workspace_rabbitmq_1 ... done
Creating genomics-workspace_workspace_1 ... error

ERROR: for genomics-workspace_workspace_1  Cannot start service workspace: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"./run_web.sh\": permission denied": unknown

ERROR: for workspace  Cannot start service workspace: OCI runtime create failed: container_linux.go:344: starting container process caused "exec: \"./run_web.sh\": permission denied": unknown
ERROR: Encountered errors while bringing up the project.
deming7h777 commented 5 years ago

container_linux.go:344: starting container process caused "exec: \"./run_web.sh\": permission denied": unknown ERROR: Encountered errors while bringing up the project.

After cloning the repo but before you build the container, navigate to the directory that you cloned the repo and run:

sudo chmod +x ./run_web.sh