fail to start studio4mychem container (likely other api instances too)

[newgene]

 make studio4mychem works just fine. However, when start the container from this image:

docker run --rm --name studio4mychem -p 8080:8080 -p 7022:7022 -p 7080:7080 -p 9001:9000 -d studio4mychem:master

hub server cannot be started correctly.

The error came from this line in run_api script.

for mychem case, app_name variable is still biothings_studio, the correct command should be

cd ~/mychem.info/src; python bin/hub.py

(note, also not sure what -a . part is for)

[zcqian]

Is this specific to mychem?

[newgene]

I think all specific api instances need to start hub this way.

[erikyao]

make studio4mychem works, but make studio4myvariant failed. Got error message at Dockerfile instruction 14:

yaoyao@ScrippsMBP docker % make studio4myvariant
docker build --force-rm \
    --build-arg STUDIO_VERSION=master \
    --build-arg BIOTHINGS_VERSION=master \
    --build-arg API_NAME=myvariant.info \
    --build-arg API_VERSION=master \
    -t studio4myvariant:$(git branch | grep ^\* | sed "s#\* ##") .
[+] Building 59.8s (19/22)                                                                                                                                                                          
 => [internal] load build definition from Dockerfile                                                                                                                                           0.0s
 => => transferring dockerfile: 37B                                                                                                                                                            0.0s
 => [internal] load .dockerignore                                                                                                                                                              0.0s
 => => transferring context: 2B                                                                                                                                                                0.0s
 => [internal] load metadata for docker.io/library/ubuntu:20.04                                                                                                                                1.4s
 => [auth] library/ubuntu:pull token for registry-1.docker.io                                                                                                                                  0.0s
 => [internal] load build context                                                                                                                                                              0.0s
 => => transferring context: 1.02kB                                                                                                                                                            0.0s
 => [ 1/17] FROM docker.io/library/ubuntu:20.04@sha256:b3e2e47d016c08b3396b5ebe06ab0b711c34e7f37b98c9d37abe794b71cea0a2                                                                        0.0s
 => CACHED [ 2/17] RUN if [ -z "master" ]; then echo "NOT SET - use --build-arg BIOTHINGS_VERSION=..."; exit 1; else : ; fi                                                                    0.0s
 => CACHED [ 3/17] RUN if [ -z "master" ]; then echo "NOT SET - use --build-arg STUDIO_VERSION=..."; exit 1; else : ; fi                                                                       0.0s
 => CACHED [ 4/17] RUN apt-get -qq -y update &&     apt-get install -y --no-install-recommends     gnupg1     curl     ca-certificates     lsb-release &&     release=`lsb_release -sc` &&     0.0s
 => CACHED [ 5/17] WORKDIR /tmp                                                                                                                                                                0.0s
 => CACHED [ 6/17] RUN curl -LO     https://github.com/lmenezes/cerebro/releases/download/v0.9.4/cerebro-0.9.4.tgz     && tar xzf cerebro-0.9.4.tgz -C /usr/local     && ln -s /usr/local/cer  0.0s
 => CACHED [ 7/17] RUN git clone http://github.com/ansible/ansible.git /tmp/ansible                                                                                                            0.0s
 => CACHED [ 8/17] WORKDIR /tmp/ansible                                                                                                                                                        0.0s
 => CACHED [ 9/17] RUN ln -sv /usr/bin/python3 bin/python                                                                                                                                      0.0s
 => CACHED [10/17] ADD ansible_playbook /tmp/ansible_playbook                                                                                                                                  0.0s
 => CACHED [11/17] ADD inventory /etc/ansible/hosts                                                                                                                                            0.0s
 => CACHED [12/17] RUN echo "GITHASH = $GITHASH"                                                                                                                                               0.0s
 => CACHED [13/17] WORKDIR /tmp/ansible_playbook                                                                                                                                               0.0s
 => ERROR [14/17] RUN if [ -n "myvariant.info" ];     then         ansible-playbook studio4api.yml             -e "biothings_version=master"             -e "studio_version=master"           58.3s
------ 

#19 47.58 TASK [Install API requirements] ************************************************
#19 57.90 fatal: [localhost]: FAILED! => {"changed": true, "cmd": "source $HOME/pyenv/bin/activate && pip install -r requirements_hub.txt", "delta": "0:00:10.163593", "end": "2021-07-16 20:09:49.352812", "msg": "non-zero return code", "rc": 1, "start": "2021-07-16 20:09:39.189219", "stderr": "  Running command git clone -q https://github.com/biothings/biothings.api.git /home/biothings/pyenv/src/biothings
  ERROR: Command errored out with exit status 1:
   command: /home/biothings/pyenv/bin/python -u -c 'import io, os, sys, setuptools, tokenize; sys.argv[0] = '\"'\"'/tmp/pip-install-olyk3kpy/bitarray_224e55fbd9954d6abd1d74de0494560d/setup.py'\"'\"'; __file__='\"'\"'/tmp/pip-install-olyk3kpy/bitarray_224e55fbd9954d6abd1d74de0494560d/setup.py'\"'\"';f = getattr(tokenize, '\"'\"'open'\"'\"', open)(__file__) if os.path.exists(__file__) else io.StringIO('\"'\"'from setuptools import setup; setup()'\"'\"');code = f.read().replace('\"'\"'\\r\\n'\"'\"', '\"'\"'\\n'\"'\"');f.close();exec(compile(code, __file__, '\"'\"'exec'\"'\"'))' bdist_wheel -d /tmp/pip-wheel-9j5drx3p
       cwd: /tmp/pip-install-olyk3kpy/bitarray_224e55fbd9954d6abd1d74de0494560d/

# Logs omitted...

  unable to execute 'x86_64-linux-gnu-gcc': No such file or directory
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

# Logs omitted...

#19 57.90 PLAY RECAP *********************************************************************
#19 57.90 localhost                  : ok=44   changed=40   unreachable=0    failed=1    skipped=0    rescued=0    ignored=0   
#19 57.90 
------
executor failed running [/bin/sh -c if [ -n "$API_NAME" ];     then         ansible-playbook studio4api.yml             -e "biothings_version=$BIOTHINGS_VERSION"             -e "studio_version=$STUDIO_VERSION"             -e "api_name=$API_NAME"             -e "api_version=$API_VERSION"             -c local;     else         ansible-playbook biothings_studio.yml             -e "biothings_version=$BIOTHINGS_VERSION"             -e "studio_version=$STUDIO_VERSION"             -c local; fi]: exit code: 2

Looks like x86_64-linux-gnu-gcc is not installed.

[erikyao]

Fix: need gcc and python3-dev to make studio4myvariant

apt-get -y install --no-install-recommends \
        # base
        ......
        gcc \  # new package  
        python3 \
        python3-dev \  # new package

[zcqian]

I'll take note of all these issues and work on a fix.

Maybe I forgot to test running the containers, only building. I'll make sure everything works.

[erikyao]

studio4mychem right now cannot index correctly because its DrugIndexer is using the old index method signature. Therefore new release functionality cannot be tested yet.

Error messages are like:

09:21:59 [3157:MainThread] - indexmanager - ERROR -- Error while running index job:
Traceback (most recent call last):
  File "/home/biothings/mychem.info/src/biothings/hub/dataindex/indexer.py", line 578, in indexed
    res = f.result()
  File "/usr/lib/python3.8/asyncio/coroutines.py", line 124, in coro
    res = func(*args, **kw)
TypeError: index() missing 2 required positional arguments: 'index_name' and 'job_manager'

[zcqian]

Fix: need gcc and python3-dev to make studio4myvariant

apt-get -y install --no-install-recommends \
        # base
        ......
        gcc \  # new package  
        python3 \
        python3-dev \  # new package

Regarding this workaround, do we plan on including these tools in the docker image? It will certainly make the image very bloated. @newgene

I will try to figure out which Python package requires building from source and see if I can do something about it, like in https://github.com/biothings/biothings_studio/commit/cca5cdc054041de9c4d1d5ed6fd2205dd5cdd73c where the Web App is built in another container but and then copied to the final image, so that the final image would not have node.js dependencies.

[newgene]

@zcqian agree, would be ideal if we can avoid including these build packages in the final container.

[newgene]

On option is you can pre-build the binary package and put it somewhere downloadable, and install it during the docker build.

I suspect it was bitarray package requiring building from src.

[zcqian]

So I tried to build the image and run it, and it's running for me correctly.

Let me explain what is going on. The run_api script will always run the bin/hub.py script from this repo. However, if the -a option is given, it will load the HubServer instance from a specific API, in this case MyChem, and run that server. This is implemented here https://github.com/biothings/biothings_studio/blob/cca5cdc054041de9c4d1d5ed6fd2205dd5cdd73c/bin/hub.py#L28-L66

The reason that the hub wasn't properly starting may be related to the refactoring we are doing in the biothings.api repo. Right now the vanilla studio doesn't start because how we modified the configuration helper classes. This will be fixed soon.

MyVariant is a different issue but I don't think we need to open a separate issue for it.

[zcqian]

I can confirm it's bitarray. Let me figure out what's the best way to do it.

[zcqian]

Everything should be working now.