buildlocal.sh script fails

FelixErnst commented 6 years ago

Hi

running the buildlocal.sh script fails on ubuntu 18.04. The following docker build (the second in the script) seems to be the culprit:

docker build $DOCKER_ADDITIONAL_BUILD_ARGS --build-arg GALAXY_REPO=$GALAXY_REPO --build-arg GALAXY_RELEASE=$GALAXY_RELEASE -t quay.io/bgruening/galaxy-init$TAG ./galaxy-init/

In the Dockerfile in galaxy-init I tracked it down to

RUN ansible-playbook /ansible/provision.yml \
    --extra-vars galaxy_server_dir=$GALAXY_ROOT \
    --extra-vars galaxy_venv_dir=$GALAXY_VIRTUAL_ENV \
    --extra-vars galaxy_log_dir=$GALAXY_LOGS_DIR \
    --extra-vars galaxy_user_name=$GALAXY_USER \
    --extra-vars galaxy_config_file=$GALAXY_CONFIG_FILE \
    --extra-vars galaxy_extras_config_condor=True \
    --extra-vars galaxy_extras_config_condor_docker=True \
    --extra-vars galaxy_extras_config_k8s_jobs=True \
    --extra-vars galaxy_minimum_version=17.09 \
    --extra-vars galaxy_extras_config_rabbitmq=False \
    --extra-vars nginx_upload_store_path=/export/nginx_upload_store \
    --extra-vars nginx_welcome_location=$NGINX_WELCOME_LOCATION \
    --extra-vars nginx_welcome_path=$NGINX_WELCOME_PATH \
    #--extra-vars galaxy_extras_config_container_resolution=True \
    #--extra-vars container_resolution_explicit=True \
    #--extra-vars container_resolution_cached_mulled=False \
    #--extra-vars container_resolution_build_mulled=False \
    --tags=ie,pbs,slurm,uwsgi,metrics,k8s -c local

It fails with a non-zero code 2. No further information is given.

Just before the fail this bit also seems to produce an error

ASK [galaxyprojectdotorg.galaxyextras : Fetch DRMAA egg for Galaxy] ***********
fatal: [localhost]: FAILED! => {"changed": true, "cmd": "/export/venv/bin/python /export/galaxy-central/scripts/fetch_eggs.py -e drmaa -c /etc/galaxy/galaxy.yml", "delta": "0:00:00.113504", "end": "2018-09-28 22:42:34.058082", "msg": "non-zero return code", "rc": 1, "start": "2018-09-28 22:42:33.944578", "stderr": "", "stderr_lines": [], "stdout": "\nEggs in this release of Galaxy have been replaced by Python's newer packaging\nformat, wheels. Please use scripts/common_startup.sh to set up your\nenvironment:\n\ncd /export/galaxy-central && ./scripts/common_startup.sh\n\nThis will create a Python virtualenv and install Galaxy's dependencies into it.\n\nIf you start Galaxy using means other than run.sh (as you probably do if you\nare seeing this message), be sure to activate the virtualenv before starting,\nusing:\n\n. /export/galaxy-central/.venv/bin/activate\n\nIf you already run Galaxy in its own virtualenv, you can reuse your existing\nvirtualenv with:\n\ncd /export/galaxy-central && ./scripts/common_startup.sh --skip-venv", "stdout_lines": ["", "Eggs in this release of Galaxy have been replaced by Python's newer packaging", "format, wheels. Please use scripts/common_startup.sh to set up your", "environment:", "", "cd /export/galaxy-central && ./scripts/common_startup.sh", "", "This will create a Python virtualenv and install Galaxy's dependencies into it.", "", "If you start Galaxy using means other than run.sh (as you probably do if you", "are seeing this message), be sure to activate the virtualenv before starting,", "using:", "", ". /export/galaxy-central/.venv/bin/activate", "", "If you already run Galaxy in its own virtualenv, you can reuse your existing", "virtualenv with:", "", "cd /export/galaxy-central && ./scripts/common_startup.sh --skip-venv"]}
    to retry, use: --limit @/ansible/provision.retry

It seems to be an egg vs wheels problem, but my Python snake charming skills are definitly out of their depth.

Any advice or help is much appreciated.

FelixErnst commented 6 years ago

On the current dev 7b88245 it fails shortly after. In 7b88245 eggs vs. wheels triggers getting the wheel

TASK [galaxyprojectdotorg.galaxyextras : Check if the Ubuntu distro is supported] ***
fatal: [localhost]: FAILED! => {"changed": false, "connection": "close", "content": "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n</body>\r\n</html>\r\n", "content_length": "162", "content_type": "text/html", "date": "Fri, 28 Sep 2018 22:56:33 GMT", "msg": "Status code was 404 and not [200]: HTTP Error 404: Not Found", "redirected": false, "server": "nginx", "status": 404, "url": "http://deb.nodesource.com/node/dists/bionic/Release", "via": "1.1 0050c18ff4e0e872772088fcaf6414f4.cloudfront.net (CloudFront)", "x_amz_cf_id": "IGCSu5IwSJH8H-q4r3tc1dLGGhCZGQz-ODFHVNEa6_LcR2Z_jEwbDA==", "x_cache": "Error from cloudfront"}
    to retry, use: --limit @/ansible/provision.retry

PLAY RECAP *********************************************************************
localhost                  : ok=12   changed=9    unreachable=0    failed=1   

The command '/bin/sh -c ansible-playbook /ansible/provision.yml     --extra-vars galaxy_server_dir=$GALAXY_ROOT     --extra-vars galaxy_venv_dir=$GALAXY_VIRTUAL_ENV     --extra-vars galaxy_log_dir=$GALAXY_LOGS_DIR     --extra-vars galaxy_user_name=$GALAXY_USER     --extra-vars galaxy_config_file=$GALAXY_CONFIG_FILE     --extra-vars galaxy_extras_config_condor=True     --extra-vars galaxy_extras_config_condor_docker=True     --extra-vars galaxy_extras_config_k8s_jobs=True     --extra-vars galaxy_minimum_version=17.09     --extra-vars galaxy_extras_config_rabbitmq=False     --extra-vars nginx_upload_store_path=/export/nginx_upload_store     --extra-vars nginx_welcome_location=$NGINX_WELCOME_LOCATION     --extra-vars nginx_welcome_path=$NGINX_WELCOME_PATH     --tags=ie,pbs,slurm,uwsgi,metrics,k8s -c local &&     apt-get autoremove -y && apt-get clean && rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*' returned a non-zero code: 2

bgruening commented 6 years ago

I'm working currently on the dev branch, any help welcome.

Is your first error happening on the master branch?

You can also try to use the rebuild images on quay.io I think. I need to work on this next week for the next Galaxy release if you can wait a bit.

FelixErnst commented 6 years ago

Yes, the first error appears using 116c770

Sure thing. I am interested in it, since my own tries with galaxy (normal installation) worked well and I roughly got the idea how it works. Your docker implementation would make it more reproducible and a bit more automated, which is currently the goal. I am happy to test and try, but my knowledge if still quite basic regarding ansible things.

btw. saw an article in the Laborjournal 👍

pcm32 commented 6 years ago

I suggest that any changes in the build local script we do them here #446 as that is suppose to deprecate the current build local script.

FelixErnst commented 6 years ago

with omiting the galaxy-init build, the docker compose works. So its just this bit for the building the galaxy-init, which doesn't work.

bgruening commented 5 years ago

@FelixErnst is this still a problem with 18.09?

FelixErnst commented 5 years ago

I am currently not able to test it. Give me a week. Can/could you reproduce it?

FelixErnst commented 5 years ago

@bgruening Sorry for the late reply. The script works now.

bgruening commented 5 years ago

Cool thanks @FelixErnst!

bgruening / docker-galaxy

buildlocal.sh script fails #458