search

galaxyproject / ansible-galaxy-tools

An Ansible role for automated installation of tools from a Tool Shed into Galaxy.
MIT License
14 stars 34 forks source link

Empty error aborts role but installation continues #63

Closed maikknebel closed 5 years ago

maikknebel commented 5 years ago

When i try to install an additional tool the installation returns an error and aborts. The webinterface is listing my installation as running and after a few it has been finished successfully. I guess this is related to #15 and #https://github.com/galaxyproject/galaxy/issues/1628

equests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))"

playbook.yml

- hosts: galaxyservers
  become: true
  vars:
      galaxy_restart_handler_enabled: false
      galaxy_tools_base_dir: "{{ galaxy_root }}/var/tmp/"
      galaxy_server_dir: "{{ galaxy_root }}/server"
      galaxy_venv_dir: "{{ galaxy_root }}/venv"
      galaxy_config_file: "{{ galaxy_root }}/config/galaxy.yml"
      galaxy_tools_api_key: 11111111
      galaxy_tools_tool_list_files: [ "files/tools.yml" ]
      galaxy_tools_galaxy_instance_url: http://127.0.0.1:8080/
  roles:
    - galaxyproject.galaxy-tools

tools.yml

tools:
- name: ncbi_blast_plus
  owner: devteam
  tool_panel_section_id: BLAST
  tool_panel_section_label: BLAST

Error

TASK [galaxyproject.galaxy-tools : Install Tool Shed tools] 
failed: [172.18.1.32] (item={u'owner': u'devteam', u'tool_panel_section_id': u'BLAST', u'name': u'ncbi_blast_plus', u'tool_panel_section_label': u'BLAST'}) => {
    "changed": false,
    "cmd": [
        "/srv/galaxy/var/tmp//venv/bin/shed-install",
        "-y",
        "name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_id: BLAST\ntool_panel_section_label: BLAST\n",
        "-a",
        "111111111111111",
        "-g",
        "http://127.0.0.1:8080/"
    ],
    "delta": "0:01:01.338944",
    "end": "2019-05-28 09:31:49.159597",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "/srv/galaxy/var/tmp//venv/bin/shed-install -y \"name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_id: BLAST\ntool_panel_section_label: BLAST\n\" -a \"111111111111111\" -g \"http://127.0.0.1:8080/\"",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "name": "ncbi_blast_plus",
        "owner": "devteam",
        "tool_panel_section_id": "BLAST",
        "tool_panel_section_label": "BLAST"
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-05-28 09:30:47.820653",
    "stderr": "(1/1) Installing tool ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:00.893317)\r\nTraceback (most recent call last):\n  File \"/srv/galaxy/var/tmp//venv/bin/shed-install\", line 10, in <module>\n    sys.exit(script_main())\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 720, in script_main\n    itm.install_tools()\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 627, in install_tools\n    install_repository_revision(tool, self.tsc)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 502, in install_repository_revision\n    response = tsc.install_repository_revision(**tool)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision\n    return self._post(url=url, payload=payload)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 152, in _post\n    files_attached=files_attached)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request\n    timeout=self.timeout)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post\n    return request('post', url, data=data, json=json, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))",
    "stderr_lines": [
        "(1/1) Installing tool ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:00.893317)",
        "Traceback (most recent call last):",
        "  File \"/srv/galaxy/var/tmp//venv/bin/shed-install\", line 10, in <module>",
        "    sys.exit(script_main())",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 720, in script_main",
        "    itm.install_tools()",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 627, in install_tools",
        "    install_repository_revision(tool, self.tsc)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_install.py\", line 502, in install_repository_revision",
        "    response = tsc.install_repository_revision(**tool)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision",
        "    return self._post(url=url, payload=payload)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 152, in _post",
        "    files_attached=files_attached)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request",
        "    timeout=self.timeout)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post",
        "    return request('post', url, data=data, json=json, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request",
        "    return session.request(method=method, url=url, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request",
        "    resp = self.send(prep, **send_kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send",
        "    r = adapter.send(request, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send",
        "    raise ConnectionError(err, request=request)",
        "requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))"
    ],
    "stdout": "",
    "stdout_lines": []
}
...ignoring
bernt-matthias commented 5 years ago

I observe the same here. When installing via the UI I often observe timeouts from the proxy. We are currently planning to increase the timeout.

maikknebel commented 5 years ago

After a few more tries... the error appears exactly after 60 seconds in the given task-step. The error message above shows that as well. 

"start": "2019-05-28 09:30:47.820653"
"end": "2019-05-28 09:31:49.159597",

The gui installation runs fine for me - is requesting the status every few seconds from the api. Just the shed-install seems to get a timeout somewhere when using the ansible role.

Edit: The error also occurres from the terminal, when i manually activate the virtualenv and call shed-install directly.

mvdbeek commented 5 years ago

Try bumping up the ephemeris version. 0.9.0 should contain https://github.com/galaxyproject/ephemeris/commit/2499ba5b0e69fea466786ce8f86434d7db678448

bernt-matthias commented 5 years ago

@maikknebel can you try to change this here: https://github.com/galaxyproject/ansible-galaxy-tools/blob/935fc8b501d73827962df6220246b4fc6b9ba272/tasks/tools.yml#L6

bernt-matthias commented 5 years ago

Hmm. In my case its actually a 502 Proxy Error. 

The proxy server could not handle the request <em><a href=\"/api/tool_shed_repositories/new/install_repository_revision\">POST&nbsp;/api/tool_shed_repositories/new/install_repository_revision</a> 
...
Reason: <strong>Error reading from remote server

Since the runtime in 0:01:01.485219 I guess this is apache's way to signal a timeout..? But might also be a misconfiguration of the Apache.

maikknebel commented 5 years ago

I tried version 0.8.0 and 0.9.0 both give me the following error:

fatal: [172.18.1.32]: FAILED! => {
    "msg": "The conditional check ''installed successfully' in install_result.stderr' failed. The error was: error while evaluating conditional ('installed successfully' in install_result.stderr): Unable to look up a name or access an attribute in template string ({% if 'installed successfully' in install_result.stderr %} True {% else %} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'StrictUndefined' is not iterable"
}

Coming from this block in tasks/instal_tool_lists.yml

- name: Install Tool Shed tools
  command: '{{ galaxy_tools_base_dir }}/venv/bin/shed-install -y "{{ item | to_nice_yaml }}" -a "{{ galaxy_tools_api_key
 }}" -g "{{ galaxy_tools_galaxy_instance_url }}"'
  register: install_result
  changed_when: "'installed successfully' in install_result.stderr"
  failed_when: ('Error installing' in install_result.stderr) or ('Missing required' in install_result.stderr) or (instal
l_result.rc != 0)
  ignore_errors: "{{ galaxy_tools_ignore_errors }}"
  notify:
    - restart galaxy
  with_items:
    - "{{ tools }}"
maikknebel commented 5 years ago

I had to replace the command, since it has changed from 0.4.0 to 0.8.0/0.9.0

command: '{{ galaxy_tools_base_dir }}/venv/bin/shed-install -y "{{ item | to_nice_yaml }}" -a "{{ galaxy_tools_api_key }}" -g "{{ galaxy_tools_galaxy_instance_url }}"'

with

command: '{{ galaxy_tools_base_dir }}/venv/bin/shed-tools install -y "{{ item | to_nice_yaml }}" -a "{{ galaxy_tools_api_key }}" -g "{{ galaxy_tools_galaxy_instance_url }}"'

The error stays the same. The role terminates after 60 seconds but the installation continiues.

mvdbeek commented 5 years ago

Is it really the exact same traceback ? This

requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))

should really not appear anymore in 0.9.0

maikknebel commented 5 years ago

Most likely, some of the scripts in the trace have changed but i guess thats ok due to the changes in ephemeris. The error message and the last calls in the trace are the same.

failed: [172.18.1.32] (item={u'owner': u'devteam', u'tool_panel_section_label': u'BLAST', u'name': u'ncbi_blast_plus'}) => {
    "changed": false,
    "cmd": [
        "/srv/galaxy/var/tmp//venv/bin/shed-tools",
        "install",
        "-y",
        "name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n",
        "-a",
        "1111111",
        "-g",
        "http://127.0.0.1:8080/"
    ],
    "delta": "0:01:01.789697",
    "end": "2019-05-28 14:33:59.683804",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "/srv/galaxy/var/tmp//venv/bin/shed-tools install -y \"name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n\" -a \"11111111\" -g \"http://127.0.0.1:8080/\"",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "name": "ncbi_blast_plus",
        "owner": "devteam",
        "tool_panel_section_label": "BLAST"
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-05-28 14:32:57.894107",
    "stderr": "Storing log file in: /tmp/ephemeris_tniPFt\r\n(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:00.669282)\r\nTraceback (most recent call last):\n  File \"/srv/galaxy/var/tmp//venv/bin/shed-tools\", line 10, in <module>\n    sys.exit(main())\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 937, in main\n    install_tool_manager.install_repositories()\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 716, in install_repositories\n    install_repository_revision(repository, self.tsc)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 517, in install_repository_revision\n    response = tool_shed_client.install_repository_revision(**repository)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision\n    return self._post(url=url, payload=payload)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 152, in _post\n    files_attached=files_attached)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request\n    timeout=self.timeout)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post\n    return request('post', url, data=data, json=json, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))",
    "stderr_lines": [
        "Storing log file in: /tmp/ephemeris_tniPFt",
        "(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:00.669282)",
        "Traceback (most recent call last):",
        "  File \"/srv/galaxy/var/tmp//venv/bin/shed-tools\", line 10, in <module>",
        "    sys.exit(main())",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 937, in main",
        "    install_tool_manager.install_repositories()",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 716, in install_repositories",
        "    install_repository_revision(repository, self.tsc)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 517, in install_repository_revision",
        "    response = tool_shed_client.install_repository_revision(**repository)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision",
        "    return self._post(url=url, payload=payload)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 152, in _post",
        "    files_attached=files_attached)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request",
        "    timeout=self.timeout)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post",
        "    return request('post', url, data=data, json=json, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request",
        "    return session.request(method=method, url=url, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request",
        "    resp = self.send(prep, **send_kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send",
        "    r = adapter.send(request, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send",
        "    raise ConnectionError(err, request=request)",
        "requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))"
    ],
    "stdout": "",
    "stdout_lines": []
}
...ignoring
mvdbeek commented 5 years ago

https://github.com/galaxyproject/ephemeris/compare/0.9.0...master ... we released 0.9.0 not so long ago, but we need another release to get that commit ...

mvdbeek commented 5 years ago

There's a new package on pypi now, 0.10.0, that should hopefully work

maikknebel commented 5 years ago

With 0.10.0 its working. Thank you.

Should the playbook in ansible galaxy be updated or the admintraining to directly pull the playbook from git? To reflect the version change.

bernt-matthias commented 5 years ago

I guess so. Maybe just open a PR?

maikknebel commented 5 years ago

I just made a clean install to test everything again... not working :( After 60 seconds in TASK [galaxyproject.galaxy-tools : Install Tool Shed tools], the script fails with the BadStatusLine-Error. My guess is, that it has worked before, because conda has cached the dependency package in the previous tries.

While this error appears, htop shows the following line that with conda installing the dependencies of the package. In the admin interface it says "installing dependencies". The Installation Monitor says, dependency not installed and no error. The dependency overview says that blast is installed. After a few minutes, the conda-task is gone from htop and nothing changed. After a restart of galaxy, the conda task appears again with the same command but taking far longer (10-15 min) than before. Another conda task appeared after that is still running... not sure if thats ok but nothing changed in the installation status. There have been no errors in the galaxy console so far.

Update: After 25 minutes, the second conda task finished and "manage dependencies" lists blast with a red cross, dependency not installed. Manage tools says installed for the first, and installed but missing dependencies for the second and third.

2019-06-04 07_52_01-Galaxy Installed tool shed repositories 2019-06-04 08_11_12-Galaxy _ German Seed Alliance _ Installed tool shed repositories 2019-06-04 07_52_24-Galaxy Installed tool shed repositories

First conda task

12398 galaxy     20   0  613M  505M 12500 D 17.2  8.5  0:58.31 /srv/galaxy/var/dependencies/_conda/bin/python /srv/galaxy/var/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __blast@2.7.1 blast=2.7.1

Second conda task

12746 galaxy     21   1  577M  469M 12428 R  9.0  7.9  0:43.84 /srv/galaxy/var/dependencies/_conda/bin/python /srv/galaxy/var/dependencies/_conda/bin/conda create -y --quiet --override-channels --channel iuc --channel conda-forge --channel bioconda --channel defaults --name __blast@_uv_ blast
TASK [galaxyproject.galaxy-tools : Install Tool Shed tools]
failed: [172.18.1.32] (item={u'owner': u'devteam', u'tool_panel_section_label': u'BLAST', u'name': u'ncbi_blast_plus'}) => {
    "changed": false,
    "cmd": [
        "/srv/galaxy/var/tmp//venv/bin/shed-tools",
        "install",
        "-y",
        "name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n",
        "-a",
        "1111111111111111",
        "-g",
        "http://127.0.0.1:8080/"
    ],
    "delta": "0:01:02.685348",
    "end": "2019-06-04 05:45:55.272057",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "/srv/galaxy/var/tmp//venv/bin/shed-tools install -y \"name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n\" -a \"11111111111111111111111\" -g \"http://127.0.0.1:8080/\"",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "name": "ncbi_blast_plus",
        "owner": "devteam",
        "tool_panel_section_label": "BLAST"
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-06-04 05:44:52.586709",
    "stderr": "Storing log file in: /tmp/ephemeris_zAoFxM\r\n(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:01.542898)\r\nTraceback (most recent call last):\n  File \"/srv/galaxy/var/tmp//venv/bin/shed-tools\", line 10, in <module>\n    sys.exit(main())\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 546, in main\n    **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 160, in install_repositories\n    result = self.install_repository_revision(repository, log)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 355, in install_repository_revision\n    response = self.tool_shed_client.install_repository_revision(**repository)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision\n    return self._post(url=url, payload=payload)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 160, in _post\n    files_attached=files_attached)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request\n    timeout=self.timeout)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post\n    return request('post', url, data=data, json=json, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request\n    return session.request(method=method, url=url, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request\n    resp = self.send(prep, **send_kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send\n    r = adapter.send(request, **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send\n    raise ConnectionError(err, request=request)\nrequests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))",
    "stderr_lines": [
        "Storing log file in: /tmp/ephemeris_zAoFxM",
        "(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:01.542898)",
        "Traceback (most recent call last):",
        "  File \"/srv/galaxy/var/tmp//venv/bin/shed-tools\", line 10, in <module>",
        "    sys.exit(main())",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 546, in main",
        "    **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 160, in install_repositories",
        "    result = self.install_repository_revision(repository, log)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 355, in install_repository_revision",
        "    response = self.tool_shed_client.install_repository_revision(**repository)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/toolshed/__init__.py\", line 146, in install_repository_revision",
        "    return self._post(url=url, payload=payload)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxy/client.py\", line 160, in _post",
        "    files_attached=files_attached)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/bioblend/galaxyclient.py\", line 137, in make_post_request",
        "    timeout=self.timeout)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 116, in post",
        "    return request('post', url, data=data, json=json, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/api.py\", line 60, in request",
        "    return session.request(method=method, url=url, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 533, in request",
        "    resp = self.send(prep, **send_kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/sessions.py\", line 646, in send",
        "    r = adapter.send(request, **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/requests/adapters.py\", line 498, in send",
        "    raise ConnectionError(err, request=request)",
        "requests.exceptions.ConnectionError: ('Connection aborted.', BadStatusLine(\"''\",))"
    ],
    "stdout": "",
    "stdout_lines": []
}
...ignoring
mvdbeek commented 5 years ago

This is very strange, the line numbers in the traceback match 0.10.0, but the exception is explicitly wrapped in 

try:
    response = self.tool_shed_client.install_repository_revision(**repository)
    ...
except ConnectionError as e:
    ...
    elif "504" in str(e) or 'Connection aborted' in str(e):
        success = self.wait_for_install(repository=repository, log=log, timeout=3600)

that condition really should evaluate to true in your traceback.

Can you print out the ephemeris version before the install and provide an explicit log file with --log_file ? (/dev/stdout should work to get the log reported in ansible)

mvdbeek commented 5 years ago

ahhhhh! Wrong ConnectionError, it's the vanilla requests.exceptions.ConnectionError that bubbles up.

>>> import requests
>>> from bioblend.galaxy.client import ConnectionError
>>>
>>> try:
...     raise requests.exceptions.ConnectionError('bla')
... except ConnectionError as e:
...     print(str(e))
...
Traceback (most recent call last):
  File "<stdin>", line 2, in <module>
requests.exceptions.ConnectionError: bla
maikknebel commented 5 years ago

This is printed by the playbook when the virtualenv is setup for the installation. Does that help? If not i will try to figure out how i can print the version of the packages in the virtualenv that is created/used by the playbook.

 "Installing collected packages: six, urllib3, certifi, chardet, idna, requests, requests-toolbelt, PyYAML, boto, bioblend, futures, MarkupSafe, Jinja2, boltons, docutils, pyparsing, packaging, galaxy-lib, ephemeris",
        "Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 PyYAML-5.1 bioblend-0.10.0 boltons-19.1.0 boto-2.49.0 certifi-2019.3.9 chardet-3.0.4 docutils-0.14 ephemeris-0.10.0 futures-3.2.0 galaxy-lib-19.5.2 idna-2.8 packaging-19.0 pyparsing-2.4.0 requests-2.22.0 requests-toolbelt-0.9.1 six-1.12.0 urllib3-1.25.3"
    ],
    "version": null,
    "virtualenv": "/srv/galaxy/var/tmp//venv"

I pulled the role from git and changed the emphemeris version from 0.4.0 to 0.10.0 manually. Should i change the bioblend versions to the current release as well? Bioblend doesnt list19.01 as supported version yet.

Logfile shows only this line:

Storing log file in: /srv/galaxy/var/tmp//install.log
(1/1) Installing repository ncbi_blast_plus from devteam to section "BLAST" at revision e25d3acf6e68 (TRT: 0:00:00.974527)
mvdbeek commented 5 years ago

No that's not necessary, https://github.com/galaxyproject/ephemeris/pull/134 will fix it. fwiw, bioblend always supports all versions of Galaxy, it uses the API and the API should never change. It's just that some functions or parameters may be missing.

mvdbeek commented 5 years ago

@maikknebel 0.10.1 is out that should work now.

maikknebel commented 5 years ago

The error has changed now, but is still there.

Installed versions in the virtualenv:

 "Successfully installed Jinja2-2.10.1 MarkupSafe-1.1.1 PyYAML-5.1 bioblend-0.10.0 boltons-19.1.0 boto-2.49.0 certifi-2019.3.9 chardet-3.0.4 docutils-0.14 ephemeris-0.10.1 futures-3.2.0 galaxy-lib-19.5.2 idna-2.8 packaging-19.0 pyparsing-2.4.0 requests-2.22.0 requests-toolbelt-0.9.1 six-1.12.0 urllib3-1.25.3"
    ],
    "version": null,
    "virtualenv": "/srv/galaxy/var/tmp/venv"

Error-Message

failed: [172.18.1.32] (item={u'owner': u'devteam', u'tool_panel_section_label': u'BLAST', u'name': u'ncbi_blast_plus'}) => {
    "changed": false,
    "cmd": [
        "/srv/galaxy/var/tmp/venv/bin/shed-tools",
        "install",
        "-y",
        "name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n",
        "-a",
        "11111111",
        "-g",
        "http://127.0.0.1:8080/"
    ],
    "delta": "0:01:02.507261",
    "end": "2019-06-04 10:58:06.309762",
    "failed_when_result": true,
    "invocation": {
        "module_args": {
            "_raw_params": "/srv/galaxy/var/tmp/venv/bin/shed-tools install -y \"name: ncbi_blast_plus\nowner: devteam\ntool_panel_section_label: BLAST\n\" -a \"11111111\" -g \"http://127.0.0.1:8080/\"",
            "_uses_shell": false,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "warn": true
        }
    },
    "item": {
        "name": "ncbi_blast_plus",
        "owner": "devteam",
        "tool_panel_section_label": "BLAST"
    },
    "msg": "non-zero return code",
    "rc": 1,
    "start": "2019-06-04 10:57:03.802501",
    "stderr": "Storing log file in: /tmp/ephemeris_dwhewP\r\n(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:01.304021)\r\nTraceback (most recent call last):\n  File \"/srv/galaxy/var/tmp/venv/bin/shed-tools\", line 10, in <module>\n    sys.exit(main())\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 547, in main\n    **kwargs)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 161, in install_repositories\n    result = self.install_repository_revision(repository, log)\n  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 373, in install_repository_revision\n    if default_err_msg in e.body:\nAttributeError: 'ConnectionError' object has no attribute 'body'",
    "stderr_lines": [
        "Storing log file in: /tmp/ephemeris_dwhewP",
        "(1/1) Installing repository ncbi_blast_plus from devteam to section \"BLAST\" at revision e25d3acf6e68 (TRT: 0:00:01.304021)",
        "Traceback (most recent call last):",
        "  File \"/srv/galaxy/var/tmp/venv/bin/shed-tools\", line 10, in <module>",
        "    sys.exit(main())",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 547, in main",
        "    **kwargs)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 161, in install_repositories",
        "    result = self.install_repository_revision(repository, log)",
        "  File \"/srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py\", line 373, in install_repository_revision",
        "    if default_err_msg in e.body:",
        "AttributeError: 'ConnectionError' object has no attribute 'body'"
    ],
    "stdout": "",
    "stdout_lines": []
}
...ignoring
mvdbeek commented 5 years ago

arg ... could you just manually change /srv/galaxy/var/tmp/venv/local/lib/python2.7/site-packages/ephemeris/shed_tools.py on line 373 from if default_err_msg in e.body to if default_err_msg in str(e): and check that it work ?

maikknebel commented 5 years ago

That worked so far. I'm going to prepare a clean installation to check if thats working too, might take a bit, i will post the result.

mvdbeek commented 5 years ago

Alright, if that works I'll push out a new version.

maikknebel commented 5 years ago

Its working, thanks.

mvdbeek commented 5 years ago

0.10.2 is out, so I guess we can close this ?

bernt-matthias commented 5 years ago

Wondering if it would be an option to check for error 502 as well. This seems to be the error raised for time outs if apache is used as proxy.

mvdbeek commented 5 years ago

Feel free to add it to https://github.com/galaxyproject/ephemeris/blob/master/src/ephemeris/shed_tools.py#L379 -- we actually don't care about this too much, we should probably just start the install in a thread, we don't care about the return value.