shed-tool install issue with "changeset_revision"

[hexylena]

I'm experiencing some issues installing tools. This had worked $some_time_ago, but now the installation log is completely full of errors like:

    * Error installing a repository (after 0:00:00.932244 seconds)! Name: bedtools,owner: iuc, revision: ['b8348686a0b9'], error: {"err_msg": "No information is available for the requested repository revision.\nOne or more of the following parameter values is likely invalid:\ntool_shed_url: https://toolshed.g2.bx.psu.edu/\nname: bedtools\nowner: iuc\nchangeset_revision: [u'b8348686a0b9']\n", "err_code": 400008}

the last time I saw this it was because changeset_revision was being interpreted as a list and then cast to a string, and then ephemeris was searching for that and failing to find it (because of course there is no revision matching the string u"['asdfasdfa']".

Any ideas on this?

Here is the full console log https://build.usegalaxy.eu/job/usegalaxy.eu/job/install-tools/12/consoleFull

It runs the command make install from our repo which runs the command shed-tools install --toolsfile $file.yaml --galaxy $url --api_key $api_key.

Running locally with increased logging level:

Installing any updated versions of tools_iuc.yaml.lock
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): galaxy.uni-freiburg.de
DEBUG:urllib3.connectionpool:https://galaxy.uni-freiburg.de:443 "GET /api/tool_shed_repositories?key=<redacted> HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): toolshed.g2.bx.psu.edu
DEBUG:urllib3.connectionpool:https://toolshed.g2.bx.psu.edu:443 "GET /api/repositories/get_ordered_installable_revisions?owner=iuc&name=bedtools HTTP/1.1" 200 None
(1/502) Installing repository bedtools from iuc to section "Operate on Genomic Intervals" at revision ['b8348686a0b9'] (TRT: 0:00:06.692430)
DEBUG:ephemeris.shed_tools:(1/502) Installing repository bedtools from iuc to section "Operate on Genomic Intervals" at revision ['b8348686a0b9'] (TRT: 0:00:06.692430)
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): galaxy.uni-freiburg.de
DEBUG:urllib3.connectionpool:https://galaxy.uni-freiburg.de:443 "POST /api/tool_shed_repositories/new/install_repository_revision?key=<redacted> HTTP/1.1" 400 None
Traceback (most recent call last):
  File "/home/hxr/.local/lib/python3.5/site-packages/ephemeris/shed_tools.py", line 609, in install_repositories
    install_repository_revision(repository, self.tsc)
  File "/home/hxr/.local/lib/python3.5/site-packages/ephemeris/shed_tools.py", line 428, in install_repository_revision
    response = tool_shed_client.install_repository_revision(**repository)
  File "/home/hxr/.local/lib/python3.5/site-packages/bioblend/galaxy/toolshed/__init__.py", line 146, in install_repository_revision
    return self._post(url=url, payload=payload)
  File "/home/hxr/.local/lib/python3.5/site-packages/bioblend/galaxy/client.py", line 160, in _post
    files_attached=files_attached)
  File "/home/hxr/.local/lib/python3.5/site-packages/bioblend/galaxyclient.py", line 146, in make_post_request
    body=r.text, status_code=r.status_code)
bioblend.ConnectionError: Unexpected HTTP status code: 400: {"err_msg": "No information is available for the requested repository revision.\nOne or more of the following parameter values is likely invalid:\ntool_shed_url: https://toolshed.g2.bx.psu.edu/\nname: bedtools\nowner: iuc\nchangeset_revision: [u'b8348686a0b9']\n", "err_code": 400008}

This sticks out to me:

DEBUG:ephemeris.shed_tools:(1/502) Installing repository bedtools from iuc to section "Operate on Genomic Intervals" at revision ['b8348686a0b9'] (TRT: 0:00:06.692430)

I really expected to see something like

DEBUG:ephemeris.shed_tools:(1/502) Installing repository bedtools from iuc to section "Operate on Genomic Intervals" at revision 'b8348686a0b9' (TRT: 0:00:06.692430)

Changing the changeset_revision to revisions in our yaml files looks like it fixes it, but it used to work. Did something change recently in the handling of changeset_revision to force it into a list?

[mvdbeek]

Hey @erasche, it looks like we've always had this as revisions, at least as far back as https://github.com/galaxyproject/ansible-galaxy-tools/commit/bfdd53c19e3bb17669bb2b3c9409cf6fe59b7e26. The helper scripts produce this under the revisions key as well. Do you think we should add a changeset_revisions alias for this ? I think I've replied on the mailing list a few times about this, so it seems this comes naturally (especially since the galaxy API/UI calls this changeset_revision)

[hexylena]

I feel like I'm going crazy when you point that out from 2015. I know I copied changeset_revision from somewhere, but I don't see it now. No, there's no need to support this, maybe I got confused from the use of changeset_revisions in so many other places.

[hexylena]

Ok, no, it's especially odd that ephemeris interprets changeset_revision as a key which stores a revision. Thinking about it, I would expect an invalid key to be ignored, but instead this invalid key is parsed (incorrectly) into trying to be a revisions that should be installed. I'd argue that that specific behaviour is unexpected.

[hexylena]

It looks like the flattening function is the culprit:

• it runs _strip_revisions which doesn't strip the conflicting key changeset_revisions
• changeset_revisions is then used internally in a "flat" version of the repo
• since there aren't a list of revisions provided in revisions, we hit the else, and the repo is appended as-is, and changeset_revisions is set to an illegal value because it was never checked.

Due to a 'copy' of the dict, rather than creating a new one with only selected / validated properties passed through.

[mvdbeek]

Yep, that's what I figured as well. Why don't we strip all invalid keys and log them as a warning ?

[hexylena]

Sounds good to me!

[rhpvorderman]

Hey @erasche . Quite some changes have been made to shed-tools with #104 . Does this issue still persist? If so, I can implement @mvdbeek's suggestion.

[hexylena]

@rhpvorderman the issue would still persist with the refactoring I'm afraid. Please either strip invalid keys or silently drop them, that would be very helpful since this is unvalidated input.

Hmm, we could add a .yaml linter with pykwalify at some point. That might be nice to have.