K8S Runner/Tools errors

Just documenting some issues discussed with Nuwan today and possible solutions:

1) Job pod error, when trying to delete input files: galaxy.objectstore CRITICAL 2019-06-02 11:00:29,685 None delete error [Errno 13] Permission denied: '/galaxy/server/database/jobs_directory/000/4/working/input_f.fastq.gz’ xref: https://github.com/crs4/container-galaxy-elixir-is-varcall/blob/master/docs/making_galaxy_and_tool_containers_share_data.md

2) Tools that require galaxy code in path: eg: Sort tool: python: can't open file '/galaxy/server/tools/filters/sorter.py': [Errno 2] No such file or directory List of these tools: https://github.com/galaxyproject/galaxy/blob/dev/lib/galaxy/tools/__init__.py#L114

Solutions discussed: -Some tools could be forced to use local runner and this would not be a problem, but not all of them seem to be simple tools so might not be ideal situations -Using an init container with rsync ensuring syncing whenever a new pod comes up but abandoned for the idea below -Using a helm post-install hook (https://github.com/technosophos/k8s-helm/blob/master/docs/charts_hooks.md) to copy all galaxy files from container to a shared volume. One option is to share the same PVC/PV (probably best option for beginning), move /galaxy/server/database to /export/database and then copy /galaxy/server/ to /export/galaxy/ and mount the /galaxy subPath only in job containers:

Some possible things to consider: will the codebase ever change without a new helm install (for example, will installing a new tool add dependencies/tools to that code), and if so, are there any tools that can be installed later that depend on the codebase? (probably won't be a problem but putting it out there)

3) Uploading a fasta file currently throws a pop-up error: It seems that the upload works, and the error is in the web pod:

[pid: 75|app: 0|req: 153/226] 10.0.14.231 () {54 vars in 1843 bytes} [Mon Jun 3 21:19:15 2019] GET /galaxy/galaxy/api/histories/f597429621d6eb2b/contents?limit=500&offset=0&details=f09437b8822035f7%2Cfb85969571388350&order=hid&v=dev&q=deleted&q=purged&q=visible&qv=False&qv=False&qv=True => generated 214 bytes in 1 msecs (HTTP/1.1 404) 2 headers in 87 bytes (1 switches on core 2)
galaxy.web.framework.decorators ERROR 2019-06-03 21:19:22,476 [p:75,w:1,m:0] [uWSGIWorker1Core3] Uncaught exception in exposed API method:
Traceback (most recent call last):
File "/galaxy/server/lib/galaxy/web/framework/decorators.py", line 282, in decorator
rval = func(self, trans, *args, **kwargs)
File "/galaxy/server/lib/galaxy/webapps/galaxy/api/history_contents.py", line 82, in index
return self.__index_v2(trans, history_id, **kwd)
File "/galaxy/server/lib/galaxy/webapps/galaxy/api/history_contents.py", line 891, in __index_v2
user=trans.user, trans=trans, view='detailed', **serialization_params))
File "/galaxy/server/lib/galaxy/managers/base.py", line 692, in serialize_to_view
return self.serialize(item, all_keys, **context)
File "/galaxy/server/lib/galaxy/managers/hdas.py", line 369, in serialize
return super(HDASerializer, self).serialize(hda, keys, user=user, **context)
File "/galaxy/server/lib/galaxy/managers/datasets.py", line 604, in serialize
serialized = super(DatasetAssociationSerializer, self).serialize(dataset_assoc, keys, **context)
File "/galaxy/server/lib/galaxy/managers/base.py", line 605, in serialize
returned[key] = self.serializers[key](item, key, **context)
File "/galaxy/server/lib/galaxy/managers/hdas.py", line 376, in serialize_display_apps
for display_app in hda.get_display_applications(trans).values():
File "/galaxy/server/lib/galaxy/model/__init__.py", line 2756, in get_display_applications
return self.datatype.get_display_applications_by_dataset(self, trans)
File "/galaxy/server/lib/galaxy/datatypes/data.py", line 551, in get_display_applications_by_dataset
value = value.filter_by_dataset(dataset, trans)
File "/galaxy/server/lib/galaxy/datatypes/display_applications/application.py", line 309, in filter_by_dataset
if link_value.filter_by_dataset(data, trans):
File "/galaxy/server/lib/galaxy/datatypes/display_applications/application.py", line 95, in filter_by_dataset
if fill_template(filter_elem.text, context=context) != filter_elem.get('value', 'True'):
File "/galaxy/server/lib/galaxy/util/template.py", line 91, in fill_template
raise first_exception or e
NotFound: cannot find 'site_id'

The following pop-up shows up in the UI:

An error occurred
An error occurred while updating information with the server. Please contact a Galaxy administrator if the problem persists.

The following information can assist the developers in finding the source of the error:

{
  "userAgent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36",
  "onLine": true,
  "version": "19.05",
  "xhr": {
    "readyState": 4,
    "responseText": "{\"err_msg\": \"Uncaught exception in exposed API method:\", \"err_code\": 0}",
    "responseJSON": {
      "err_msg": "Uncaught exception in exposed API method:",
      "err_code": 0
    },
    "status": 500,
    "statusText": "error"
  },
  "options": {
    "parse": true,
    "silent": true,
    "emulateHTTP": false,
    "emulateJSON": false,
    "textStatus": "error",
    "errorThrown": ""
  },
  "url": "https://3.210.77.230/galax/galaxy/api/histories/f597429621d6eb2b/contents/datasets/fb85969571388350",
  "model": {
    "state": "ok",
    "deleted": false,
    "purged": false,
    "name": "Pasted Entry",
    "accessible": true,
    "data_type": "",
    "file_ext": "",
    "file_size": 0,
    "meta_files": [],
    "misc_blurb": "",
    "misc_info": "",
    "tags": [],
    "history_id": "f597429621d6eb2b",
    "history_content_type": "dataset",
    "hid": 3,
    "visible": true,
    "model_class": "HistoryDatasetAssociation",
    "update_time": "2019-06-04T01:14:11.038Z",
    "extension": "fasta",
    "type_id": "dataset-fb85969571388350",
    "id": "fb85969571388350",
    "create_time": "2019-06-04T01:14:05.574Z",
    "url": "/galaxy/api/histories/f597429621d6eb2b/contents/fb85969571388350",
    "dataset_id": "fb85969571388350",
    "type": "file",
    "urls": {
      "purge": "/galaxy/datasets/fb85969571388350/purge_async",
      "display": "/galaxy/datasets/fb85969571388350/display/?preview=True",
      "edit": "/galaxy/datasets/edit?dataset_id=fb85969571388350",
      "download": "/galaxy/datasets/fb85969571388350/display?to_ext=",
      "report_error": "/galaxy/dataset/errors?id=fb85969571388350",
      "rerun": "/galaxy/tool_runner/rerun?id=fb85969571388350",
      "show_params": "/galaxy/datasets/fb85969571388350/show_params",
      "visualization": "/galaxy/visualization",
      "meta_download": "/galaxy/dataset/get_metadata_file?hda_id=fb85969571388350&metadata_name="
    }
  },
  "user": {
    "id": null,
    "username": "(anonymous user)",
    "total_disk_usage": 25,
    "nice_total_disk_usage": "25 bytes",
    "quota_percent": null,
    "is_admin": false,
    "preferences": {}
  }
}

CloudVE / galaxy-helm

K8S Runner/Tools errors #20