search

phnmnl / container-galaxy-k8s-runtime

PhenoMeNal runtime for Galaxy running inside a container orchestrator
Apache License 2.0
5 stars 18 forks source link

Failure when setting defining a dynamic destination for biosigner tool #165

Closed pkrog closed 6 years ago

pkrog commented 6 years ago

When setting destination as:

    <tool id="Biosigner" destination="dynamic-k8s-large" resources="all"/>

The container is not found at run time and we get the following message:

Checking with container resolver [ExplicitContainerResolver[]] found description [None]

We have to define old fashioned destination for biosigner:

      <destination id="biosigner-container" runner="k8s">
      <param id="docker_repo_override">container-registry.phenomenal-h2020.eu</param>
      <param id="docker_owner_override">phnmnl</param>
      <param id="docker_image_override">biosigner</param>
      <param id="docker_tag_override">v2.2.7_cv1.3.15</param>
      <param id="max_pod_retrials">3</param>
      <param id="docker_enabled">true</param>
    </destination>
pkrog commented 6 years ago

When running another container (Univariate), which as a dynamic destination defined, we get the following debug messages from within ContainerFinder::find_container() (file lib/galaxy/tools/deps/containers.py):

galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,985 ContainerFinder::find_container 01
galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,986 ContainerFinder::find_container 02
galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,986 ContainerFinder::find_container 03
galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,986 ContainerFinder::find_container 04
galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,986 Container classes {'singularity': <class 'galaxy.tools.deps.containers.SingularityContainer'>, 'docker': <class 'galaxy.tools.deps.containers.DockerContainer'>}
galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,986 Container type singularity
galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,986 Container ID None
galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,986 Container type docker
galaxy.tools.deps.containers INFO 2018-03-07 14:53:39,986 Container ID container-registry.phenomenal-h2020.eu/phnmnl/univariate:v2.2.3_cv1.3.29
galaxy.jobs.command_factory INFO 2018-03-07 14:53:40,008 Built script [/opt/galaxy_data/database/job_working_directory/000/4/tool_script.sh] for tool command [univariate_wrapper.R  dataMatrix_in "/opt/galaxy_data/files/000/dataset_1.dat" sampleMetadata_in "/opt/galaxy_data/files/000/dataset_2.dat" variableMetadata_in "/opt/galaxy_data/files/000/dataset_3.dat"  facC "gender" tesC "ttest" adjC "fdr" thrN "0.05"  variableMetadata_out "/opt/galaxy_data/files/000/dataset_4.dat" figure "/opt/galaxy_data/files/000/dataset_5.dat" information "/opt/galaxy_data/files/000/dataset_6.dat"]
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 ContainerFinder::find_container 01
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 ContainerFinder::find_container 02
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 ContainerFinder::find_container 03
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 ContainerFinder::find_container 04
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 Container classes {'singularity': <class 'galaxy.tools.deps.containers.SingularityContainer'>, 'docker': <class 'galaxy.tools.deps.containers.DockerContainer'>}
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 Container type singularity
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 Container ID None
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 Container type docker
galaxy.tools.deps.containers INFO 2018-03-07 14:53:40,173 Container ID container-registry.phenomenal-h2020.eu/phnmnl/univariate:v2.2.3_cv1.3.29

For biosigner we get:

galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,016 ContainerFinder::find_container 01
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,016 ContainerFinder::find_container 02
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,016 ContainerFinder::find_container 03
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 ContainerFinder::find_container 04
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 Container classes {'singularity': <class 'galaxy.tools.deps.containers.SingularityContainer'>, 'docker': <class 'galaxy.tools.deps.containers.DockerContainer'>}
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 Container type singularity
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 Container ID None
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 Container type docker
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 Container ID None
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 ContainerFinder::find_container 05
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 ContainerFinder::find_best_container_description 01
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,017 Checking with container resolver [ExplicitContainerResolver[]] found description [None]
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,018 ContainerFinder::find_container 06
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,018 ContainerFinder::find_container 08
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,018 ContainerFinder::find_container 10
galaxy.jobs.command_factory INFO 2018-03-07 14:56:06,047 Built script [/opt/galaxy_data/database/job_working_directory/000/5/tool_script.sh] for tool command [biosigner_wrapper.R  dataMatrix_in "/opt/galaxy_data/files/000/dataset_1.dat" sampleMetadata_in "/opt/galaxy_data/files/000/dataset_2.dat" variableMetadata_in "/opt/galaxy_data/files/000/dataset_3.dat"  respC "gender"  methodC "all" bootI "5" tierC "S" pvalN "0.05" seedI "123"  variableMetadata_out "/opt/galaxy_data/files/000/dataset_7.dat" figure_tier "/opt/galaxy_data/files/000/dataset_8.dat" figure_boxplot "/opt/galaxy_data/files/000/dataset_9.dat" information "/opt/galaxy_data/files/000/dataset_10.dat"]
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,256 ContainerFinder::find_container 01
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,257 ContainerFinder::find_container 02
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,257 ContainerFinder::find_container 03
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,257 ContainerFinder::find_container 04
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,257 Container classes {'singularity': <class 'galaxy.tools.deps.containers.SingularityContainer'>, 'docker': <class 'galaxy.tools.deps.containers.DockerContainer'>}
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,257 Container type singularity
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,257 Container ID None
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,258 Container type docker
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,258 Container ID None
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,258 ContainerFinder::find_container 05
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,258 ContainerFinder::find_best_container_description 01
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,258 Checking with container resolver [ExplicitContainerResolver[]] found description [None]
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,258 ContainerFinder::find_container 06
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,258 ContainerFinder::find_container 08
galaxy.tools.deps.containers INFO 2018-03-07 14:56:06,258 ContainerFinder::find_container 10
galaxy.jobs.runners ERROR 2018-03-07 14:56:06,259 (5) Unhandled exception calling queue_job
Traceback (most recent call last):
  File "/galaxy/lib/galaxy/jobs/runners/__init__.py", line 104, in run_next
    method(arg)
  File "/galaxy/lib/galaxy/jobs/runners/kubernetes.py", line 107, in queue_job
    "spec": self.__get_k8s_job_spec(job_wrapper)
  File "/galaxy/lib/galaxy/jobs/runners/kubernetes.py", line 161, in __get_k8s_job_spec
    k8s_job_spec = {"template": self.__get_k8s_job_spec_template(job_wrapper)}
  File "/galaxy/lib/galaxy/jobs/runners/kubernetes.py", line 175, in __get_k8s_job_spec_template
    "containers": self.__get_k8s_containers(job_wrapper)
  File "/galaxy/lib/galaxy/jobs/runners/kubernetes.py", line 215, in __get_k8s_containers
    "image": self._find_container(job_wrapper).container_id,
AttributeError: 'NullContainer' object has no attribute 'container_id'
pkrog commented 6 years ago

So we've got no container ID associated with docker type.

pkrog commented 6 years ago

With univariate:

destination_info ['limits_cpu', 'requests_memory', 'tools_id', 'limits_memory', 'max_pod_retrials', 'docker_owner_override', 'docker_tag_override', 'docker_enabled', 'requests_cpu', 'docker_image_override', 'docker_repo_override']

With biosigner:

destination_info ['docker_enabled', 'requests_cpu', 'requests_memory', 'limits_memory', 'limits_cpu']
pkrog commented 6 years ago

With univariate:

galaxy.jobs.runners INFO 2018-03-08 07:54:25,202 BaseJobRunner::put 02 job_wrapper.job_destination.params {'limits_cpu': 0.8, 'requests_memory': 0.5, 'tools_id': ['Univariate'], 'limits_memory': 0.9, 'max_pod_retrials': 3, 'docker_owner_override': 'phnmnl', 'docker_tag_override': 'v2.2.3_cv1.3.29', 'docker_enabled': True, 'requests_cpu': 0.4, 'docker_image_override': 'univariate', 'docker_repo_override': 'container-registry.phenomenal-h2020.eu'}

With biosigner:

galaxy.jobs.runners INFO 2018-03-08 07:55:04,837 BaseJobRunner::put 02 job_wrapper.job_destination.params {'docker_enabled': True, 'requests_cpu': 1.5, 'requests_memory': 1.8, 'limits_memory': 5, 'limits_cpu': 4}
pcm32 commented 6 years ago

Fixed with the PR above.