galaxyproject / galaxy

Data intensive science for everyone.
https://galaxyproject.org
Other
1.38k stars 998 forks source link

environment module dependency #4550

Closed alpapan closed 7 years ago

alpapan commented 7 years ago

Hello

I can't seem to find documentation on how to get the environment modules setup correctly. Not sure where ModuleDependency is defined. Error below (after resolving #4452). Any ideas?

XML is

  <modules modulecmd="/usr/bin/modulecmd" modulepath="/data_storage/sysadmin/modules" versionless="true" prefetch="true" find_by="avail"/>

And galaxy release_17.05 is used

URL: https://galaxy.gpi.westernsydney.edu.au/admin/manage_tool_dependencies
File '/data_storage/www-cluster/galaxy/lib/galaxy/web/framework/middleware/error.py', line 154 in __call__
  app_iter = self.application(environ, sr_checker)
File '/data_storage/www-cluster/galaxy/.venv/lib/python2.7/site-packages/paste/recursive.py', line 85 in __call__
  return self.application(environ, start_response)
File '/data_storage/www-cluster/galaxy/.venv/lib/python2.7/site-packages/paste/httpexceptions.py', line 640 in __call__
  return self.application(environ, start_response)
File '/data_storage/www-cluster/galaxy/lib/galaxy/web/framework/base.py', line 135 in __call__
  return self.handle_request( environ, start_response )
File '/data_storage/www-cluster/galaxy/lib/galaxy/web/framework/base.py', line 214 in handle_request
  body = method( trans, **kwargs )
File '/data_storage/www-cluster/galaxy/lib/galaxy/web/framework/decorators.py', line 98 in decorator
  return func( self, trans, *args, **kwargs )
File '/data_storage/www-cluster/galaxy/lib/galaxy/web/base/controllers/admin.py', line 1154 in manage_tool_dependencies
  requirements_status=view.toolbox_requirements_status,
File '/data_storage/www-cluster/galaxy/lib/galaxy/tools/deps/views.py', line 210 in toolbox_requirements_status
  for r, tids in self.tool_ids_by_requirements.items()}
File '/data_storage/www-cluster/galaxy/lib/galaxy/tools/deps/views.py', line 210 in <dictcomp>
  for r, tids in self.tool_ids_by_requirements.items()}
File '/data_storage/www-cluster/galaxy/lib/galaxy/tools/deps/views.py', line 213 in get_requirements_status
  dependencies = self.show_dependencies(tool_requirements_d, installed_tool_dependencies)
File '/data_storage/www-cluster/galaxy/lib/galaxy/tools/deps/views.py', line 55 in show_dependencies
  dependencies_per_tool = {tool: self._dependency_manager.requirements_to_dependencies(requirements, **kwds) for tool, requirements in tool_requirements_d.items()}
File '/data_storage/www-cluster/galaxy/lib/galaxy/tools/deps/views.py', line 55 in <dictcomp>
  dependencies_per_tool = {tool: self._dependency_manager.requirements_to_dependencies(requirements, **kwds) for tool, requirements in tool_requirements_d.items()}
File '/data_storage/www-cluster/galaxy/lib/galaxy/tools/deps/__init__.py', line 120 in requirements_to_dependencies
  requirement_to_dependency = self._requirements_to_dependencies_dict(requirements, **kwds)
File '/data_storage/www-cluster/galaxy/lib/galaxy/tools/deps/__init__.py', line 168 in _requirements_to_dependencies_dict
  log.debug(dependency.resolver_msg)
File '/data_storage/www-cluster/galaxy/lib/galaxy/tools/deps/resolvers/__init__.py', line 258 in resolver_msg
  return "Using dependency %s version %s of type %s" % (self.name, self.version, self.dependency_type)
AttributeError: 'ModuleDependency' object has no attribute 'name'
alpapan commented 7 years ago

got it.... It seems this happens if there is NO tool that uses type=modules...

Not sure if this means I should close the issue or somehow improve the error message eg

        <requirement type="modules">
mvdbeek commented 7 years ago

got it.... It seems this happens if there is NO tool that uses type=modules...

No, that is not right. Please read the introduction in https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html and then https://docs.galaxyproject.org/en/master/admin/dependency_resolvers.html#environment-modules-dependency-resolver.

If then you still encounter problems please include the relevant sections of your dependency_resolvers_conf.xml file.

alpapan commented 7 years ago

Hello They were included in the issue above

  <modules modulecmd="/usr/bin/modulecmd" modulepath="/data_storage/sysadmin/modules" versionless="true" prefetch="true" find_by="avail"/>
mvdbeek commented 7 years ago

Did you modify tools to state <requirement type="modules"> ? This must always be package, not modules.

alpapan commented 7 years ago

Hello It is my own tool so it is not in a shed. I had it as modules because one of the galaxy docs said so.

changed it to package. i assume the it checks using the resolvers until it finds one? (the rest of the dependency file is as default).

by the way, the environment_modules_mapping.yml.sample is empty (and there is no environment_modules_mapping.yml)

will test tomorrow

alpapan commented 7 years ago
'ModuleDependency' object has no attribute 'name'
nsoranzo commented 7 years ago

I had it as modules because one of the galaxy docs said so.

@alpapan If you can tell us which doc, we can fix it.

alpapan commented 7 years ago

Hello

Sorry I can't find the documentation at the moment.

However I do get the error and I just understand where it may be coming from....:

AttributeError: 'ModuleDependency' object has no attribute 'name'

Here is the tool XML that when included causes the above error (and when removed, it also removes the error).

planemo check

$ planemo lint --fail_level error
Linting tool /data_storage/www-cluster/galaxy/tools/alexiesoft_dew2/alexiesoft_dew.xml
Applying linter tests... WARNING
.. WARNING: No tests found, most tools should define test cases.
.. WARNING: No valid test(s) found.
Applying linter output... CHECK
.. INFO: 1 outputs found.
Applying linter inputs... CHECK
.. INFO: Found 10 input parameters.
Applying linter help... CHECK
.. CHECK: Tool contains help section.
.. CHECK: Help contains valid reStructuredText.
Applying linter general... CHECK
.. CHECK: Tool defines a version [21Apr16].
.. CHECK: Tool defines a name [DEW].
.. CHECK: Tool defines an id [alexiesoft_dew].
.. CHECK: Tool specifies profile version [17.05].
Applying linter command... CHECK
.. INFO: Tool contains a command.
Applying linter citations... WARNING
.. WARNING: No citations found, consider adding citations to your tool.
Applying linter tool_xsd... CHECK
.. INFO: File validates against XML schema.

Macro:

<macros>
    <token name="@VERSION@">21Apr16</token>
    <xml name="requirements">
        <requirements>
          <requirement type="package" version="@VERSION@">DEW</requirement>
        </requirements>
    </xml>
    <xml name="stdio">
        <stdio>
        </stdio>
    </xml>
</macros>

Tool's xml (with some code replaced with [....] to reduce it)

<tool id="alexiesoft_dew" name="DEW" version="21Apr16" profile="17.05">
    <description>One button differential gene expression with DEW</description>
   <macros>
        <import>macros.xml</import>
    </macros>
    <expand macro="requirements" />
    <expand macro="stdio" />

    <command><![CDATA[
        #set $read_1_final_list = " "
        dew.pl
                -infile $reference_genes
                -sequence $sequence
                -format $select_format

                #set $read_1_original_names = " "
                #set $uid_path = " "
                #for $read_1 in $read_1_list
                        #set $read_1_final_list += str($read_1) + " "
                        #set $read_1_original_names += str($read_1.name) + " "
                #end for

                -1read $read_1_final_list

                -uid current

                -do_galaxy_cleanup

                -sample_names $read_1_original_names

                #if $settings.advanced == "a"
                        $settings.no_graphs
                        $settings.contextual
                        $settings.isoforms
                        $settings.genomewide
                #end if
    ]]></command>
<inputs>
[......]
</inputs>
    <outputs>
[....]
    </outputs>
    <help><![CDATA[
        This tools estimates gene expression from any FASTA file
    ]]></help>
</tool>

There is also a temporary .shed.yml but not sure if you want that.

The Dependency XML is now

<dependency_resolvers>
  <tool_shed_packages />
  <galaxy_packages />
  <conda />
  <galaxy_packages versionless="true" />
  <conda versionless="true" />
  <modules modulecmd="/usr/bin/modulecmd" modulepath="/data_storage/sysadmin/modules" versionless="true" prefetch="true" find_by="avail"/>
</dependency_resolvers>

Note that my module is a function but i'm using modulecmd= the actual file

$ type module
module is a function
module ()
{
    eval `/usr/bin/modulecmd sh $*`
}

$ uname -a 
Linux hill 4.4.0-93-generic #116~14.04.1-Ubuntu SMP Mon Aug 14 16:07:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

$ module --version
VERSION=3.2.10
DATE=2012-12-21
[...]

$ git branch
* release_17.05

Any ideas would be appreciated. i'm sure i'm doing something simple wrong