PythonPackage support for custom extras

[jpecar]

Need to be able to pass extra parameters down to installation command. An example of this is Toil: https://toil.readthedocs.io/en/latest/gettingStarted/install.html#extras

[boegel]

@jpecar I took a quick stab at this in #1517 (totally untested), can you see if that works for you?

In the easyconfig, it should suffice to add something like:

extras = 'all'

[jpecar]

Should the effect of this extras be visible in the log file? If I look for "INFO cmd" in the log file I see the commands being run, but it does not include the string I pass through extras. Also the fact that required dependencies are not installed suggest that it is in fact not passed down correctly.

[boegel]

@jpecar If you use eb --trace --include-easyblocks pythonpackage.py test.eb (where you replace test.eb with the name of your easyconfig file), you should see the command pass by.

Make sure you're using an updated pythonpackage.py of course.

If that doesn't show what you expect, please share your easyconfig file, it's possible I overlooked something...

[jpecar]

I can get it passed down to install_cmd if I use this combination: use_pip = True use_pip_for_deps = True installopts = 'toil[cwl,wdl]' I can also see that it starts to process dependencies since one of them is failing with python 3.7. I think know how to handle that.

But I expect it should work with your patch as well?

[boegel]

@jpecar The intention of my changes is that extras = 'cwl,wdl' works too. If it doesn't, I need to look into why it doesn't work (please share your easyconfig for easy testing on my side in that case).

[jpecar]

It is very simple:

easyblock = 'PythonPackage'

name = 'toil'
version = '3.17.0'
versionsuffix = '-Python-%(pyver)s'

homepage = 'https://github.com/DataBiosphere/toil'
description = """Toil is a scalable, efficient, cross-platform pipeline management system, written entirely in Python, and designed around the principles of functional programming."""

toolchain = {'name': 'foss', 'version': '2018b'}

source_urls = [PYPI_SOURCE] 
sources = [SOURCE_TAR_GZ] 

dependencies = [('Python', '3.7.0'),
                            ('libffi', '3.2.1'),
]

sanity_check_paths = {
    'files': ['bin/toil'],
    'dirs': ['lib/python%(pyshortver)s/site-packages'],
}

sanity_check_commands = [
        'cwltool --help'
]

moduleclass = 'lang'

Try adding extras='cwl' to it. If it works, the build should succeed. If it doesn't, cwltool should complain about missing dependency on galaxy-lib.

I ended up modifying the source so the extensions I want get built by default.

[surak]

Is this on the documentation yet?

[boegel]

Extras can be installed by listing them via pip_install_extras, see #1980.

For example:

use_pip_extras = 'foo,bar'

Will result in installing the Pythin package with pip install --prefix %(installdir)s .[foo,bar]