Request for Comments, as regards `common_deps`

fgeorgatos commented 9 years ago

Hellos to all,

I have been bugging fellows for a while yet this should not wait any longer, we discussed this in yesterday's EB hangout:

we need to specify (as far as feasible) a preferred list of common dependencies; It does not have to be final/perfect/ideal, but there has to be some guideline today; if not we will pay the high price of effort duplication in the next weeks/months.

This issue intends to supersede #384, so I copy-paste from that:

easyblock = 'Bundle'

name = 'common_deps'
version = '2.0'

homepage = 'http://hpcbios.readthedocs.org/en/latest/HPCBIOS_2013-01.html'
description = """The purpose of this collection is to provide common dependencies in a bundle,
 so that software/modules can be mixed and matched easily for composite pipelines 
 in Life Sciences and other fields"""

toolchain = {'name': 'goolf', 'version': '1.7.20'}

dependencies = [
    ('bzip2', '1.0.6'),
    ('zlib', '1.2.8'),
    ('libreadline', '6.3'),
    ('ncurses', '5.9'),
# **the ones beyond this line are also known as biodeps**
    ('Boost', '1.58.0', '-Python-2.7.10'),
    ('SAMtools', '0.1.19'),
    ('Perl', '5.20.0', '-bare'),
    ('Java', '1.7.0_80', '', True),
    ('libpng', '1.6.17'),
]

moduleclass = 'bundle'

This is intentionally a conservative upgrade over biodeps/1.6, to minimize issues.

I am a little bit inclined to serve life scientists' needs (because applications are many and painful), however your arguments should be shot from all directions, regardless of my priorities.

Request For Comments, as regards effects of version advancement/creep for:

SAMtools/1.2 is known to be better than others; however @rjeschmi reports TopHat compatibility issues, so 0.1.19 might be more prudent choice
Perl/5.22 was released last week; @rjeschmi reports that 5.20 has greater compatibility and should play conservative
ncurses/6.0 pre-release is out since last week; it should be backwards compatible with 5.9 line; my guess is that we will assume "do-not-fix-if-not-broken" for now, although 6.0 would be nice
Boost is known to be incompatible with itself. For now we make the assumption 1.58 is OK
Java/1.8 is supposed to be better and compatible; not all might agree, fi. look for GATK here: http://stackoverflow.com/questions/21402136/java-1-8-versus-java-1-7-compatibility-issue

In short, the above proposition is a somewhat conservative upgrade, with the outlook of making a more advanced common_deps release in the future.

your feedback?!

fgeorgatos commented 9 years ago

and also, ever important:

Python/2.7.10 is assumed to be the latest and greatest. This will inevitably leave behind things like QIIME which require 2.7.3, but we cannot let everybody wait in the name of intricacies of a single package...

rjeschmi commented 9 years ago

The goal is a reasonable starting point.

Oh you might want to pick a version of cmake, particularly one that would be compatible with bcl2fastq. It was something I hat tried to get working in the merge sprint but didn't have luck and redists a bunch of deps that we would probably build with easybuild

On Fri, Jun 12, 2015, 6:52 AM Fotis Georgatos notifications@github.com wrote:

and also, ever important:

Python/2.7.10 is assumed to be the latest and greatest. This will inevitably leave behind things like QIIME which require 2.7.3, but we cannot let everybody wait in the name of intricacies of a single package...

— Reply to this email directly or view it on GitHub https://github.com/hpcugent/easybuild-easyconfigs/issues/1689#issuecomment-111447157 .

fgeorgatos commented 9 years ago

@rjeschmi: I can already smell a dependency hell approaching ;-) but let's do our best...

pescobar commented 9 years ago

I would not add samtools as dependency for all my bioinformatics tools. Neither perl, python or java. IMHO, I would define as biodeps just the latest versions of

bzip2 zlib libreadline ncurses libpng

I think the version of Perl/Python/Java should not be binded to bionfo tools. That should be a site-specific decision.....my two cents....

fgeorgatos commented 9 years ago

@pescobar: OK, thanks.

As I see it, we converge as of today (ncurses/6.0 is not released yet). In the meeting we thought of renaming biodeps to common_deps, to make it more appealing for other fields; ie. it is going to include: bzip2, zlib, libreadline, ncurses, libpng

N.B. Not having a PR for common_deps today does not block us from now on to advance:

as long as people stick on these versions, many new PRs are going to be aligned:

dependencies = [
    ('bzip2', '1.0.6'),
    ('zlib', '1.2.8'),
    ('libreadline', '6.3'),
    ('ncurses', '5.9'),
    ('libpng', '1.6.17'),
]

Thanks for pruning a good part of the ongoing dependency hell tree!

fgeorgatos commented 9 years ago

IMHO, cmake is invariably a builddep, and as such there is no need to specify version (anything goes). Unless, there is a counter-example, I'd suggest to keep options open here! F.

On 12 June 2015 at 12:07, Robert Schmidt notifications@github.com wrote:

The goal is a reasonable starting point.

Oh you might want to pick a version of cmake, particularly one that would be compatible with bcl2fastq. It was something I hat tried to get working in the merge sprint but didn't have luck and redists a bunch of deps that we would probably build with easybuild

On Fri, Jun 12, 2015, 6:52 AM Fotis Georgatos notifications@github.com wrote:

and also, ever important:

Python/2.7.10 is assumed to be the latest and greatest. This will inevitably leave behind things like QIIME which require 2.7.3, but we cannot let everybody wait in the name of intricacies of a single package...

— Reply to this email directly or view it on GitHub < https://github.com/hpcugent/easybuild-easyconfigs/issues/1689#issuecomment-111447157

.

— Reply to this email directly or view it on GitHub https://github.com/hpcugent/easybuild-easyconfigs/issues/1689#issuecomment-111453172 .

echo "sysadmin know better bash than english"|sed s/min/mins/ \ | sed 's/better bash/bash better/' # signal detected in a CERN forum

fgeorgatos commented 9 years ago

@npklein: are you satisfied with the above versions? do they make sense in your context?

npklein commented 9 years ago

@fgeorgatos looking through our custom .eb files we don't use those dependencies very often, and most dependencies I only see back in the .eb files for the different version of 2 or 3 different programs, so I don't think I can give a meaningful comment about common dependencies.

pescobar commented 9 years ago

I have just hit this problem rebuilding my python-2.7.5 installation https://github.com/yyuu/pyenv/issues/149

That bug report is for ubuntu 14.04 but I found the same issue in centos6. By now, I will stay in libreadline version 6.2

fgeorgatos commented 9 years ago

@pescobar: only now I notice your comment on libreadline/6.2, thanks! I can't recall such issues on rh6. I am not changing course yet, but if this bug bugs more people, this should certainly be revised. Ideally, we should have automation to catch such issues, as early as possible.

boegel commented 9 years ago

This involves Python 2.7.5, which is rather ancient.

Maybe we should agree that the list of common deps only applies to the most recent software versions, and that it's OK to diverge from this in case there are technical reasons to do so?

easybuilders / easybuild-easyconfigs

Request for Comments, as regards `common_deps` #1689