search

easybuilders / easybuild-framework

EasyBuild is a software installation framework in Python that allows you to install software in a structured and robust way.
https://easybuild.io
GNU General Public License v2.0
152 stars 202 forks source link

providing example easyconfig files for recent software versions #228

Closed fgeorgatos closed 11 years ago

fgeorgatos commented 12 years ago

(original topic was Add support for more GCC versions, changed by @boegel)

Either:

Search for gcc in the following page about which versions are interesting: http://distrowatch.com/table.php?distribution=redhat

fyi. these versions can become important when you (cross-)compile for such distros.

boegel commented 12 years ago

Correct me if I'm wrong, but I think EasyBuild already supports building these now.

It's just a matter of copying the example easyconfig file, changing the version in it, and giving it to EasyBuild. Soon (once #227 is merged), you'll be able to use the example easyconfig and just supply --software-version=4.7.1 on the command line.

JensTimmerman commented 12 years ago

But adding 2 extra gcc examples is quite informative to other people...

boegel commented 12 years ago

We currently already have 6 different easyconfig files for GCC, including one for GCC v4.7.0 .

I feel adding one for v4.7.1 will only get in the way, especially if only the version is different.

Maybe we should have some other way of indicating which versions we tested with?

JensTimmerman commented 12 years ago

Ah, ok, we seem to have enough already indeed.

see issue #208 for a way of indication who ever built a working version with what .eb file.

fgeorgatos commented 12 years ago

Hi guys,

I think you need to take the lib dependencies into account and some debugging done for downloading the tarballs!

diff GCC-4.7.?-CLooG-PPL.eb
0a1,6
> # This file is an EasyBuild recipy as per
https://github.com/hpcugent/easybuild
> #
> # Copyright:: Copyright (c) 2012 University of Luxembourg / LCSB
> # Author::    Fotis Georgatos <fotis.georgatos@uni.lu>
> # License::   MIT/GPL
>
2c8
< version='4.7.0'
---
> version='4.7.1'
14,17c20,23
<          'gmp-5.0.4.tar.bz2',
<          'mpfr-3.1.0.tar.gz',
<          'mpc-0.9.tar.gz',
<          'cloog-0.16.1.tar.gz',
---
>          'gmp-5.0.5.tar.bz2',
>          'mpfr-3.1.1.tar.gz',
>          'mpc-1.0.tar.gz',
>          'cloog-0.16.3.tar.gz',
25c31
<             'http://www.bastoul.net/cloog/pages/download/count.php3?url=.',
# CLooG official
---
>             'http://www.bastoul.net/cloog/pages/download/', # CLooG
official

More generally, I am working also on older versions which also have issues with lib dependencies; if you accept the pull, I would consider this issue closed status but, still continue contributing some more versions under the #228 theme.

I am of course in favor of removing duplicates where possible but, note that I take the effort to check them on both redhat & debian platforms so I'd consider them a bit more verified than existing ones.

On Fri, Aug 31, 2012 at 11:01 AM, Jens Timmerman notifications@github.comwrote:

Ah, ok, we seem to have enough already indeed.

— Reply to this email directly or view it on GitHubhttps://github.com/hpcugent/easybuild/issues/228#issuecomment-8186238.

boegel commented 12 years ago

I'm starting to think we need a separate easyconfigs repo for that, instead of shipping that as a part of EasyBuild.

I do see the value of letting people know that we can build the latest GCC + ClooG/PPL versions, but doing that by adding an extra easyconfig file for each and every version update (of one of the tools involved) to the EasyBuild codebase doesn't seem like the right way to go.

We need to think about what we want to put in easybuild/easyconfigs. Maybe that could be the place for all easyconfigs that were tested with EasyBuild, but I don't like the fact that it then becomes part of the EasyBuild release.

Should we separate out the easyconfigs folder to a separate public repository, where we can accept contributions by others, or maybe even allow automated pull requests being made to it when people used EasyBuild to 'successfully' build something? These easyconfigs would then include (a list of) buildstats, showing on which systems the build worked.

How we would define successfully is then a whole different matter, of course. Do we need to generate some form of testing output, to show that the build was tested as well?

fgeorgatos commented 12 years ago

Hi there,

On Sat, Sep 1, 2012 at 11:35 AM, Kenneth Hoste notifications@github.comwrote:

How we would define successfully is then a whole different matter, of course. Do we need to generate some form of testing output, to show that the build was tested as well?

As said in some of our early communications, you will inevitably arrive in a position of having to manage easyconfig contributions - nothing new here, same business as linux distros' contributed packages. The only question at this point is only how much control you would like to exercise in this and how.

Note that these contributions are not "experimental" per se (since they have been tested and used for real-world tasks and solve specific issues) so they should be able to find their way to other sites' easybuild resources faster.

In our case, these recent gcc builts allowed to: a) provide the very latest of gnu compiler toolkit for compatibility tests (gcc v4.7.1) b) succesfully compile latest CUDA toolkit in the debian squeeze environment (gcc v4.3.5) I guess other sites have similar needs, so it is good to share tested builds. Because of lib dependencies, these configurations are not trivial (to describe on the command line) Eg. think of the compilers versions used in past RHEL & SLES releases, they are certainly interesting for a good majority of HPC sites.

Perhaps, one way to do this is to ride on EasyBuild's feature to commit to a git tree (github?) the successful builds and, thus, automatigally contribute to a common pool. Needless to say, if this idea is accepted there is an opt-in/opt-out discussion here and either one will do fine as long as people are well informed on default tool's behaviour. (I would start with the "default share" approach since reverting it later is harmless).

cheers, Fotis

boegel commented 12 years ago

Assuming the an easyconfig file is well tested just because the build worked is a bit short-sighted I think.

We have no way of telling what people did on their end to make it work, neither do we (currently) have a way to let us know how the resulting binaries were tested/validated.

I think we need to look into this first, before accepting automated contributions to some central repository. Reviewing everything that comes in will be too time-consuming, and just accepting anything that rolls in is a no-go imho.

fgeorgatos commented 12 years ago

Hi there,

On Mon, Sep 3, 2012 at 10:35 AM, Kenneth Hoste notifications@github.comwrote:

I think we need to look into this first, before accepting automated contributions to some central repository. Reviewing everything that comes in will be too time-consuming, and just accepting anything that rolls in is a no-go imho.

Indeed, I think that reviewing all incoming .eb files except for basic checks -& 1 test build- is an impossible task!

  • eg. what would you do with "easybuild configuration for Eliza v42, this version passes the turing test"? :)

OTOH, it's good to define what are the acceptance criteria so that it is understood beforehand which items fit in the default distribution.

In the meantime, I'll let my GCC builds end up in the experimental repo, since I don't know what else to do with them.

cheers, Fotis

boegel commented 12 years ago

We will split up easybuild into easybuild-framework, easybuild-easyblocks and easybuild-easyconfings repos.

In the latter repo, we will host all example easyconfig files that were tested by us, and there I see no issue to include easyconfigs for all possible versions of a particular software package (if it's been tested well).

Later, we may look into some kind of contrib repo, where everyone can push easyconfig files too, with fewer restrictions on testing and such. Just to have a central place where people can find easyconfig files. We might even look into automated pushes to this repo on successful EasyBuild runs.

I'm going to move this to milestone 1.0, and it might be further delayed to post 1.0. I would only consider this discussion closed if we have a central place where people can push easyconfigs (and potentially easyblocks) to without too much hassle.

fgeorgatos commented 12 years ago

Hi there,

first, congratulations for the recent push related to #99/#272, this really gets us a tick closer to v0.9 and far more stable APIs.

And it's about time to start yet another round of pkgsrc experiments.

Later, we may look into some kind of contrib repo, where everyone can push

So far, I had assumed that this repo would fulfill the "untested" purpose: https://github.com/fgeorgatos/easybuild.experimental ie. we need an area where people can share their easyconfigs with intentional lax attitude as regards code style, compatibility or, even basic testing. Think of the blind pkgsrc-based builds.

Feel free to use it for experimentation or otherwise (auto-git-push?); just let me know how it would be fit in the bigger easybuild business.

I admit a CONTRIBUTING file should be added, basically stating to create your subdirectory in there and do whatever you want under it.

We might even look into automated pushes to this repo on successful EasyBuild runs.

Yes, as said earlier, it may be wise to start first with an "opt-out" attitude, so that we later move to "opt-in", if people think that's best. (ie. prefer to attract the "open" guys at the early stages of an OSS tool)

The opposite strategy would be too risky to follow, for any production sites, since some of them might be compiling codes they consider "confidential". (ie. first opt-in, then opt-out might create wild community opposition)

cheers, Fotis

boegel commented 12 years ago

I think we can definitely use the easybuild.experimental repo now as a way of allowing easy contribution, with the aim of getting it cleaned up and tested for inclusion in the main repos. We should make sure it's documented on the central wiki somwhere, can you take care of that?

I agree, people should explicitely state that they want EasyBuild to auto-push stuff to e.g. easybuild.experimental, and we should encourage them to enable this, instead of just enabling it by default.

fgeorgatos commented 12 years ago

Hi Kenneth,

On Mon, Oct 22, 2012 at 11:18 AM, Kenneth Hoste notifications@github.comwrote:

I think we can definitely use the easybuild.experimental repo now as a way of allowing easy contribution, with the aim of getting it cleaned up and tested for inclusion in the main repos. We should make sure it's documented on the central wiki somwhere, can you take care of that?

I can write some text if you need that but, the current github wiki workflow is a little bit troublesome for me... anyway, here is the banner text:

""" The easybuild.experimental repo has been created while having in mind the community at large and those who simply wish to try out things without committing to any particular quality level:

I agree, people should explicitely state that they want EasyBuild to auto-push stuff to e.g. easybuild.experimental, and we should encourage them to enable this, instead of just enabling it by default.

OK, it seems you converge on the concept of opt-in. Better something, than nothing :)

cheers, Fotis

boegel commented 11 years ago

@fgeorgatos: I created a wiki page linked from the front page, see https://github.com/hpcugent/easybuild/wiki/Experimental-repo .

This should be sufficient for now, do you agree?

I'm closing this issue, because since the split having easyconfig files for recent software versions isn't an issue anymore. The easybuild-easyconfigs package can hosts tons of easyconfig files, including several ones for a single software package which only differ in version.