search

easybuilders / easybuild

EasyBuild - building software with ease
http://easybuild.io
GNU General Public License v2.0
464 stars 143 forks source link

Using own toolchain to compile and install python #364

Closed maikenp closed 7 years ago

maikenp commented 7 years ago

Hi, I am trying to compile python using a toolchain I have created with gcc, intel mkl, open mpi.

I get the error -bash-4.1$ eb Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2-bare.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot --installpath-modules $HOME/easybuild --include-toolchains=/cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/all/gimklompi/520-20172-1102 ERROR: Failed to parse configuration options: 'Failed to import easyblock easybuild.toolchains.520-20172-1102 from /tmp/eb-WzANum/included-toolchains/easybuild/toolchains: No module named 520-20172-1102' So easybuild does not seem to find the module, however, when I look into the location stated I see the module file: -bash-4.1$ ls /tmp/eb-WzANum/included-toolchains/easybuild/toolchains 520-20172-1102 compiler fft __init__.py linalg mpi

Details: I have called this toolchain gimklompi And it has currently a version (ignore ugly version number) 520-20172-1102.

As far as I can see the modulefile of gimklompi is placed where it should in /cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/ which is where easybuild is placed on our hpc system.

I then try to use this toolchain to install python: eb Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2-bare.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot --installpath-modules $HOME/easybuild --include-toolchains=/cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/all/gimklompi/520-20172-1102

My Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2 easyconfig file looks like this:

name = 'Python'
version = '2.7.10'
versionsuffix = '-bare'

homepage = 'http://python.org/'
description = """Python is a programming language that lets you work more quickly and integrate your systems
 more effectively."""

toolchain = {'name': 'gimklompi', 'version': '520-20172-1102'}
toolchainopts = {'pic': True, 'opt': True, 'optarch': True}

source_urls = ['http://www.python.org/ftp/%(namelower)s/%(version)s/']
sources = [SOURCE_TGZ]

""" python needs bzip2 to build the bz2 package """
dependencies = [
    ('bzip2', '1.0.6'),
    ('zlib', '1.2.8'),
    ('libreadline', '6.3'),
    ('ncurses', '5.9'),
    ('SQLite', '3.8.10.2'),
    ('Tk', '8.6.4', '-no-X11'),  # this requires a full X11 stack
    # OS dependency should be preferred if the os version is more recent then this version,
    # it's nice to have an up to date openssl for security reasons
    # ('OpenSSL', '1.0.1m'),
]

osdependencies = [('openssl-devel', 'libssl-dev', 'libopenssl-devel')]

""" bare installation: no extensions included """
exts_list = []

moduleclass = 'lang'

In the step-by-step guide here: https://github.com/easybuilders/easybuild/wiki/Step-by-step-guide#step7 the part that I need is not yet included it seems. There might be some other guides somewhere?

ocaisa commented 7 years ago

Hi,

You given the location of a module file but toolchain definitions in EasyBuild are python files. Here's an example that's very close to what you require: https://github.com/easybuilders/easybuild-framework/blob/master/easybuild/toolchains/iomkl.py Your toolchains file should be called gomkl.py and look like (untested):

from easybuild.toolchains.gompi import Gompi from easybuild.toolchains.fft.intelfftw import IntelFFTW from easybuild.toolchains.linalg.intelmkl import IntelMKL

class Gomkl(Gompi, IntelMKL, IntelFFTW): """ Compiler toolchain with GCC, OpenMPI, Intel Math Kernel Library (MKL) and Intel FFTW wrappers. """ NAME = 'gomkl' SUBTOOLCHAIN = Gompi.NAME

To check it was imported correctly use the command line options '--include-toolchains=/path/to/gomkl.py --list-toolchains' and make sure your toolchain is in the list

Alan

On 18 July 2017 at 16:29, maikenp notifications@github.com<mailto:notifications@github.com> wrote:

Hi, I am trying to compile python using a toolchain I have created with gcc, intel mkl, open mpi.

I get the error -bash-4.1$ eb Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2-bare.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot --installpath-modules $HOME/easybuild --include-toolchains=/cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/all/gimklompi/520-20172-1102 ERROR: Failed to parse configuration options: 'Failed to import easyblock easybuild.toolchains.520-20172-1102 from /tmp/eb-WzANum/included-toolchains/easybuild/toolchains: No module named 520-20172-1102' So easybuild does not seem to find the module, however, when I look into the location stated I see the module file: -bash-4.1$ ls /tmp/eb-WzANum/included-toolchains/easybuild/toolchains 520-20172-1102 compiler fft init.py linalg mpi

Details: I have called this toolchain gimklompi And it has currently a version (ignore ugly version number) 520-20172-1102.

As far as I can see the modulefile of gimklompi is placed where it should in /cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/ which is where easybuild is placed on our hpc system.

I then try to use this toolchain to install python: eb Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2-bare.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot --installpath-modules $HOME/easybuild --include-toolchains=/cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/all/gimklompi/520-20172-1102

My Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2 easyconfig file looks like this: `name = 'Python' version = '2.7.10' versionsuffix = '-bare'

homepage = 'http://python.org/' description = """Python is a programming language that lets you work more quickly and integrate your systems more effectively."""

toolchain = {'name': 'gimklompi', 'version': '520-20172-1102'} toolchainopts = {'pic': True, 'opt': True, 'optarch': True}

source_urls = ['http://www.python.org/ftp/%(namelower)s/%(version)s/'] sources = [SOURCE_TGZ]

python needs bzip2 to build the bz2 package

dependencies = [ ('bzip2', '1.0.6'), ('zlib', '1.2.8'), ('libreadline', '6.3'), ('ncurses', '5.9'), ('SQLite', '3.8.10.2'), ('Tk', '8.6.4', '-no-X11'), # this requires a full X11 stack

OS dependency should be preferred if the os version is more recent then this version,

it's nice to have an up to date openssl for security reasons

('OpenSSL', '1.0.1m'),

]

osdependencies = [('openssl-devel', 'libssl-dev', 'libopenssl-devel')]

bare installation: no extensions included

exts_list = []

moduleclass = 'lang' ` In the step-by-step guide here: https://github.com/easybuilders/easybuild/wiki/Step-by-step-guide#step7 the part that I need is not yet included it seems. There might be some other guides somewhere?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/easybuilders/easybuild/issues/364, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADqZtVzcyxb-gyz_b7FYxigZxdKy-Fkmks5sPMFVgaJpZM4Obbtn.

-- Dr. Alan O'Cais E-CAM Software Manager Juelich Supercomputing Centre Forschungszentrum Juelich GmbH 52425 Juelich, Germany

Phone: +49 2461 61 5213 Fax: +49 2461 61 6656 E-mail: a.ocais@fz-juelich.demailto:a.ocais@fz-juelich.de WWW: http://www.fz-juelich.de/ias/jsc/EN

Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt

ocaisa commented 7 years ago

If you want to keep using the name you already had then you could module the relevant line: NAME = 'gimklomp' ...but that name doesn't really follow the format for similar toolchains

On 18 July 2017 at 16:52, Alan O'Cais a.ocais@fz-juelich.de<mailto:a.ocais@fz-juelich.de> wrote: Hi,

You given the location of a module file but toolchain definitions in EasyBuild are python files. Here's an example that's very close to what you require: https://github.com/easybuilders/easybuild-framework/blob/master/easybuild/toolchains/iomkl.py Your toolchains file should be called gomkl.py and look like (untested):

from easybuild.toolchains.gompi import Gompi from easybuild.toolchains.fft.intelfftw import IntelFFTW from easybuild.toolchains.linalg.intelmkl import IntelMKL

class Gomkl(Gompi, IntelMKL, IntelFFTW): """ Compiler toolchain with GCC, OpenMPI, Intel Math Kernel Library (MKL) and Intel FFTW wrappers. """ NAME = 'gomkl' SUBTOOLCHAIN = Gompi.NAME

To check it was imported correctly use the command line options '--include-toolchains=/path/to/gomkl.py --list-toolchains' and make sure your toolchain is in the list

Alan

On 18 July 2017 at 16:29, maikenp notifications@github.com<mailto:notifications@github.com> wrote:

Hi, I am trying to compile python using a toolchain I have created with gcc, intel mkl, open mpi.

I get the error -bash-4.1$ eb Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2-bare.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot --installpath-modules $HOME/easybuild --include-toolchains=/cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/all/gimklompi/520-20172-1102 ERROR: Failed to parse configuration options: 'Failed to import easyblock easybuild.toolchains.520-20172-1102 from /tmp/eb-WzANum/included-toolchains/easybuild/toolchains: No module named 520-20172-1102' So easybuild does not seem to find the module, however, when I look into the location stated I see the module file: -bash-4.1$ ls /tmp/eb-WzANum/included-toolchains/easybuild/toolchains 520-20172-1102 compiler fft init.py linalg mpi

Details: I have called this toolchain gimklompi And it has currently a version (ignore ugly version number) 520-20172-1102.

As far as I can see the modulefile of gimklompi is placed where it should in /cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/ which is where easybuild is placed on our hpc system.

I then try to use this toolchain to install python: eb Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2-bare.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot --installpath-modules $HOME/easybuild --include-toolchains=/cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/modules/all/gimklompi/520-20172-1102

My Python-2.7.10-gimklomp-5.2.0-2017.2-1.10.2 easyconfig file looks like this: `name = 'Python' version = '2.7.10' versionsuffix = '-bare'

homepage = 'http://python.org/' description = """Python is a programming language that lets you work more quickly and integrate your systems more effectively."""

toolchain = {'name': 'gimklompi', 'version': '520-20172-1102'} toolchainopts = {'pic': True, 'opt': True, 'optarch': True}

source_urls = ['http://www.python.org/ftp/%(namelower)s/%(version)s/'] sources = [SOURCE_TGZ]

python needs bzip2 to build the bz2 package

dependencies = [ ('bzip2', '1.0.6'), ('zlib', '1.2.8'), ('libreadline', '6.3'), ('ncurses', '5.9'), ('SQLite', '3.8.10.2'), ('Tk', '8.6.4', '-no-X11'), # this requires a full X11 stack

OS dependency should be preferred if the os version is more recent then this version,

it's nice to have an up to date openssl for security reasons

('OpenSSL', '1.0.1m'),

]

osdependencies = [('openssl-devel', 'libssl-dev', 'libopenssl-devel')]

bare installation: no extensions included

exts_list = []

moduleclass = 'lang' ` In the step-by-step guide here: https://github.com/easybuilders/easybuild/wiki/Step-by-step-guide#step7 the part that I need is not yet included it seems. There might be some other guides somewhere?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://github.com/easybuilders/easybuild/issues/364, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADqZtVzcyxb-gyz_b7FYxigZxdKy-Fkmks5sPMFVgaJpZM4Obbtn.

-- Dr. Alan O'Cais E-CAM Software Manager Juelich Supercomputing Centre Forschungszentrum Juelich GmbH 52425 Juelich, Germany

Phone: +49 2461 61 5213<tel:02461%20615213> Fax: +49 2461 61 6656<tel:02461%20616656> E-mail: a.ocais@fz-juelich.demailto:a.ocais@fz-juelich.de WWW: http://www.fz-juelich.de/ias/jsc/EN

-- Dr. Alan O'Cais E-CAM Software Manager Juelich Supercomputing Centre Forschungszentrum Juelich GmbH 52425 Juelich, Germany

Phone: +49 2461 61 5213 Fax: +49 2461 61 6656 E-mail: a.ocais@fz-juelich.demailto:a.ocais@fz-juelich.de WWW: http://www.fz-juelich.de/ias/jsc/EN

Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt

maikenp commented 7 years ago

Thanks a lot for your answers. I do have a toolchain file (but did not post it), looking like this:

from easybuild.toolchains.compiler.gcc import Gcc
from easybuild.toolchains.linalg.intelmkl import IntelMKL
from easybuild.toolchains.mpi.openmpi import OpenMPI

class GIMKLOMPI(Gcc, IntelMKL, OpenMPI):
    """ Compiler toolchain with GCC, OpenMPI and IntelMKL"""
    NAME = 'gimklompi'

After constructing the easyconfig file and running the eb command on the easyconfig file, it produces a module file. But thanks for the tips, and link, I will have a look and try again!

ocaisa commented 7 years ago

Your missing your fft definition in the file you posted (so your toolchain is incomplete)

On 19 July 2017 at 09:16, maikenp notifications@github.com<mailto:notifications@github.com> wrote:

Thanks a lot for your answers. I do have a toolchain file (but did not post it), looking like this:

`from easybuild.toolchains.compiler.gcc import Gcc from easybuild.toolchains.linalg.intelmkl import IntelMKL from easybuild.toolchains.mpi.openmpi import OpenMPI

class GIMKLOMPI(Gcc, IntelMKL, OpenMPI): """ Compiler toolchain with GCC, OpenMPI and IntelMKL""" NAME = 'gimklompi'`

After constructing the easyconfig file and running the eb command on the easyconfig file, it produces a module file. But thanks for the tips, and link, I will have a look and try again!

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/easybuilders/easybuild/issues/364#issuecomment-316294667, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ADqZtbXZBZTLgOJBNIGHVhGLjsitMw5nks5sPa1OgaJpZM4Obbtn.

-- Dr. Alan O'Cais E-CAM Software Manager Juelich Supercomputing Centre Forschungszentrum Juelich GmbH 52425 Juelich, Germany

Phone: +49 2461 61 5213 Fax: +49 2461 61 6656 E-mail: a.ocais@fz-juelich.demailto:a.ocais@fz-juelich.de WWW: http://www.fz-juelich.de/ias/jsc/EN

Forschungszentrum Juelich GmbH 52425 Juelich Sitz der Gesellschaft: Juelich Eingetragen im Handelsregister des Amtsgerichts Dueren Nr. HR B 3498 Vorsitzender des Aufsichtsrats: MinDir Dr. Karl Eugen Huthmacher Geschaeftsfuehrung: Prof. Dr.-Ing. Wolfgang Marquardt (Vorsitzender), Karsten Beneke (stellv. Vorsitzender), Prof. Dr.-Ing. Harald Bolt, Prof. Dr. Sebastian M. Schmidt

maikenp commented 7 years ago

As far as I have understood, the steps to use your own produced toolchain to compile e.g. python would be the ones below.

Could you be so kind to have a look at my steps and comment if something is wrong and/or missing?

General note: there is surely something I am misunderstanding here. And: also used this page for instructions: https://github.com/easybuilders/easybuild/wiki/Compiler-toolchains

PREFIX=<location-of-your-easybuild-package> e.g. PREFIX=/cluster/software/VERSIONS/easybuild/3.1.2

  1. Create you custom toolchain file, (gomkl.py.txt) and place it in $PREFIX/toolchains

  2. Create your easyconfig file for the custom toolchain, Placed in $PREFIX/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/g/ (where g is the first letter of my toolchain). (gomkl-1.0.0.eb.txt)

  3. Create the toolchain to use for installing and building python using the custom toolchain example attached (python2.7.10-gomkl.py.txt).

  4. Create the easyconfig file for the python installation. Place it as in point 2, but now under folder p/ (Python-2.7.10-gomkl-1.0.0-bare.eb.txt)

  5. Do on custom toolchain: eb gomkl-1.0.0.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot -> I am here showing my exact command used for my system. -> this produces a modulefile which is placed in $PREFIX/software/EasyBuild/3.1.2/modules/

  6. Install python using the custom toolchain: module use $PREFIX/software/EasyBuild/3.1.2/modules module load gomkl-1.1.0 eb Python-2.7.10-gomkl-1.1.0-bare.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot --include-toolchains=$PREFIX/software/EasyBuild/3.1.2/modules/all/gomkl/1.0.0 -> this produces a modulefile which again is placed in $PREFIX/software/EasyBuild/3.1.2/modules/`

  7. More steps needed?

Question1 related to 1.):

Must I download the python module gompi and linalg in order to use it in my custom toolchain, or is this magically taken care of in the background? Right now the packages exist by name in my $PREFIX/toolchains folder, but they are empty (only init.py file).

Question2 related to 1.):

When I ask for gompi, and I do not currently have this installed in my system, will easybuild fix that in the background? I suppose I must tell easybuild where to pick up the source for this? However, if I understand the step-by-step guide correctly, the --robot option will try to resolve this for me?

Question3 related to 2.):

When asking for e.g. intelmkl and version 11.3 and intelfftw version 3.3.4, and I dont have this as a module already (not in module avail list), I seem to get an error

ERROR: Irresolvable dependencies encountered: imkl/11.3.3.210, FFTW/3.3.4

I suppose I then need to create a toolchain and hence module file for these also, specifying where the sources can be collected and so on. Or is there some settings I can issue in order for easybuild to fix this behind the scenes?

maikenp commented 7 years ago

Comment on the above, in https://github.com/easybuilders/easybuild/wiki/Compiler-toolchains there is a remark:

Important remark: the list of dependencies in the toolchain environment module should match the list of classes in the toolchain definition as implemented by the Python module in easybuild.toolchains. More specifically, the names of the dependency modules should match the NAME of the toolchain elements as specified in the easybuild.toolchains.* Python classes for them.

However, looking into e.g. the IntelMKL class, there is no NAME variable for the IntelMKL class. Likewise for the IntelFFTW class. So I am not sure what to use.

I tried to use the names I found in e.g. a easyconfig file for intelmkl of FFTW, but here I am not sure if I am choosing the correct name, as it is a bit cumbersome to get an overview of the exact one I would like. It is not straightforward to know if I am indeed picking the correct module name. E.g. in easybuild-easyconfigs/easybuild/easyconfigs/f/FFTW/FFTW-2.1.5-intel-2017a.eb I find name = 'FFTW'

ocaisa commented 7 years ago

For step 1 to actually work you need to add the command line option

--include-toolchains=$PREFIX/toolchains/*.py

whenever you plan on using the toolchain you defined (step 6, what you currently have there is incorrect, there are no .py toolchain definitions in $PREFIX/software/EasyBuild/3.1.2/modules/all/gomkl/1.0.0 this is just a module file ).

Step 3 is incorrect, you've already defined the toolchain you are going to use (which is gomkl)

maikenp commented 7 years ago

Replying/commenting on my own Question3: In http://easybuild.readthedocs.io/en/latest/Using_the_EasyBuild_command_line.html#robot-search-path it becomes clear that the name to use is the name of the actual easyconfig file.

maikenp commented 7 years ago

Replying/commenting on my own Question2: For this I will need to point to an existing easyconfig file. So if gompi with the correct version name exists as an eb file, then the robot option will install and build this on the fly.

maikenp commented 7 years ago

Hi again. I am now trying again, see my new easyconfig file (gomkl-2.0.0.eb.txt)

I have defined dependencies using the existing easyconfig files that can be found in easybuild/easyconfig https://github.com/easybuilders/easybuild-easyconfigs/tree/master/easybuild/easyconfigs

These are (as gomkl-2.0.0.eb shows): GCC-5.4.0-2.26 ifort-2016.3.210-GCC-5.4.0-2.26 #not sure if I need this one OpenMPI-1.10.3-GCC-5.4.0-2.26 imkl-11.3.3.210-iompi-2016.09-GCC-5.4.0-2.26

I am using this combination since I am basing my toolchain on the easyconfig file: https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/i/iomkl/iomkl-2017.01.eb but I dont want intel compiler for the openmpi, so I omitted this in the dependencies.

When running eb gomkl-1.0.0.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot I get the error:

ERROR: Build of /cluster/software/VERSIONS/easybuild/3.1.2/software/EasyBuild/3.1.2/lib/python2.7/site-packages/easybuild_easyconfigs-3.1.2-py2.7.egg/easybuild/easyconfigs/m/M4/M4-1.4.17-GCCcore-5.4.0.eb failed (err: 'build failed (first 300 chars): get_software_root software root for binutils was not found in environment')

See full log: easybuild-M4-1.4.17-20170720.130515.xXvxV.log.txt

What could be the problem?

Another issue:

If I use the option --include-toolchains=$PREFIX/toolchains which was suggested earlier on, I get the following: -bash-4.1$ eb gomkl-2.0.0.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --include-toolchains=$PREFIX/toolchains --robot ERROR: Failed to parse configuration options: "No files found using glob pattern '/cluster/software/VERSIONS/easybuild/3.1.2/toolchains'"

I have -bash-4.1$ ls $PREFIX/toolchains/ compiler fft gimklompi.py.old gomkl.py __init__.py linalg mpi python2.7.10-gimklompi.py.old python2.7.10-gomkl.py README README~

Yet another:

You say:

Step 3 is incorrect, you've already defined the toolchain you are going to use (which is gomkl)

But once I get to the point that I want to compile python with gomkl, how do I tell easybuild to do that if I am not supposed to use a toolchain and easyconfig file for this?

ocaisa commented 7 years ago

Sorry, it should have been --include-toolchains=$PREFIX/toolchains/gomkl.py.

You don't need ifort so you should remove it form your toolchain definition.

Did you modify the M4 easyconfig? It has a dependency on binutils so this would have to be built unless you have deleted it or commented it out.

To tell your python which toolchain to use, you do that in the easyconfig with toolchain = {'name': 'gomkl', 'version': '1.0.0'}

You may get a lot of failed dependencies since there are no easyconfigs for the python dependencies with your toolchain. If you really want to do this you will get a lot more mileage from:

eb --try-toolchain=gomkl,1.0.0 --robot Python-2.7.12-GCC-5.4.0-2.26-bare.eb

In general though it sounds like you need to do some more reading of the docs. As a beginner, defining your own toolchains is a bit of complex task until you have a deeper understanding of how things work. foss works great out of the box and is commonly used by a lot of people, that is a good place to get started.

maikenp commented 7 years ago

Thanks a lot for your reply. Yes, I will definitely need to read the documentation more carefully. Was just so hopeful to see a step-by-step guide (https://github.com/easybuilders/easybuild/wiki/Step-by-step-guide) to what I am trying to do, however, it was not complete, so I was hoping for something similar to get a kick-start.

Maiken

boegel commented 7 years ago

@maikenp The documentation at https://github.com/easybuilders/easybuild/wiki/Step-by-step-guide is very outdated, which partially explains the problems/questions popping up I suspect...

Unfortunately I don't see that being ported to readthedocs/updated any time soon...

As @ocaisa already mentioned: composing toolchains is not that trivial, there's a lot of gotchas left and right...

If your questions have been answered, please close this issue.

maikenp commented 7 years ago

It has not really been answered, but I will close the issue anyway. I suppose I had the wrong understanding of easybuild, that it was a tool also to easily build your own toolchains :)

boegel commented 7 years ago

@maikenp Which toolchain are you trying to compose, which components (compiler, MPI, libs)?

maikenp commented 7 years ago

It is to compile python using gcc, intel mkl, open mpi.

On 1 Sep 2017, at 11:50, Kenneth Hoste notifications@github.com<mailto:notifications@github.com> wrote:

@maikenphttps://github.com/maikenp Which toolchain are you trying to compose, which components (compiler, MPI, libs)?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/easybuilders/easybuild/issues/364#issuecomment-326539730, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AVKZEE9bY53NLPLtakAjXxOwvsEqFxcoks5sd9NkgaJpZM4Obbtn.

boegel commented 7 years ago

@maikenp We have toolchains like that, see iomkl, so there's no need to compose a custom toolchain?

Just creating easyconfigs for the particular mix of versions you need should be sufficient, you can use the ones in https://github.com/easybuilders/easybuild-easyconfigs/tree/master/easybuild/easyconfigs/i/iomkl as a starting point.

maikenp commented 7 years ago

Yes, I did use that as a starting point. And tried to include gcc instead of icc. See post from July 20:

Hi again. I am now trying again, see my new easyconfig file (gomkl-2.0.0.eb.txt)

I have defined dependencies using the existing easyconfig files that can be found in easybuild/easyconfig https://github.com/easybuilders/easybuild-easyconfigs/tree/master/easybuild/easyconfigs

These are (as gomkl-2.0.0.eb shows): GCC-5.4.0-2.26 ifort-2016.3.210-GCC-5.4.0-2.26 #not sure if I need this one OpenMPI-1.10.3-GCC-5.4.0-2.26 imkl-11.3.3.210-iompi-2016.09-GCC-5.4.0-2.26

I am using this combination since I am basing my toolchain on the easyconfig file: https://github.com/easybuilders/easybuild-easyconfigs/blob/master/easybuild/easyconfigs/i/iomkl/iomkl-2017.01.eb but I dont want intel compiler for the openmpi, so I omitted this in the dependencies.

When running eb gomkl-1.0.0.eb --modules-tool EnvironmentModulesC --module-syntax Tcl --robot I get the error:

However, the error with binutils is now solved. So, we will try again and get back to you if we still have problems!