Having trouble configuring case001

[8i161029]

I just followed install.md to build picongpu. As configuring case001 the terminal showed that:

cmake command: cmake -DCUDA_ARCH=sm_20 -DCMAKE_INSTALL_PREFIX=/home/lai/paramSets/case001 -DPIC_EXTENSION_PATH=/home/lai/paramSets/case001 /home/lai/src/picongpu -- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "5.5") -- Compiling as C++11... -- Debug version -- Boost version: 1.57.0 -- Found the following Boost libraries: -- program_options -- regex -- filesystem -- system -- thread -- math_tr1 -- Found CUDA: /usr/local/cuda (found suitable version "8.0", minimum required is "5.0") -- Boost version: 1.57.0 -- Found CUDA: /usr/local/cuda (found version "8.0") -- Could NOT find NVML (missing: NVML_LIBRARY) -- Boost version: 1.57.0 -- Found the following Boost libraries: -- program_options -- Found 'adios_config': /usr/bin/adios_config -- The directory provided by 'adios_config -d' does not exist: -- Could NOT find ADIOS (missing: ADIOS_LIBRARIES ADIOS_INCLUDE_DIRS) (Required is at least version "1.10.0") -- libSplash supports PARALLEL output -- Could NOT find Freetype (missing: FREETYPE_LIBRARY FREETYPE_INCLUDE_DIRS) CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: PNGwriter_LIBRARIES linked by target "picongpu" in directory /home/lai/src/picongpu/src/picongpu

-- Configuring incomplete, errors occurred! See also "/home/lai/build/CMakeFiles/CMakeOutput.log". See also "/home/lai/build/CMakeFiles/CMakeError.log".

I have installed CUDA , ADIOS ,and PNGwriter . I don't know why they cannot be found. Please help me finishing it.

[8i161029]

I check the bash_mpirun.tpl . Here is the content:

 .TBG_author=${MY_NAME:+--author \"${MY_NAME}\"}

# 4 gpus per node if we need more than 4 gpus else same count as TBG_tasks
.TBG_gpusPerNode=`if [ $TBG_tasks -gt 4 ] ; then echo 4; else echo $TBG_tasks; fi`

#number of cores per parallel node / default is 2 cores per gpu on k20 queue
TBG_coresPerNode="$(( TBG_gpusPerNode * 2 ))"

# use ceil to caculate nodes
TBG_nodes="$(( ( TBG_tasks + TBG_gpusPerNode -1 ) / TBG_gpusPerNode))"
## end calculations ##

echo 'Running program...'

cd !TBG_dstPath

export MODULES_NO_OUTPUT=1
#. ~/picongpu.profile
unset MODULES_NO_OUTPUT

#set user rights to u=rwx;g=r-x;o=---
umask 0027

mkdir simOutput 2> /dev/null
cd simOutput

mpirun --display-map -am !TBG_dstPath/tbg/openib.conf --mca mpi_leave_pinned 0 -x LD_LIBRARY_PATH -npernode !TBG_gpusPerNode -n !TBG_tasks !TBG_dstPath/picongpu/bin/cuda_memtest.sh

if [ $? -eq 0 ] ; then
  mpirun  -tag-output --display-map -am !TBG_dstPath/tbg/openib.conf --mca mpi_leave_pinned 0 -x LD_LIBRARY_PATH -npernode !TBG_gpusPerNode -n !TBG_tasks !TBG_dstPath/picongpu/bin/picongpu !TBG_author !TBG_programParams | tee output
fi

mpirun -npernode !TBG_gpusPerNode -n !TBG_tasks killall -9 picongpu 2>/dev/null || true

Sorry I don't know how to well present this on github...I only copy and paste.

[8i161029]

Besides, last few months I tried to build PIConGPU in my pc for practice. From this month I can try to build PIConGPU in the server in our lab. It is Tesla k80 and OS is centOS7. I make a check for MPI. Here is the log: locate libmpi.so:

/opt/mpich-3.2_install/lib/.libs/libmpi.so /opt/mpich-3.2_install/lib/.libs/libmpi.so.12 /opt/mpich-3.2_install/lib/.libs/libmpi.so.12.1.0 /opt/mpich_by_gcc4/lib/libmpi.so /opt/mpich_by_gcc4/lib/libmpi.so.12 /opt/mpich_by_gcc4/lib/libmpi.so.12.1.0 /usr/lib/mpich-3.2/lib/libmpi.so.12 /usr/lib/mpich-3.2/lib/libmpi.so.12.1.0 /usr/lib64/mpich-3.2/lib/libmpi.so.12 /usr/lib64/mpich-3.2/lib/libmpi.so.12.1.0 /usr/lib64/openmpi/lib/libmpi.so /usr/lib64/openmpi/lib/libmpi.so.12 /usr/lib64/openmpi/lib/libmpi.so.12.0.3 /usr/local/lib/libmpi.so /usr/local/lib/libmpi.so.12 /usr/local/lib/libmpi.so.12.1.0

find /usr/lib -name "libmpi.so*":

/usr/lib/mpich-3.2/lib/libmpi.so.12 /usr/lib/mpich-3.2/lib/libmpi.so.12.1.0

Are there any problems with MPI installation? And are install steps the same as I did on Ubuntu16.04? Thank you!

[ax3l]

For the bash_mpirun.tpl file: I guess you only have 1 GPU? In that case, set it accordingly in TBG_gpusPerNode (to 1) :-)

Your .cfg file is unchanged? (Can you post it?)

For the new system: ideally, let us discuss one issue per GitHub issue / thread so we don't mix up things. But so far, this system looks ok! Take care to link only against one version of MPI, maybe you want to deinstall the OpenMPI on it and only keep mpich 3 (or vice versa). Be aware that on this machine, a K80 is used which exposes to the user as if it would be 2 GPUs :)

[8i161029]

I am building PIConGPU in the new system now. I try to completely build it. When I was installing ADIOS, here was an error :

/usr/lib64/libpthread.so.0: error adding symbols: DSO missing from command line collect2: error: ld returned 1 exit status make[3]: *** [adios_list_methods] Error 1

Maybe I don't have some libraries. Could this error be solved? I hope I can completely build it. So I want to install all libraries if I can.

As for previous PIConGPU installation in my PC, my PC is repairing... so I maybe start that in 5 days.

Thank you!

[ax3l]

So far so excellent!

For a PC version, just use libSplash (HDF5) instead of ADIOS, they have the same features.

For older versions of adios, there has been the bug you reported when building it, try writing

LDFLAGS="-pthread" CFLAGS="-fPIC" ./configure ...

to fix this. (but as I said, libSplash/HDF5 is enough)

[8i161029]

Thank you! I installed ADIOS successfully. Then I try to install splash2txt . Here is the log:

[81i61029@messi build]$ cmake -DCMAKE_INSTALL_PREFIX=$PICSRC/src/tools/bin $PICSRC/src/tools/splash2txt -- The C compiler identification is GNU 4.8.5 -- The CXX compiler identification is GNU 4.8.5 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Setting build type to 'Debug' as none was specified. -- Building Debug -- Found MPI_C: /usr/local/lib/libmpi.so
-- Found MPI_CXX: /usr/local/lib/libmpicxx.so;/usr/local/lib/libmpi.so
-- Found HDF5: /home/81i61029/lib/hdf5/lib/libhdf5.so;/usr/lib64/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so
-- libSplash supports PARALLEL output -- Found Splash: /home/81i61029/lib/splash/lib/libsplash.a;/home/81i61029/lib/hdf5/lib/libhdf5.so;/usr/lib64/libz.so;/usr/lib64/libdl.so;/usr/lib64/libm.so;/usr/local/lib/libmpi.so;/usr/local/lib/libmpicxx.so;/usr/local/lib/libmpi.so (found suitable version "1.6.0", minimum required is "1.6.0") -- libSplash compiler definitions: -D_LARGEFILE_SOURCE;-D_LARGEFILE64_SOURCE;-D_BSD_SOURCE;-DENABLE_HDF5=1 -- Found 'adios_config': /home/81i61029/lib/adios/bin/adios_config -- ADIOS linker flags (unparsed): -L/home/81i61029/lib/adios/lib -ladios -L/usr/lib64 -lz -pthread -lm -- Found adios in /home/81i61029/lib/adios/lib/libadios.a -- Found z in /usr/lib64/libz.so -- Found m in /usr/lib64/libm.so -- Found ADIOS: /home/81i61029/lib/adios/lib/libadios.a;/usr/lib64/libz.so;/usr/lib64/libm.so (found suitable version "1.10.0", minimum required is "1.10.0") -- Boost version: 1.64.0 -- Found the following Boost libraries: -- program_options -- regex -- Configuring done -- Generating done -- Build files have been written to: /home/81i61029/build

Then I entered make. However, there were 2 errors:

[81i61029@messi build]$ make Scanning dependencies of target splash2txt [ 33%] Building CXX object CMakeFiles/splash2txt.dir/splash2txt.cpp.o /home/81i61029/src/picongpu/src/tools/splash2txt/splash2txt.cpp:221:5: warning:identifier ‘nullptr’ is a keyword in C++11 [-Wc++0x-compat] ITools *tools = nullptr; ^ /home/81i61029/src/picongpu/src/tools/splash2txt/splash2txt.cpp: In function ‘int main(int, char)’: /home/81i61029/src/picongpu/src/tools/splash2txt/splash2txt.cpp:221:21: error: nullptr’ was not declared in this scope ITools tools = nullptr; ^ make[2]: [CMakeFiles/splash2txt.dir/splash2txt.cpp.o] Error 1 make[1]: [CMakeFiles/splash2txt.dir/all] Error 2 make: [all] Error 2

Could you please help me fixing this? Thank you!

[8i161029]

In addition, I have some questions about steps of testing case001. Are the steps:

1. clone $HOME/src/picongpu/examples/LaserWakefield to $HOME/paramSets/case001
2. cd $HOME/paramSets/case001
3. pic-configure $HOME/paramSets/case001
4. make
5. make -j install

Do I need to move to $HOME/build to do any steps? Besides, what is $SCRATCH mentioned in USAGE.rst? And how to get it? And after step 5, is that I should try this command tbg -s qsub -c submit/0016gpus.cfg -t submit/hypnos/k20_profile.tpl $PICHOME/runs/testBatch01 ? Methods in USAGE.rst are a little different from previous guide install.md so I am confused.

Sorry about many questions. Thank you very much!

[ax3l]

you can skip splash2txt - the tool is outdated and will probably be removed soon or replaced by a general openPMD script #1544

[ax3l]

your steps are nearly correct, but you don't run pic-configure inside your input files but in a temporary build directory.

Do I need to move to $HOME/build to do any steps?

Yes, instead of 2. do: cd $HOME/build

See this section of our manual for the full details.

Besides, what is $SCRATCH mentioned in USAGE.rst? And how to get it?

Since a few weeks, that's also explained in the link above in the first section :)

[...] Second, decide where to store your output of simulations which needs to be placed on a high-bandwidth, large-storage file system which we will refer to as $SCRATCH.

For a first test you can also use your home directory $HOME as simulation directory $SCRATCH.

Sorry about many questions. Thank you very much!

Don't worry, that's normal. The files you mention are also actually part of our manual which you can find in its full glory under https://picongpu.readthedocs.io . The best thing might be if you read the first two chapters INSTALLATION and USAGE page by page.

And after step 5, is that I should try this command

The tool tbg is the last step which copies your input and executables to an output directory ("scratch") where it is then executed. You need to adjust it's parameters to only take one GPU (.cfg file) if you are running locally.

But before you get there, you will adjust your .cfg file accordingly (see the manual first). The final command will then likely look like:

#  interactive      use one GPU            run with mpirun         output directory
tbg -s bash -c submit/0001gpus.cfg -t submit/bash/bash_mpirun.tpl $HOME/runs/lwfa_001

[8i161029]

Hello, I have some questions. I found that the version of GCC in this server is 4.8.5. But my CUDA is version 8.0, so I need GCC version >5. Then I installed GCC-6.3.0 in $HOME/lib/gcc-6.3.0. However, I have problem setting environment variable... I already set this line in .bash_profile:

PATH=$HOME/lib/gcc-6.3.0/bin:$PATH:HOME/.local/bin:$HOME/bin

I check gcc --version and c++ --version Here are the output:

gcc (GCC) 6.3.0 Copyright (C) 2016 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

c++ (GCC) 6.3.0 Copyright (C) 2016 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

But as I configured case001, output showed that:

cmake command: cmake -DCUDA_ARCH=20 -DCMAKE_INSTALL_PREFIX=/home/81i61029/paramSets/case001 -DPIC_EXTENSION_PATH=/home/81i61029/paramSets/case001 /home/81i61029/src/picongpu/src/picongpu -- The C compiler identification is GNU 4.8.5 -- The CXX compiler identification is GNU 4.8.5 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/c++ -- Check for working CXX compiler: /usr/bin/c++ -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Looking for pthread.h -- Looking for pthread.h - found -- Looking for pthread_create -- Looking for pthread_create - not found -- Looking for pthread_create in pthreads -- Looking for pthread_create in pthreads - not found -- Looking for pthread_create in pthread -- Looking for pthread_create in pthread - found . . .

C compiler is still GNU 4.8.5. I think I set the path wrongly. Could you please teach me how to set up GCC environment variable?

Besides, I see many " Cmake warning" in configure output. Such like:

Cmake warning at /opt/cmake-3.8.0 - rc2/Modules/FindBoost . cmake : 761 (message): Imported targets not available for Boost version 106400 Call Stack (most recent call first) :
/opt/cmake-3.8.0 - rc2/Modules/FindBoost . cmake : 865 (_Boost_COMPONENT_DEPENDENCIES) /opt/cmake-3.8.0-rc2/Modules/FindBoost.cmake:1456 (_Boost_MISSING_DEPENDENCIES) CMakeLists.txt:152 (find_package)

Is this matter? Any problems between Cmake and Boost? Although that Cmake warning, configuring is completed. Maybe it is normal. I think I am getting there. Thank you for your help!

[ax3l]

I found that the version of GCC in this server is 4.8.5. But my CUDA is version 8.0, so I need GCC version >5.

That's not fully correct. CUDA 8.0 supports GCC up to 5.X (not at least GCC 5). The PIConGPU requirements are noted in our INSTALL.rst files (see the online version here) which are GCC 4.9-5.X.

So yes, at least for PIConGPU 0.3.0 (latest release) GCC 4.8 is too old and 6.X is too new (due to CUDA 8.0 limitations).

C compiler is still GNU 4.8.5. I think I set the path wrongly.

Just export the variables CC and CXX in your picongpu.profile. If you change compilers, make sure all your dependencies are build with the same compiler to avoid ABI conflicts such as symbol mismatches.

For example:

export CC=$HOME/lib/gcc-6.3.0/bin/gcc
export CXX=$HOME/lib/gcc-6.3.0/bin/g++

CMake should then report the correct compiler choice :)

[ax3l]

Imported targets not available for Boost version 106400

this just means your CMake is too old (or your boost version is too new). Just use Boost 1.61.0 instead, for example in case it causes troubles :) Those are "just" cmake hints chaining boost-internal dependencies together for some compiled boost libs, but usually it should still work since we request boost components rather explicitly in our CMakeLists.txt.

[8i161029]

Hello, I did locate picongpu.profile and here is the output:

/home/81i61029/src/picongpu/src/picongpu/submit/hypnos-hzdr/picongpu.profile.example /home/81i61029/src/picongpu/src/picongpu/submit/lawrencium-lbnl/picongpu.profile.example /home/81i61029/src/picongpu/src/picongpu/submit/pizdaint-cscs/picongpu.profile.example /home/81i61029/src/picongpu/src/picongpu/submit/taurus-tud/picongpu.profile.example /home/81i61029/src/picongpu/src/picongpu/submit/titan-ornl/picongpu.profile.example

I am not sure which picongpu.profile should I edit. I edited ...hypnos-hzdr/picongpu.profile.example and configured. But GNU was still 4.8.5. Could you please teach me which file should I edit? Thank you very much!

[ax3l]

The picongpu.profile script is a bash script that sets up your (bash) environment, such as paths to installed dependencies, LD_LIBRARY_PATH, etc. See this chapter of the manual.

We usually place it in $HOME/ and source it

. $HOME/picongpu.profile

first after opening a terminal or connecting to a machine.

This is just a way to avoid placing all modifications of a specific software (such as PIConGPU) in your $HOME/.profile (or $HOME/.bashrc) since they might influence other installed software.

[8i161029]

Thank you! Can I just add export CC=$HOME/lib/gcc-6.3.0/bin/gcc and export CXX=$HOME/lib/gcc-6.3.0/bin/g++ in ~/.bash_profile? Since this user home folder only installed PIConGPU. I set all the environment variables in this file. If picongpu.profile is necessary, is that I should clone ...hypnos-hzdr/picongpu.profile.example to $HOME and save as picongpu.profile then edit it and source it?

I tried add that two export commands in ~/.bash_profile and configured. Output showed that GNU is 6.3.0 and configured successfully. Then I make. Here is the error:

[81i61029@messi build]$ make Scanning dependencies of target picongpu-hostonly [ 5%] Building CXX object CMakeFiles/picongpu-hostonly.dir/include/ArgsParser.cpp.o g++: error: _LARGEFILE_SOURCE: No such file or directory g++: error: _LARGEFILE64_SOURCE: No such file or directory g++: error: _BSD_SOURCE: No such file or directory make[2]: [CMakeFiles/picongpu-hostonly.dir/include/ArgsParser.cpp.o] Error 1 make[1]: [CMakeFiles/picongpu-hostonly.dir/all] Error 2 make: *** [all] Error 2

g++ --version is:

g++ (GCC) 6.3.0 Copyright (C) 2016 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Maybe I wrongly installed gcc-6.3.0. I list my step to install gcc:

1. sudo yum install libmpc-devel mpfr-devel gmp-devel
2. wget http://mirrors.concertpass.com/gcc/releases/gcc-6.3.0/gcc-6.3.0.tar.bz2
3. tar jxvf gcc-6.3.0.tar.bz2
4. cd gcc-6.3.0
5. contrib/download_prerequisites
6. mkdir ../gcc-build
7. cd ../gcc-build
8. ../gcc-6.3.0/configure -v --enable-languages=c,c++ --disable-multilib --prefix=$HOME/lib/gcc-6.3.0
9. make -j
10. mkdir $HOME/lib/gcc-6.3.0
11. make install
12. add PATH=$HOME/lib/gcc-6.3.0/bin:$PATH:HOME/.local/bin:$HOME/bin in ~/.bash_profile

Could you please teach me where is the problem? So sorry to keep bothering you. I am so thankful for your help.

[ax3l]

Yes, you can also take the vars from one of the example profiles and put them directly in to your .bashrc or .profile.

Output showed that GNU is 6.3.0 [...]

Please stay with GCC 5.X since CUDA 8 does not support GCC >5: see my comment above

Note: You need to set and source your picongpu.profile (or .bashrc) variables before building the dependencies of PIConGPU, so all libs are build with the same compiler.

Side note for an alternative install method: if you can install nvidia-docker on your machine, the setup can be as easy as:

nvidia-docker run -it ax3l/picongpu

[8i161029]

I have installed gcc-5.2.0 in /usr/local/bin: g++ --version:

g++ (GCC) 5.2.0 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

c++ --version:

c++ (GCC) 5.2.0 Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

And also cloned ...hypnos-hzdr/picongpu.profile.example to $HOME and save as picongpu.profile Already added the two export commands: export CC=/usr/local/bin/gcc export CXX=/usr/local/bin/g++

in picongpu.profile then sourced it. (Soured ~/.bash_profile too since environment variables in it)

Output of configure showed that GNU is 5.2.0 and configured down. But as I entered make there was still errors:

Scanning dependencies of target picongpu-hostonly [ 5%] Building CXX object CMakeFiles/picongpu-hostonly.dir/include/ArgsParser.cpp.o g++: error: _LARGEFILE_SOURCE: No such file or directory g++: error: _LARGEFILE64_SOURCE: No such file or directory g++: error: _BSD_SOURCE: No such file or directory make[2]: [CMakeFiles/picongpu-hostonly.dir/include/ArgsParser.cpp.o] Error 1 make[1]: [CMakeFiles/picongpu-hostonly.dir/all] Error 2 make: *** [all] Error 2

I check cmake --version:

cmake version 3.8.0-rc2 CMake suite maintained and supported by Kitware (kitware.com/cmake).

and cc --version:

cc (GCC) 4.8.5 20150623 (Red Hat 4.8.5-11) Copyright (C) 2015 Free Software Foundation, Inc. This is free software; see the source for copying conditions. There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Is it the problem that cc is still GCC-4.8.5? how to change it? Thank you!

[ax3l]

I am not sure how to control the cc/c++ links on RHEL, on debian/ubuntu they are just links to /etc/alteratives/cc|c++ which in turn are links to the actual compiler that the package system uses. Those are not the same as the CXX/CC environment vars that control CMake, and your CMake output looks fine to me.

The problems seems to be more in the g++ error you get

g++: error: _LARGEFILE_SOURCE: No such file or directory

Did you recompile the dependencies, such as Boost, with GCC 5.2? Alternatively, it could indicate a broken install of g++ or unmatching std libs.

If you are having trouble to get a clean environment on your desktop, there are two alternatives I can recommend you to get an install done:

• use our nvidia-docker image

or

• install spack, setup a GCC 5.X compiler with it and install all dependencies:

  spack install cmake
  spack install cuda
  spack install openmpi@2.1.1
  spack install zlib@1.2.11
  spack install boost@1.61.0
  spack install pngwriter@0.6.0
  spack install libsplash@1.6.0 ^hdf5~fortran

[8i161029]

Thank you! Maybe I reinstall Boost and try again? What else libraries are dependencies? If this method does not work, I will try to install spack as you suggested.

[ax3l]

Sounds good! All software in INSTALL.rst are dependencies :)

[8i161029]

Hi. I thought there are many people work on this machine and the environment is complicated. So I installed spack. As I wanted to setup a GCC 5.4.0 compiler with it, I made a check: spack compiler info gcc@5.4.0 :

spack compiler info gcc@5.4.0 gcc@5.4.0: paths: cc = /home/81i61029/lib/gcc-5.4.0/bin/gcc cxx = /home/81i61029/lib/gcc-5.4.0/bin/g++ f77 = None fc = None modules = [] operating system = centos7

And I also checked original gcc in this machine: spack compiler info gcc@4.8.5:

gcc@4.8.5: paths: cc = /usr/bin/gcc cxx = /usr/bin/g++ f77 = /usr/bin/gfortran fc = /usr/bin/gfortran modules = [] operating system = centos7

I found that GCC-5.4.0 compiler doesn't support Fortran. Does it matter? How to set it? Thank you!

[ax3l]

When compiling your own gcc, make sure you add --enable-languages=c,c++,fortran on the configure command to also build gfortran. You will need it for compiling openmpi (unfortunately).

here is an example:

configure CFLAGS="-fPIC" CXXFLAGS="-fPIC" \
    --prefix=$gccDir \
    --build=x86_64-linux-gnu --enable-languages=c,c++,fortran \
--enable-shared --disable-multilib

then just add it properly to spack as you tried above, you will now also have gfortran for gcc@5.4.0 :)

[8i161029]

I reinstall GCC-5.4.0 and configured as your example(only changed prefix to $HOME/lib/gcc-5.4.0). I made a check: spack compiler info gcc@5.4.0:

spack compiler info gcc@5.4.0 gcc@5.4.0: paths: cc = /home/81i61029/lib/gcc-5.4.0/bin/gcc cxx = /home/81i61029/lib/gcc-5.4.0/bin/g++ f77 = None fc = None modules = [] operating system = centos7

I found it still did not support Fortran. I thought maybe I need to set it. I entered spack config edit compilers and changed f77 and fc path to /home/81i61029/lib/gcc-5.4.0/bin/gfortran. now the output of spack compiler info gcc@5.4.0 is :

gcc@5.4.0: paths: cc = /home/81i61029/lib/gcc-5.4.0/bin/gcc cxx = /home/81i61029/lib/gcc-5.4.0/bin/g++ f77 = /home/81i61029/lib/gcc-5.4.0/bin/gfortran fc = /home/81i61029/lib/gcc-5.4.0/bin/gfortran modules = [] operating system = centos7

Can I make this change? I am afraid that this change is wrong. If it is ok, I can install all dependencies. Right? Thank you!

[8i161029]

Besides, after install these libraries, how to run PIConGPU on spack? Are there any user manuals? Could you please teach me or give me some links? Thank you very much!

[ax3l]

In order to run PIConGPU with dependencies installed via spack, first set up spack. You already set your compilers in $HOME/.spack/compilers.yaml and do additionally add the following:

modules:
  enable::
    - lmod
  lmod:
    core_compilers:
      - 'gcc@5.4.0'
    hash_length: 0
    gcc:
      environment:
        set:
          CC: gcc
          CXX: g++
          FC: gfortran
          F90: gfortran
          F77: gfortran
    openmpi:
      environment:
        set:
          OMPI_MCA_mpi_leave_pinned: '0'

to $HOME/.spack/modules.yaml.

Now install lmod so you can use the generated packages easily.

Install Dependencies

This part is still a bit verbose and will soon become shorter. But don't despair, you only need to do it once :)

Now run the following once (for install):

spack install cmake
spack install cuda
spack install openmpi@2.1.1
spack install zlib@1.2.11
spack install boost@1.61.0
spack install pngwriter@0.6.0
spack install libsplash@1.6.0 ^hdf5~fortran

This command will take a while and installs all dependencies from source.

The last step is to create a $HOME/picongpu.profile file with the following content (please adjust the first three lines as necessary with your paths and the other lines as you need it, e.g. to add your name to the meta data of generated output):

# Modules #####################################################################
#
. /usr/share/lmod/5.8/init/bash
. $HOME/src/spack/share/spack/setup-env.sh
export MODULEPATH=$HOME/src/spack/share/spack/lmod/linux-centos7-x86_64/Core

# Core Dependencies (based on gcc 5.4.0)
module load cmake
module load boost
module load cuda
module load openmpi

# Plugins (optional)
module load zlib libpng freetype pngwriter
module load hdf5 libsplash

# either use libSplash or ADIOS for file I/O
#module load adios

# Environment #################################################################
#
export PICSRC=/home/`whoami`/src/picongpu
export PIC_PROFILE=$(cd $(dirname $BASH_SOURCE) && pwd)"/"$(basename $BASH_SOURCE)

# send me mails on job (b)egin, (e)nd, (a)bortion or (n)o mail
export MY_MAILNOTIFY="n"
export MY_MAIL="someone@example.com"
export MY_NAME="$(whoami) <$MY_MAIL>"

export PATH=$PATH:$PICSRC
export PATH=$PATH:$PICSRC/src/splash2txt/build
export PATH=$PATH:$PICSRC/src/tools/bin

export PYTHONPATH=$PICSRC/src/tools/lib/python:$PYTHONPATH

# "tbg" default options #######################################################
#   - interactive (bash + mpiexec)
export TBG_SUBMIT="bash"
export TBG_TPLFILE="etc/picongpu/bash/bash_mpiexec.tpl"

Using PIConGPU

Open a new terminal and run . ~/picongpu.profile once. You can now use PIConGPU with all it's environment as described here: https://picongpu.readthedocs.io/en/latest/usage/basics.html

[8i161029]

Thank you for your response! My compilers.yaml is in $HOME/.spack/linux/compilers.yaml and I cannot find modules.yaml Here is the output of find .spack :

[81i61029@messi ~]$ find $HOME/.spack /home/81i61029/.spack /home/81i61029/.spack/cache /home/81i61029/.spack/cache/providers /home/81i61029/.spack/cache/providers/.builtin-index.yaml.lock /home/81i61029/.spack/cache/providers/builtin-index.yaml /home/81i61029/.spack/linux /home/81i61029/.spack/linux/compilers.yaml

Can I create a modules.yaml in /home/81i61029/.spack/linux ?

And is the installation status of my GCC-5.4.0 okay? Because I made some change in compilers.yaml. output of spack compiler info gcc@5.4.0 :

gcc@5.4.0: paths: cc = /home/81i61029/lib/gcc-5.4.0/bin/gcc cxx = /home/81i61029/lib/gcc-5.4.0/bin/g++ f77 = /home/81i61029/lib/gcc-5.4.0/bin/gfortran fc = /home/81i61029/lib/gcc-5.4.0/bin/gfortran modules = [] operating system = centos7

Thank you!

[ax3l]

Yes just go ahead and create it :)

Your compilers.yaml looks good, otherwise we will know as soon as soon as openmpi compiles.

[8i161029]

Hi, I met some questions while installing lmod.

I met errors when I using Luarocks :

[81i61029@messi ~]$ luarocks install luaposix Installing https://luarocks.org/luaposix-34.0.1-3.src.rock Error: LuaRocks 2.4.1 bug (please report at https://github.com/keplerproject/luarocks/issues). ...61029/lib/luarocks/share/lua/5.1/luarocks/fs/lua.lua:784: attempt to call field 'mkdtemp' (a nil value) stack traceback: ...61029/lib/luarocks/share/lua/5.1/luarocks/fs/lua.lua:784: in function 'make_temp_dir' ...i61029/lib/luarocks/share/lua/5.1/luarocks/fetch.lua:74: in function 'fetch_url_at_temp_dir' ...i61029/lib/luarocks/share/lua/5.1/luarocks/fetch.lua:142: in function 'fetch_and_unpack_rock' ...i61029/lib/luarocks/share/lua/5.1/luarocks/build.lua:358: in function <...i61029/lib/luarocks/share/lua/5.1/luarocks/build.lua:352> (tail call): ? ...i61029/lib/luarocks/share/lua/5.1/luarocks/build.lua:407: in function <...i61029/lib/luarocks/share/lua/5.1/luarocks/build.lua:396> (tail call): ? (tail call): ? (tail call): ? C: in function 'xpcall' ...lib/luarocks/share/lua/5.1/luarocks/command_line.lua:180: in function 'run_command' ...cks/lib/luarocks/rocks/luarocks/2.4.1-1/bin/luarocks:33: in main chunk

Please teach me how to fix this.

Maybe I can install lua Filesystem (lfs) and luaposix by using Package Manager to avoid this problem. Could you please teach me how to get lua Filesystem (lfs) and luaposix by using Package Manager in CentOS? Thank you!

[ax3l]

Huh! Please report that bug with your operating system in https://github.com/keplerproject/luarocks/issues as the error states (or https://github.com/TACC/Lmod)

In the meantime, let us use "simple" environment modules.

sudo yum install environment-modules

Set

modules:
  tcl:
    hash_length: 0
    gcc:
      environment:
        set:
          CC: gcc
          CXX: g++
          FC: gfortran
          F90: gfortran
          F77: gfortran
    openmpi:
      environment:
        set:
          OMPI_MCA_mpi_leave_pinned: '0'

to $HOME/.spack/modules.yaml.

The first lines of your $HOME/picongpu.profile now need to read:

# Modules #####################################################################
#
. /usr/share/modules/init/bash
. $HOME/src/spack/share/spack/setup-env.sh
export MODULEPATH=$HOME/src/spack/share/spack/modules/linux-centos7-x86_64

[8i161029]

Thank you for your response! May I ask a question? Without this line

core_compilers:

• 'gcc@5.4.0'

in the amended modules.yaml, does spack still use gcc-5.4.0? During installing dependencies, I saw all dependencies installed in original compiler gcc-4.4.7. For example:

[81i61029@messi ~]$ spack install zlib@1.2.11 ==> zlib is already installed in /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/zlib-1.2.11-m6bccmouozijnhr2kkbobyxlvzp3f6dg ==> zlib@1.2.11 : marking the package explicit

And in /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64 I can only see gcc-4.4.7 Is it normal? Should I also find gcc-5.4.0 in this directory?

Thank you!

[8i161029]

I tried to change command: spack install zlib@1.2.11 %gcc@5.4.0. Now I can see folder gcc-5.4.0 in /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64 and folder zlib-1.2.11-ec535e2ikkpl7hd4y454t3yydjqorja6 in gcc-5.4.0. Should I make this change to all dependencies? After this does spack use gcc-5.4.0? Thank you!

[ax3l]

Yes, that's the right way to do it!

In order to spare always typing %gcc@5.4.0, just set this compiler as your default compiler in a new file in $HOME/packages.yaml:

packages:
  all:
    compiler: [gcc@5.4.0]

You can also keep the other compiler included but with less priority:

packages:
  all:
    compiler: [gcc@5.4.0, gcc@4.4.7]

The file in modules.yaml was a setting for lmod, which would have organized the compiled libs in a hierarchy "compiler -> MPI -> ..." for you. If you are using environment modules instead the module load might become a bit more longish but we will manage that or use or spack load instead.

[8i161029]

I met trouble during install dependencies after I changed default compiler. I could not find packages.yaml in $HOME So I added gcc@5.4.0 in $HOME/lib/spack/etc/spack/defaults/packages.yaml in front of original line. Here is the packages.yaml:

packages: all: compiler: [gcc@5.4.0, gcc, intel, pgi, clang, xl, nag] providers: awk: [gawk] blas: [openblas] daal: [intel-parallel-studio+daal] elf: [elfutils] golang: [gcc] ipp: [intel-parallel-studio+ipp] lapack: [openblas] mkl: [intel-parallel-studio+mkl] mpe: [mpe2] mpi: [openmpi, mpich] opencl: [pocl] openfoam: [openfoam-com, openfoam-org, foam-extend] pil: [py-pillow] scalapack: [netlib-scalapack]

Then I installed cmake, here is error:

==> Error: ProcessError: Command exited with status 11: './bootstrap' '--prefix=/home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-5.4.0/cmake-3.8.1-ankgfj3uhenjmia46zyod2a3a5rvehcy' '--parallel=64' '--no-system-jsoncpp' '--no-system-libs' '--no-qt-gui' '--' '-DCMAKE_USE_OPENSSL=ON' /home/81i61029/lib/spack/var/spack/repos/builtin/packages/cmake/package.py:128, in install: 93 def install(self, spec, prefix): 94 # Consistency check 95 self.validate(spec) 96
97 options = [ 98 '--prefix={0}'.format(prefix), 99 '--parallel={0}'.format(make_jobs)] 100 if spec.satisfies("@3.2:"): 101 options.append( 102 # jsoncpp requires CMake to build 103 # use CMake-provided library to avoid circular dependency 104 '--no-system-jsoncpp' 105 ) 106
107 if '+ownlibs' in spec: 108 # Build and link to the CMake-provided third-party libraries 109 options.append('--no-system-libs') 110 else: 111 # Build and link to the Spack-installed third-party libraries 112 options.append('--system-libs') 113
114 if '+qt' in spec: 115 options.append('--qt-gui') 116 else: 117 options.append('--no-qt-gui') 118
119 if '+doc' in spec: 120 options.append('--sphinx-html') 121 options.append('--sphinx-man') 122
123 if '+openssl' in spec: 124 options.append('--') 125 options.append('-DCMAKE_USE_OPENSSL=ON') 126
127 bootstrap = Executable('./bootstrap')

128 bootstrap(*options) 129
130 make() 131 if self.run_tests: 132 make('test') # some tests fail, takes forever 133 make('install') See build log for details: /tmp/81i61029/spack-stage/spack-stage-kV8UBz/cmake-3.8.1/spack-build.out

Is that I shouldn't change $HOME/lib/spack/etc/spack/defaults/packages.yaml? should I create a packages.yaml in $HOME? Thank you!

update: I did things I mentioned above. But it is still the same situation.

[ax3l]

No, don't adjust the $HOME/lib/spack/etc/spack/defaults/packages.yaml. Just add the lines to a new file in your $HOME/.spack/ dir.

[ax3l]

what's the output of /tmp/81i61029/spack-stage/spack-stage-kV8UBz/cmake-3.8.1/spack-build.out ? looks like building cmake 3.8.1 with gcc 5.4.0 seems to fail o.0

[ax3l]

If you already have an external cmake on your system and it's new enough, you can also just take that version. CMake is only needed as a tool.

Assuming your system already has a recent binary, say CMake 3.8.1 in /usr, then just add to your $HOME/.spack/packages.yaml:

packages:
  all:
    compiler: [gcc@5.4.0, gcc@4.4.7]
  cmake:
    paths:
      cmake@3.8.1%gcc@5.4.0 arch=linux-centos7-x86_64: /usr
    buildable: False

[ax3l]

If that fails, we just go with gcc@4.9.4 instead (compile from source, the rest is the same).

[8i161029]

Oh no. I created a new file packages.yaml in $HOME/.spack/linux. Then install cmake again. The output is the same. I am afraid that my wrongly installed gcc-5.4.0 before. So I reinstall gcc@5.4.0 by spack. Now my gcc@5.4.0 is in $HOME/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr Output of spack compiler info gcc@5.4.0:

gcc@5.4.0: paths: cc = /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr/bin/gcc cxx = /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr/bin/g++ f77 = /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr/bin/gfortran fc = /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr/bin/gfortran modules = [] operating system = centos7

I tried to install cuda. And it was successful. My problem now is I cannot install cmake by spack with gcc@5.4.0.

[8i161029]

Oh sorry I miss your message. Here is the output of /tmp/81i61029/spack-stage/spack-stage-kV8UBz/cmake-3.8.1/spack-build.out

==> './bootstrap' '--prefix=/home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-5.4.0/cmake-3.8.1-ankgfj3uhenjmia46zyod2a3a5rvehcy' '--parallel=64' '--no-system-jsoncpp' '--no-system-libs' '--no-qt-gui' '--' '-DCMAKE_USE_OPENSSL=ON'

CMake 3.8.1, Copyright 2000-2017 Kitware, Inc. and Contributors C compiler on this system is: /home/81i61029/lib/spack/lib/spack/env/gcc/gcc C++ compiler on this system is: /home/81i61029/lib/spack/lib/spack/env/gcc/g++ Makefile processor on this system is: gmake /home/81i61029/lib/spack/lib/spack/env/gcc/g++ is GNU compiler /home/81i61029/lib/spack/lib/spack/env/gcc/g++ has setenv /home/81i61029/lib/spack/lib/spack/env/gcc/g++ has unsetenv /home/81i61029/lib/spack/lib/spack/env/gcc/g++ does not have environ in stdlib.h /home/81i61029/lib/spack/lib/spack/env/gcc/g++ has stl wstring /home/81i61029/lib/spack/lib/spack/env/gcc/g++ has <ext/stdio_filebuf.h>

gmake: cmake' is up to date. /tmp/81i61029/spack-stage/spack-stage-kV8UBz/cmake-3.8.1/Bootstrap.cmk/cmake: /lib64/libstdc++.so.6: versionGLIBCXX_3.4.20' not found (required by /tmp/81i61029/spack-stage/spack-stage-kV8UBz/cmake-3.8.1/Bootstrap.cmk/cmake) /tmp/81i61029/spack-stage/spack-stage-kV8UBz/cmake-3.8.1/Bootstrap.cmk/cmake: /lib64/libstdc++.so.6: version `GLIBCXX_3.4.21' not found (required by /tmp/81i61029/spack-stage/spack-stage-kV8UBz/cmake-3.8.1/Bootstrap.cmk/cmake)

[ax3l]

With spack, the best thing is to install a compiler externally before starting with it. This can be done either manually, via system package manager or via a second spack instance, see this link: http://spack.readthedocs.io/en/latest/getting_started.html#build-your-own-compiler

[ax3l]

I tried to verify your problem with gcc 5.4.0 and cmake 3.8.1 and could not reproduce it. Seems really like your gcc 5.4.0 was not installed properly? Did you bootstrap it (make bootstrap)? It looks like it is still using the wrong libstdc++ of your system

[ax3l]

Your spack compiler info gcc@5.4.0: looks wrong. It points to 4.4.7 paths instead of your install for gcc 5.4.

Can you post your $HOME/spack/compilers.yaml again, please? Make sure there are not several of those in your $HOME/.spack subfolders

[8i161029]

Ok, thank you! I think I wrongly installed GCC again... Now I try to reinstall gcc 5.4.0 with second spack instance. Is that I need to install a new spack and get gcc@5.4.0 by it, then dospack compiler find on original spack?

[ax3l]

When in doubt, please also read the getting started sections of spack very carefully again, just to make sure you did not miss a step and understand what I am posting :)

[ax3l]

This short getting started section in the spack manual also tells you, that you can just add the new compiler like this:

spack compiler find /home/81i61029/lib/spack-tools/opt/...path...to...gcc5.4.0

while $HOME/lib/spack-tools is the second spack instance that you use solely for installing gcc 5.4.0

[8i161029]

Thank you a lot! I am installing gcc 5.4.0 by second spack instance. May I ask what is the function of make bootstrap? I never did it while I install gcc-5.4.0.

Now my steps to install a new gcc@5.4.0 are:

1. get another spack in $HOME/spack-tools
2. use the new spack to do spack install gcc@5.4.0
3. back to the original spack and do spack compiler find $HOME/spack-tools/spack/opt/...path...to...gcc5.4.0

Are there any things need to do but I forget?

[ax3l]

Thank you a lot! I am installing gcc 5.4.0 by second spack instance.

that sounds good!

Are there any things need to do but I forget?

yes, please do not forget to replace the ...path...to...gcc5.4.0 in the path in step 3 :)

May I ask what is the function make bootstrap? I never did it while I install gcc-5.4.0.

You won't need that now, but usually one installs gcc similar to this and could add make bootstrap before make install to build the compiler again with itself to get rid of all external dependencies...

[8i161029]

So if I install gcc@5.4.0 by second spack, I don't need to worry about bootstrap? Because I see the error start from ./bootstrap while I install cmake in my comment above.

After I installed gcc@5.4.0 by second spack, do I need to delete the

gcc@5.4.0: paths: cc = /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr/bin/gcc cxx = /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr/bin/g++ f77 = /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr/bin/gfortran fc = /home/81i61029/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7/gcc-5.4.0-32u3h5tpy7z5xpznwcaclgiko7zdjurr/bin/gfortran modules = [] operating system = centos7

in compilers.yaml? Or just do the command spack compiler find? Thank you!

[ax3l]

So if I install gcc@5.4.0 by second spack, I don't need to worry about bootstrap?

exactly.

After I installed gcc@5.4.0 by second spack, do I need to delete the ...

yes, just remove the $HOME/.spack/compilers.yaml to start clean and then run the spack compiler find ... which will add it again and properly formatted for you.

[8i161029]

I finally installed cmake successfully !

First, After installing new gcc@5.4.0, I did spack install cmake. But it failed... so I check what happened. And I found there are many lines of Warning: Module file already exists : skipping creation and some errors mentioned about the directory dotkit So I moved to the directories modules and dotkit in $HOME/lib/spack/share/spack/ and deleted all files in these two directories. Then I also removed the directory $HOME/lib/spack/opt/spack/linux-centos7-x86_64/gcc-4.4.7 Eventually I entered spack install cmake, and it was successful ! I don't know which move I did that made me finish it. Just let you know I did these things.

Thank you!

update: All dependencies installed successfully! I will set picongpu.profile tomorrow. Thank you a lot!