search

yambo-code / yambo

This is the official GPL repository of the yambo code
http://www.yambo-code.eu/
GNU General Public License v2.0
91 stars 35 forks source link

issues building external libraries (particularly iotk) #24

Open jamie-arcc opened 4 years ago

jamie-arcc commented 4 years ago

there seem to be issues building the external libraries during make all. iotk first seems to not create its destination include directory before copying a file into it. manually creating the directory and running it again gets past that, but then it fails to find make.sys when its makefile is run.

[jschnaitter@ec59 yambo-4.4.0]$ make all
for target in yambo ypp a2y p2y  yambo_ph ypp_ph yambo_rt ypp_rt yambo_nl ypp_nl yambo_kerr      ; do rm -f "/groups/arcc/temp/yambo-4.4.0/bin/$target" ; make $target; if test ! -f "/groups/arcc/temp/yambo-4.4.0/bin/$target" ; then echo "$target build failed"; break;fi ; done
make[1]: Entering directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0'
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/iotk-y1.2.2.tar.gz’ and ‘lib/archive/iotk-y1.2.2.tar.gz’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/keep-extlibs-stamp’ and ‘lib/archive/keep-extlibs-stamp’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/libxc-2.2.3.tar.gz’ and ‘lib/archive/libxc-2.2.3.tar.gz’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/Makefile’ and ‘lib/archive/Makefile’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/Makefile.loc’ and ‘lib/archive/Makefile.loc’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/package.list’ and ‘lib/archive/package.list’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/config/missing’ and ‘config/missing’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/libxc/Makefile.loc’ and ‘lib/libxc/Makefile.loc’ are the same file

>>>[Making libxc]<<<
make[2]: Entering directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0/lib/libxc'
make[2]: Nothing to be done for `all'.
make[2]: Leaving directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0/lib/libxc'
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/iotk-y1.2.2.tar.gz’ and ‘lib/archive/iotk-y1.2.2.tar.gz’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/keep-extlibs-stamp’ and ‘lib/archive/keep-extlibs-stamp’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/libxc-2.2.3.tar.gz’ and ‘lib/archive/libxc-2.2.3.tar.gz’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/Makefile’ and ‘lib/archive/Makefile’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/Makefile.loc’ and ‘lib/archive/Makefile.loc’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/archive/package.list’ and ‘lib/archive/package.list’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/config/missing’ and ‘config/missing’ are the same file
cp: ‘/groups/arcc/temp/yambo-4.4.0/lib/iotk/Makefile.loc’ and ‘lib/iotk/Makefile.loc’ are the same file
/bin/sh: line 1: test: /groups/arcc/temp/yambo-4.4.0/lib/iotk/make_iotk.inc: binary operator expected

>>>[Making iotk]<<<
make[2]: Entering directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0/lib/iotk'
if test -d iotk ; then \
( cd iotk;  make loclib_only ; make iotk.x ) ; fi
make[3]: Entering directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0/lib/iotk/iotk'
Makefile:2: ../make.sys: No such file or directory
make[3]: *** No rule to make target `../make.sys'.  Stop.
make[3]: Leaving directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0/lib/iotk/iotk'
make[3]: Entering directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0/lib/iotk/iotk'
Makefile:2: ../make.sys: No such file or directory
make[3]: *** No rule to make target `../make.sys'.  Stop.
make[3]: Leaving directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0/lib/iotk/iotk'
make[2]: *** [package-ready-stamp] Error 2
make[2]: Leaving directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0/lib/iotk'
make[1]: *** [ext-libs] Error 2
make[1]: Leaving directory `/lustre/fs0/groups/arcc/temp/yambo-4.4.0'
yambo build failed

I think it might be an issue with the test statements. at one point /bin/sh was complaining that test was expecting a binary operator, but now it isn't.

it also seems that this makefile breaks when run in parallel. If I try to run make with -j, libxc fails to build, claiming it cannot find xc.h.

ikirker commented 3 years ago

I'm also having these problems.

For the binary operator expected error in particular, I poked through make --debug=j, and at the point that gives this error and all those cp bits above >>>[Making iotk]<<<, it does this (prettified, the original is all on two lines):

if test "yes" = yes ; then
    LIBS2DO="iotk" ; \
        DIR2GO="lib" ;
    VPATH="/tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib" ;
    if test ! -d "lib/archive" ; then 
        mkdir -p "lib/archive" ;
    fi ;                  
    if [ "/tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3" != "/tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018" ] ; then
        cp /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/archive/* lib/archive;
        cp /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/config/missing config/ ;
    fi ;                                        
    for ldir in $LIBS2DO; do                       
        if test ! -f "/tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018/lib/lib$ldir.a" ; then
            if [ "/tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3" != "/tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018" ] ; then
                if test ! -d "$DIR2GO/$ldir" ; then                                    
                    mkdir -p "$DIR2GO/$ldir" ;                                                         
                fi ;                                                                                            
                if test -e $VPATH/$ldir/Makefile.loc; then                                                               
                    cp $VPATH/$ldir/Makefile.loc $DIR2GO/$ldir/ ;                                                                        
                fi ;                                                                                                                              
                if test -e $VPATH/$ldir/Makefile.lib; then                                                                                                 
                    cp $VPATH/$ldir/Makefile.lib $DIR2GO/$ldir/ ;                                                                                                          
                fi ;
                if test -e $VPATH/$ldir/*inc*;        then                                                                                                                          
                    cp $VPATH/$ldir/*inc* $DIR2GO/$ldir/ ;                                                                                                                                   
                fi ;
            fi ;                                                                                                                                                                                             
            echo " " ;
            echo ">>>[Making $ldir]<<<" ;                                                                                                                                                                                   cd $DIR2GO/$ldir ;
            cp Makefile.loc Makefile ;                                                                                                                                                                                      make VPATH=$VPATH/$ldir || exit "$?" ;
            cd ../../ ;                                                                                                                                                                                                 fi        
    done;                                                                                                                                                                                                       fi

This is the relevant part:

if test -e $VPATH/$ldir/*inc*;

This expands to two files, not the single file that seems to be expected:

$ ls /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk/*inc*
/tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk/make_iotk.inc  /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk/make_iotk.inc.in

So you get:

$ test -e /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk/make_iotk.inc  /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk/make_iotk.inc.in
-bash: test: /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk/make_iotk.inc: binary operator expected

You can delete the make_iotk.inc.in to get past that problem -- I assume it's not needed once you've generated the make_iotk.inc file -- but then I hit a problem where the iotk build tries to copy a file to a directory it hasn't created yet:

>>>[Making iotk]<<<
make[2]: Entering directory `/tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk'
if test -d iotk ; then ( cd iotk;  \
    if test -e /tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018/lib/iotk/make.sys ; then rm /tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018/lib/iotk/make.sys ; fi ; \
        if test -e /tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018/lib/iotk/make_iotk.inc ; then \
           cp /tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018/lib/iotk/make_iotk.inc /tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018/lib/iotk/make.sys ; \
        fi ; \
        if test -e /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk/iotk_specials.h ; then \
           cp /tmp/yambo-build.mWF9pPNjGQ/yambo-4.5.3/lib/iotk/iotk_specials.h /tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018/lib/iotk/iotk/include ; \
        fi ) ; \
fi
cp: cannot create regular file ‘/tmp/yambo-test-prefix.ZGVnLoVoTL/yambo/4.5.3/intel-2018/lib/iotk/iotk/include’: No such file or directory
sangallidavide commented 3 years ago

We could change 

    if test -e $$VPATH/$$ldir/*inc*;        then cp $$VPATH/$$ldir/*inc* $$DIR2GO/$$ldir/ ;        fi ;\

with

    for file in $VPATH/$ldir/*inc*; do  cp $$file $$DIR2GO/$$ldir/ ;   done ;\

would that work?

ikirker commented 3 years ago

You probably still want the test inside the loop, or it'll give you problems if the glob doesn't match anything.

E.g.:

$ for dud_value in *this_does_not_match_any_files*; do echo "$dud_value"; done
*this_does_not_match_any_files*
sangallidavide commented 3 years ago

Try2

    count=`ls -1 $$VPATH/$$ldir/*inc* 2>/dev/null | wc -l` ;\
    if [ $$count != 0 ] ; then  cp $$VPATH/$$ldir/*inc*  $$DIR2GO/$$ldir/ ;   fi ;\
ikirker commented 3 years ago

That would work, I think. (Assuming the dependencies on the extra commands isn't a problem.)