search

gms-bbg / gamess-issues

GAMESS issue tracking
7 stars 1 forks source link

Trouble porting to an IBM Power8 platform running Linux #15

Open tomoppe opened 4 years ago

tomoppe commented 4 years ago

Dear Sir:

I am having problems porting GAMESS (September 30, 2019 R2 Public Release) to Hokulea, an IBM Power8 platform at MHPCC that runs Linux.

hokulea02> uname -a Linux hokulea02.mhpcc.hpc.mil 3.10.0-957.41.1.el7.ppc64le #1 SMP Fri Nov 15 11:53:55 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

I have compiled all files at "-O0" and used "blas.o" instead of any optimized BLAS/LAPACK library such as ESSL. I have tried all the "ifdef" choices in "zunix.c", that is

ifdef AXP64 no

ifdef CRAY no

ifdef CRAYX1 no

ifdef CRAYXD1 no

ifdef CRAYXT3 no

ifdef FUJITSU no

ifdef IBM32 no

ifdef IBM64 no

ifdef IBMBG no

ifdef IBMPPC64 no

ifdef LINUX32 no

ifdef LINUX64 no

ifdef NECSX no

ifdef SGI32 no

ifdef SGI64 no

ifdef SUN32 no

ifdef SUN64 no

ifdef WINDOWS32 no

ifdef WINDOWS64 no

ifdef WINTEL no

The "no" indicates I tried it and the 47 tests in the regression suite do not all pass. About half the tests pass for those choices for which the link step "lked" is successful. 24 Passed 23 Failed 18 tests terminated abnormally.

I am not using LAPI (still works?) but a pure MPI build using spectrum-mpi. My loaded modules are:

hokulea02> module list Currently Loaded Modulefiles: 1) pbs/18.1.4 2) xlc/16.1.1.3 3) xlf/16.1.1.3 4) spectrum-mpi/10.3.0.1

I set OBJECT_MODE=64 for all compilations and linking. I am using the IBM XL compilers, version 16.1.1.3.

Do you have any advice for porting GAMESS to such a platform?

Thank you.

By the way, in the text below, shouldn't it be ACML instead of AMCL? AMD Core Math Library.

Tom Oppe, ERDC DSRC

Make sure these boxes are checked before submitting your issue - thank you!

colleeneb commented 4 years ago

This has worked successfully on Summit (IBM Power9 + V100 platform) before, so we should be able to get it to work here, too. I think we always used ESSL, though. Could you provide your install.info so we can compare to what we've used on Summit?

It would also help to know what kind of errors you're getting. Could you provide some of the error outputs when it terminates abnormally? I'm curious about why you see 18 abnormal terminations.

Also: You are correct about ACML, we'll fix this.

colleeneb commented 4 years ago

I just double checked that the tests passed on Summit. I had to make one change in lked (not in the public version yet, but it's in the development branch, so it will be there for the next public release):

diff --git a/lked b/lked
index 7dc7724..40f0f85 100755
--- a/lked
+++ b/lked
@@ -351,7 +351,7 @@ if ($TARGET == ibm64) then
    set BLAS='blaswrap.o'
 # also adding in norm2 since XLF2008 doesn't support norm2 right now.
    if ($GMS_FORTRAN == xlf) then
-       set BLAS='$BLAS norm2.o'
+       set BLAS="$BLAS norm2.o"
    endif
    set VECTOR=' '
    set QUICHE='zunix.o'

I also needed to change most of the #!/bin/csh to #!/bin/csh -f, but this might be specific to Summit.

All the tests passed in serial (we don't expect all to pass in parallel), so it is still working there. I put my install.info below. This is with ESSL, I haven't tried just blas.o. After the config, I just compiled with make ddi, make modules, make -j8.

bash-4.2$ cat install.info
#!/bin/csh

#   Compilation configuration for GAMESS
#   Generated on login1
#   Generated at Sun Jan 19 12:58:39 EST 2020

#                GAMESS Paths                  #
setenv GMS_PATH            /autofs/nccs-svm1_home1/bertoni/summit/gamess
setenv GMS_BUILD_DIR       /autofs/nccs-svm1_home1/bertoni/summit/gamess

#                Machine Type                  #
setenv GMS_TARGET          ibm64

#            FORTRAN Compiler Setup            #
setenv GMS_FORTRAN         xlf
setenv GMS_XL_PATH  /sw/summit/xl/16.1.1-3

#         Mathematical Library Setup           #
setenv GMS_MATHLIB         essl
setenv GMS_MATHLIB_PATH    /sw/summit/essl/6.1.0-2/essl/6.1
setenv GMS_AMD_BLAS_PATH
#         parallel message passing model setup
setenv GMS_DDI_COMM        mpi
setenv GMS_MPI_LIB         spectrum
setenv GMS_MPI_PATH        /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/xl-16.1.1-3/spectrum-mpi-10.3.0.1-20190611-aqjt3jo53mogrrhcrd2iufr435azcaha

#   Michigan State University Coupled Cluster  #
setenv GMS_MSUCC           false

# Please match any manual changes to the       #
# GMS_MSUCC flag in /autofs/nccs-svm1_home1/bertoni/summit/gamess/Makefile
# before running make                          #

#         LIBCCHEM CPU/GPU Code Interface      #
setenv GMS_LIBCCHEM        false

#      Intel Xeon Phi Build: none/knc/knl      #
setenv GMS_PHI             none

#         Shared Memory Type: sysv/posix       #
setenv GMS_SHMTYPE         sysv

#      GAMESS OpenMP support: true/false       #
setenv GMS_OPENMP          false

# Please match any manual changes to the       #
# GMS_OPENM flag in /autofs/nccs-svm1_home1/bertoni/summit/gamess/Makefile
# before running make                          #

#             Singularity Setup                #
setenv GMS_CONTAINER
setenv GMS_CONTAINER_PATH  /autofs/nccs-svm1_home1/bertoni/summit/gamess

################################################
# Added any additional environmental variables #
# or module loads below if needed.             #
################################################
setenv GMS_FPE_FLAGS       ''
tomoppe commented 4 years ago

Dear Ms. Bertoni,

Thank you very much for your help.

The attached file gives details on a compilation similar to what you did (make ddi, make modules, make). I first made the changes you recommended. The link step failed because it could not find "norm2.o". In "compall", the "norm2.src" file is compiled only if ($GMS_OPENMP == true) but $GMS_OPENMP is false in the "install.info" file. So I just did:

./comp norm2 lked

and got a successful link. The 47 regression tests all passed with this executable.

I am comparing your compilation and linking options to mine to see why I am having the problem with half the regression tests failing. Perhaps it is because you included "-gfortran" in your linker options and I did not. I will send you a message if I figure anything out.

Tom Oppe

-----Original Message----- From: Colleen Bertoni [mailto:notifications@github.com] Sent: Sunday, January 19, 2020 12:47 PM To: gms-bbg/gamess-issues gamess-issues@noreply.github.com Cc: Oppe, Thomas C ERDC-RDE-ITL-MS Contractor Thomas.C.Oppe@erdc.dren.mil; Author author@noreply.github.com Subject: Re: [gms-bbg/gamess-issues] Trouble porting to an IBM Power8 platform running Linux (#15)

I just double checked that the tests passed on Summit. I had to make one change in lked (not in the public version yet, but it's in the development branch, so it will be there for the next public release):

diff --git a/lked b/lked index 7dc7724..40f0f85 100755 --- a/lked +++ b/lked @@ -351,7 +351,7 @@ if ($TARGET == ibm64) then set BLAS='blaswrap.o'

also adding in norm2 since XLF2008 doesn't support norm2 right now.

if ($GMS_FORTRAN == xlf) then

I also needed to change most of the #!/bin/csh to #!/bin/csh -f, but this might be specific to Summit.

All the tests passed in serial (we don't expect all to pass in parallel), so it is still working there. I put my install.info below. This is with ESSL, I haven't tried just blas.o. After the config, I just compiled with make ddi, make modules, make -j8.

bash-4.2$ cat install.info

!/bin/csh

Compilation configuration for GAMESS

Generated on login1

Generated at Sun Jan 19 12:58:39 EST 2020

GAMESS Paths

setenv GMS_PATH /autofs/nccs-svm1_home1/bertoni/summit/gamess setenv GMS_BUILD_DIR /autofs/nccs-svm1_home1/bertoni/summit/gamess

Machine Type

setenv GMS_TARGET ibm64

FORTRAN Compiler Setup

setenv GMS_FORTRAN xlf setenv GMS_XL_PATH /sw/summit/xl/16.1.1-3

Mathematical Library Setup

setenv GMS_MATHLIB essl setenv GMS_MATHLIB_PATH /sw/summit/essl/6.1.0-2/essl/6.1 setenv GMS_AMD_BLAS_PATH

parallel message passing model setup

setenv GMS_DDI_COMM mpi setenv GMS_MPI_LIB spectrum setenv GMS_MPI_PATH /autofs/nccs-svm1_sw/summit/.swci/1-compute/opt/spack/20180914/linux-rhel7-ppc64le/xl-16.1.1-3/spectrum-mpi-10.3.0.1-20190611-aqjt3jo53mogrrhcrd2iufr435azcaha

Michigan State University Coupled Cluster

setenv GMS_MSUCC false

Please match any manual changes to the

GMS_MSUCC flag in /autofs/nccs-svm1_home1/bertoni/summit/gamess/Makefile

before running make

LIBCCHEM CPU/GPU Code Interface

setenv GMS_LIBCCHEM false

Intel Xeon Phi Build: none/knc/knl

setenv GMS_PHI none

Shared Memory Type: sysv/posix

setenv GMS_SHMTYPE sysv

GAMESS OpenMP support: true/false

setenv GMS_OPENMP false

Please match any manual changes to the

GMS_OPENM flag in /autofs/nccs-svm1_home1/bertoni/summit/gamess/Makefile

before running make

Singularity Setup

setenv GMS_CONTAINER setenv GMS_CONTAINER_PATH /autofs/nccs-svm1_home1/bertoni/summit/gamess

################################################

Added any additional environmental variables

or module loads below if needed.

################################################ setenv GMS_FPE_FLAGS ''

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub Blockedhttps://github.com/gms-bbg/gamess-issues/issues/15?email_source=notifications&email_token=AMLWS3GXYXMCYMM4M3MJUI3Q6SNZPA5CNFSM4KIY5ARKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKZJ4A#issuecomment-576034032Blocked , or unsubscribe Blockedhttps://github.com/notifications/unsubscribe-auth/AMLWS3EMLNVAT7PSJBI3EEDQ6SNZPANCNFSM4KIY5ARABlocked . Blockedhttps://github.com/notifications/beacon/AMLWS3CH7TA675WE6B3FKZLQ6SNZPA5CNFSM4KIY5ARKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJKZJ4A.gifBlocked

GAMESS version: September 30, 2019 R2 Public Release

Changes to distribution:

  1. lked, line 354

    changed set BLAS='$BLAS norm2.o' to set BLAS="$BLAS norm2.o"

  2. Changed "#!/bin/csh" to "#!/bin/csh -f" in

    config ddi/compddi compall comp lked

hokulea02> uname -a Linux hokulea02.mhpcc.hpc.mil 3.10.0-957.41.1.el7.ppc64le #1 SMP Fri Nov 15 11:53:55 UTC 2019 ppc64le ppc64le ppc64le GNU/Linux

hokulea02> module list Currently Loaded Modulefiles: 1) pbs/18.1.4 3) xlf/16.1.1.3 2) xlc/16.1.1.3 4) spectrum-mpi/10.3.0.1

hokulea02> xlf -qversion IBM XL Fortran for Linux, V16.1.1 (5725-C75, 5765-J15) Version: 16.01.0001.0003

hokulea02> xlc -qversion IBM XL C/C++ for Linux, V16.1.1 (5725-C73, 5765-J13) Version: 16.01.0001.0003

"install.info"

!/bin/csh

Compilation configuration for GAMESS

Generated on hokulea02.mhpcc.hpc.mil

Generated at Sun Jan 19 20:06:22 HST 2020

GAMESS Paths

setenv GMS_PATH /gpfs/scratch/oppe/ABTP/app/gamess setenv GMS_BUILD_DIR /gpfs/scratch/oppe/ABTP/app/gamess

Machine Type

setenv GMS_TARGET ibm64

FORTRAN Compiler Setup

setenv GMS_FORTRAN xlf setenv GMS_XL_PATH /opt/ibm/16.1.1.3

Mathematical Library Setup

setenv GMS_MATHLIB essl setenv GMS_MATHLIB_PATH /opt/ibmmath/essl/6.1 setenv GMS_AMD_BLAS_PATH

parallel message passing model setup

setenv GMS_DDI_COMM mpi setenv GMS_MPI_LIB spectrum setenv GMS_MPI_PATH /gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1

Michigan State University Coupled Cluster

setenv GMS_MSUCC false

Please match any manual changes to the

GMS_MSUCC flag in /gpfs/scratch/oppe/ABTP/app/gamess/Makefile

before running make

LIBCCHEM CPU/GPU Code Interface

setenv GMS_LIBCCHEM false

Intel Xeon Phi Build: none/knc/knl

setenv GMS_PHI none

Shared Memory Type: sysv/posix

setenv GMS_SHMTYPE sysv

GAMESS OpenMP support: true/false

setenv GMS_OPENMP false

Please match any manual changes to the

GMS_OPENM flag in /gpfs/scratch/oppe/ABTP/app/gamess/Makefile

before running make

Singularity Setup

setenv GMS_CONTAINER
setenv GMS_CONTAINER_PATH /gpfs/scratch/oppe/ABTP/app/gamess

################################################

Added any additional environmental variables

or module loads below if needed.

################################################ setenv GMS_FPE_FLAGS ''

Output from "make ddi"

/gpfs/scratch/oppe/ABTP/app/gamess/ddi/compddi Building the Distributed Data Interface library started at Sun Jan 19 20:07:14 HST 2020

Compiling for machine type ibm64, using communication model mpi, System V shared memory option set true, with maxima of 168 processors/node and 4608 nodes.

This compilation should produce the DDI library libddi.a

The MPI-1 software library to be used is spectrum, with the include file 'mpi.h' at /gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include

Attempting to remove binary files from previous DDI build... rm: No match.

Beginning the DDI compilation at Sun Jan 19 20:07:14 HST 2020

Compiling common object: soc_create.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o soc_create.o -c soc_create.c unset echo Finished compiling: soc_create.o

Compiling common object: std_system.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o std_system.o -c std_system.c unset echo Finished compiling: std_system.o

Compiling common object: tcp_sockets.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o tcp_sockets.o -c tcp_sockets.c unset echo Finished compiling: tcp_sockets.o

Compiling common object: debug.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o debug.o -c debug.c unset echo Finished compiling: debug.o

Compiling common object: parse_node_args.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o parse_node_args.o -c parse_node_args.c unset echo Finished compiling: parse_node_args.o

Compiling DDI object: ddi.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi.o -c ddi.c endif unset echo Finished compiling: ddi.o

Compiling DDI object: ddi_send.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_send.o -c ddi_send.c endif unset echo Finished compiling: ddi_send.o

Compiling DDI object: ddi_recv.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_recv.o -c ddi_recv.c endif unset echo Finished compiling: ddi_recv.o

Compiling DDI object: ddi_get.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_get.o -c ddi_get.c endif unset echo Finished compiling: ddi_get.o

Compiling DDI object: ddi_put.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_put.o -c ddi_put.c endif unset echo Finished compiling: ddi_put.o

Compiling DDI object: ddi_acc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_acc.o -c ddi_acc.c endif unset echo Finished compiling: ddi_acc.o

Compiling DDI object: ddi_getacc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_getacc.o -c ddi_getacc.c endif unset echo Finished compiling: ddi_getacc.o

Compiling DDI object: ddi_distrib.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_distrib.o -c ddi_distrib.c endif unset echo Finished compiling: ddi_distrib.o

Compiling DDI object: ddi_subpatch.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_subpatch.o -c ddi_subpatch.c endif unset echo Finished compiling: ddi_subpatch.o

Compiling DDI object: ddi_index.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_index.o -c ddi_index.c endif unset echo Finished compiling: ddi_index.o

Compiling DDI object: ddi_recvany.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_recvany.o -c ddi_recvany.c endif unset echo Finished compiling: ddi_recvany.o

Compiling DDI object: ddi_create.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_create.o -c ddi_create.c endif unset echo Finished compiling: ddi_create.o

Compiling DDI object: ddi_destroy.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_destroy.o -c ddi_destroy.c endif unset echo Finished compiling: ddi_destroy.o

Compiling DDI object: ddi_server.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_server.o -c ddi_server.c endif unset echo Finished compiling: ddi_server.o

Compiling DDI object: ddi_finalize.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_finalize.o -c ddi_finalize.c endif unset echo Finished compiling: ddi_finalize.o

Compiling DDI object: ddi_memory.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_memory.o -c ddi_memory.c endif unset echo Finished compiling: ddi_memory.o

Compiling DDI object: ddi_lapi.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_lapi.o -c ddi_lapi.c endif unset echo Finished compiling: ddi_lapi.o

Compiling DDI object: ddi_isend.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_isend.o -c ddi_isend.c endif unset echo Finished compiling: ddi_isend.o

Compiling DDI object: ddi_irecv.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_irecv.o -c ddi_irecv.c endif unset echo Finished compiling: ddi_irecv.o

Compiling DDI object: ddi_wait.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_wait.o -c ddi_wait.c endif unset echo Finished compiling: ddi_wait.o

Compiling DDI object: ddi_signals.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_signals.o -c ddi_signals.c endif unset echo Finished compiling: ddi_signals.o

Compiling DDI object: ddi_id.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_id.o -c ddi_id.c endif unset echo Finished compiling: ddi_id.o

Compiling DDI object: ddi_timer.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_timer.o -c ddi_timer.c endif unset echo Finished compiling: ddi_timer.o

Compiling DDI object: ddi_gdlb.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_gdlb.o -c ddi_gdlb.c endif unset echo Finished compiling: ddi_gdlb.o

Compiling DDI object: ddi_dlb.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_dlb.o -c ddi_dlb.c endif unset echo Finished compiling: ddi_dlb.o

Compiling DDI object: ddi_dlb_proc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_dlb_proc.o -c ddi_dlb_proc.c endif unset echo Finished compiling: ddi_dlb_proc.o

Compiling DDI object: ddi_init.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_init.o -c ddi_init.c 1500-030: (I) INFORMATION: Init_mpi(int, char **): Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192. endif unset echo Finished compiling: ddi_init.o

Compiling DDI object: shmem_ipc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o shmem_ipc.o -c shmem_ipc.c endif unset echo Finished compiling: shmem_ipc.o

Compiling DDI object: ddi_comm.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm.o -c ddi_comm.c endif unset echo Finished compiling: ddi_comm.o

Compiling DDI object: ddi_comm_group.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_group.o -c ddi_comm_group.c endif unset echo Finished compiling: ddi_comm_group.o

Compiling DDI object: ddi_comm_send.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_send.o -c ddi_comm_send.c endif unset echo Finished compiling: ddi_comm_send.o

Compiling DDI object: ddi_comm_recv.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_recv.o -c ddi_comm_recv.c endif unset echo Finished compiling: ddi_comm_recv.o

Compiling DDI object: ddi_comm_sync.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_sync.o -c ddi_comm_sync.c endif unset echo Finished compiling: ddi_comm_sync.o

Compiling DDI object: ddi_comm_gsum.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_gsum.o -c ddi_comm_gsum.c endif unset echo Finished compiling: ddi_comm_gsum.o

Compiling DDI object: ddi_comm_bcast.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_bcast.o -c ddi_comm_bcast.c endif unset echo Finished compiling: ddi_comm_bcast.o

Compiling DDI object: ddi_smp.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_smp.o -c ddi_smp.c endif unset echo Finished compiling: ddi_smp.o

Compiling DDI object: ddi_comm_create.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_create.o -c ddi_comm_create.c ddi_comm_create.c:363:45: warning: format specifies type 'int' but the argument has type 'MPI_Comm' (aka 'struct ompi_communicator_t ') [-Wformat] fprintf(stdout," world_comm = %i\n",comm->world_comm); ~~ ^~~~ ddi_comm_create.c:364:45: warning: format specifies type 'int' but the argument has type 'MPI_Comm' (aka 'struct ompi_communicator_t ') [-Wformat] fprintf(stdout," compute_comm = %i\n",comm->compute_comm); ~~ ^~~~~~ ddi_comm_create.c:366:41: warning: format specifies type 'int' but the argument has type 'MPI_Comm' (aka 'struct ompi_communicator_t ') [-Wformat] fprintf(stdout," smp_comm = %i\n",comm->smp_comm); ~~ ^~~~~~ ddi_comm_create.c:367:42: warning: format specifies type 'int' but the argument has type 'MPI_Comm' (aka 'struct ompi_communicator_t ') [-Wformat] fprintf(stdout," node_comm = %i\n",comm->node_comm); ~~ ^~~~~~~ 4 warnings generated. endif unset echo Finished compiling: ddi_comm_create.o

Compiling DDI object: ddi_comm_nproc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_nproc.o -c ddi_comm_nproc.c endif unset echo Finished compiling: ddi_comm_nproc.o

Compiling DDI object: ddi_comm_nnode.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_nnode.o -c ddi_comm_nnode.c endif unset echo Finished compiling: ddi_comm_nnode.o

Compiling DDI object: ddi_util.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_util.o -c ddi_util.c endif unset echo Finished compiling: ddi_util.o

Compiling DDI object: mmath.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o mmath.o -c mmath.c endif unset echo Finished compiling: mmath.o

Compiling DDI object: ddi_arr.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr.o -c ddi_arr.c endif unset echo Finished compiling: ddi_arr.o

Compiling DDI object: ddi_arr_acc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_acc.o -c ddi_arr_acc.c endif unset echo Finished compiling: ddi_arr_acc.o

Compiling DDI object: ddi_arr_add.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_add.o -c ddi_arr_add.c endif unset echo Finished compiling: ddi_arr_add.o

Compiling DDI object: ddi_arr_dot.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_dot.o -c ddi_arr_dot.c endif unset echo Finished compiling: ddi_arr_dot.o

Compiling DDI object: ddi_arr_scalar.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_scalar.o -c ddi_arr_scalar.c endif unset echo Finished compiling: ddi_arr_scalar.o

Compiling DDI object: ddi_arr_select.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_select.o -c ddi_arr_select.c endif unset echo Finished compiling: ddi_arr_select.o

Compiling DDI object: ddi_scattered_data.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_scattered_data.o -c ddi_scattered_data.c endif unset echo Finished compiling: ddi_scattered_data.o

Compiling DDI object: ddi_scatter_acc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_scatter_acc.o -c ddi_scatter_acc.c endif unset echo Finished compiling: ddi_scatter_acc.o

Compiling FORTRAN wrappers for DDI xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -DINT_SIZE=long -D_UNDERSCORES=0 -o ddi_fortran.o -c ddi_fortran.c unset echo Finished compiling: ddi_fortran.o

DDI compilation ended successfully. Sun Jan 19 20:07:21 HST 2020 5.760u 1.201s 0:07.15 97.3% 0+0k 0+0io 0pf+0w /gpfs/scratch/oppe/ABTP/app/gamess/ddi/compddi Building the Distributed Data Interface library started at Sun Jan 19 20:07:21 HST 2020

Compiling for machine type ibm64, using communication model mpi, System V shared memory option set true, with maxima of 168 processors/node and 4608 nodes.

This compilation should produce the DDI library libddi.a

The MPI-1 software library to be used is spectrum, with the include file 'mpi.h' at /gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include

Attempting to remove binary files from previous DDI build...

Beginning the DDI compilation at Sun Jan 19 20:07:21 HST 2020

Compiling common object: soc_create.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o soc_create.o -c soc_create.c unset echo Finished compiling: soc_create.o

Compiling common object: std_system.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o std_system.o -c std_system.c unset echo Finished compiling: std_system.o

Compiling common object: tcp_sockets.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o tcp_sockets.o -c tcp_sockets.c unset echo Finished compiling: tcp_sockets.o

Compiling common object: debug.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o debug.o -c debug.c unset echo Finished compiling: debug.o

Compiling common object: parse_node_args.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o parse_node_args.o -c parse_node_args.c unset echo Finished compiling: parse_node_args.o

Compiling DDI object: ddi.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi.o -c ddi.c endif unset echo Finished compiling: ddi.o

Compiling DDI object: ddi_send.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_send.o -c ddi_send.c endif unset echo Finished compiling: ddi_send.o

Compiling DDI object: ddi_recv.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_recv.o -c ddi_recv.c endif unset echo Finished compiling: ddi_recv.o

Compiling DDI object: ddi_get.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_get.o -c ddi_get.c endif unset echo Finished compiling: ddi_get.o

Compiling DDI object: ddi_put.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_put.o -c ddi_put.c endif unset echo Finished compiling: ddi_put.o

Compiling DDI object: ddi_acc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_acc.o -c ddi_acc.c endif unset echo Finished compiling: ddi_acc.o

Compiling DDI object: ddi_getacc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_getacc.o -c ddi_getacc.c endif unset echo Finished compiling: ddi_getacc.o

Compiling DDI object: ddi_distrib.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_distrib.o -c ddi_distrib.c endif unset echo Finished compiling: ddi_distrib.o

Compiling DDI object: ddi_subpatch.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_subpatch.o -c ddi_subpatch.c endif unset echo Finished compiling: ddi_subpatch.o

Compiling DDI object: ddi_index.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_index.o -c ddi_index.c endif unset echo Finished compiling: ddi_index.o

Compiling DDI object: ddi_recvany.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_recvany.o -c ddi_recvany.c endif unset echo Finished compiling: ddi_recvany.o

Compiling DDI object: ddi_create.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_create.o -c ddi_create.c endif unset echo Finished compiling: ddi_create.o

Compiling DDI object: ddi_destroy.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_destroy.o -c ddi_destroy.c endif unset echo Finished compiling: ddi_destroy.o

Compiling DDI object: ddi_server.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_server.o -c ddi_server.c endif unset echo Finished compiling: ddi_server.o

Compiling DDI object: ddi_finalize.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_finalize.o -c ddi_finalize.c endif unset echo Finished compiling: ddi_finalize.o

Compiling DDI object: ddi_memory.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_memory.o -c ddi_memory.c endif unset echo Finished compiling: ddi_memory.o

Compiling DDI object: ddi_lapi.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_lapi.o -c ddi_lapi.c endif unset echo Finished compiling: ddi_lapi.o

Compiling DDI object: ddi_isend.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_isend.o -c ddi_isend.c endif unset echo Finished compiling: ddi_isend.o

Compiling DDI object: ddi_irecv.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_irecv.o -c ddi_irecv.c endif unset echo Finished compiling: ddi_irecv.o

Compiling DDI object: ddi_wait.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_wait.o -c ddi_wait.c endif unset echo Finished compiling: ddi_wait.o

Compiling DDI object: ddi_signals.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_signals.o -c ddi_signals.c endif unset echo Finished compiling: ddi_signals.o

Compiling DDI object: ddi_id.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_id.o -c ddi_id.c endif unset echo Finished compiling: ddi_id.o

Compiling DDI object: ddi_timer.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_timer.o -c ddi_timer.c endif unset echo Finished compiling: ddi_timer.o

Compiling DDI object: ddi_gdlb.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_gdlb.o -c ddi_gdlb.c endif unset echo Finished compiling: ddi_gdlb.o

Compiling DDI object: ddi_dlb.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_dlb.o -c ddi_dlb.c endif unset echo Finished compiling: ddi_dlb.o

Compiling DDI object: ddi_dlb_proc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_dlb_proc.o -c ddi_dlb_proc.c endif unset echo Finished compiling: ddi_dlb_proc.o

Compiling DDI object: ddi_init.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_init.o -c ddi_init.c 1500-030: (I) INFORMATION: Init_mpi(int, char **): Additional optimization may be attained by recompiling and specifying MAXMEM option with a value greater than 8192. endif unset echo Finished compiling: ddi_init.o

Compiling DDI object: shmem_ipc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o shmem_ipc.o -c shmem_ipc.c endif unset echo Finished compiling: shmem_ipc.o

Compiling DDI object: ddi_comm.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm.o -c ddi_comm.c endif unset echo Finished compiling: ddi_comm.o

Compiling DDI object: ddi_comm_group.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_group.o -c ddi_comm_group.c endif unset echo Finished compiling: ddi_comm_group.o

Compiling DDI object: ddi_comm_send.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_send.o -c ddi_comm_send.c endif unset echo Finished compiling: ddi_comm_send.o

Compiling DDI object: ddi_comm_recv.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_recv.o -c ddi_comm_recv.c endif unset echo Finished compiling: ddi_comm_recv.o

Compiling DDI object: ddi_comm_sync.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_sync.o -c ddi_comm_sync.c endif unset echo Finished compiling: ddi_comm_sync.o

Compiling DDI object: ddi_comm_gsum.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_gsum.o -c ddi_comm_gsum.c endif unset echo Finished compiling: ddi_comm_gsum.o

Compiling DDI object: ddi_comm_bcast.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_bcast.o -c ddi_comm_bcast.c endif unset echo Finished compiling: ddi_comm_bcast.o

Compiling DDI object: ddi_smp.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_smp.o -c ddi_smp.c endif unset echo Finished compiling: ddi_smp.o

Compiling DDI object: ddi_comm_create.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_create.o -c ddi_comm_create.c ddi_comm_create.c:363:45: warning: format specifies type 'int' but the argument has type 'MPI_Comm' (aka 'struct ompi_communicator_t ') [-Wformat] fprintf(stdout," world_comm = %i\n",comm->world_comm); ~~ ^~~~ ddi_comm_create.c:364:45: warning: format specifies type 'int' but the argument has type 'MPI_Comm' (aka 'struct ompi_communicator_t ') [-Wformat] fprintf(stdout," compute_comm = %i\n",comm->compute_comm); ~~ ^~~~~~ ddi_comm_create.c:366:41: warning: format specifies type 'int' but the argument has type 'MPI_Comm' (aka 'struct ompi_communicator_t ') [-Wformat] fprintf(stdout," smp_comm = %i\n",comm->smp_comm); ~~ ^~~~~~ ddi_comm_create.c:367:42: warning: format specifies type 'int' but the argument has type 'MPI_Comm' (aka 'struct ompi_communicator_t ') [-Wformat] fprintf(stdout," node_comm = %i\n",comm->node_comm); ~~ ^~~~~~~ 4 warnings generated. endif unset echo Finished compiling: ddi_comm_create.o

Compiling DDI object: ddi_comm_nproc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_nproc.o -c ddi_comm_nproc.c endif unset echo Finished compiling: ddi_comm_nproc.o

Compiling DDI object: ddi_comm_nnode.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_comm_nnode.o -c ddi_comm_nnode.c endif unset echo Finished compiling: ddi_comm_nnode.o

Compiling DDI object: ddi_util.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_util.o -c ddi_util.c endif unset echo Finished compiling: ddi_util.o

Compiling DDI object: mmath.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o mmath.o -c mmath.c endif unset echo Finished compiling: mmath.o

Compiling DDI object: ddi_arr.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr.o -c ddi_arr.c endif unset echo Finished compiling: ddi_arr.o

Compiling DDI object: ddi_arr_acc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_acc.o -c ddi_arr_acc.c endif unset echo Finished compiling: ddi_arr_acc.o

Compiling DDI object: ddi_arr_add.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_add.o -c ddi_arr_add.c endif unset echo Finished compiling: ddi_arr_add.o

Compiling DDI object: ddi_arr_dot.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_dot.o -c ddi_arr_dot.c endif unset echo Finished compiling: ddi_arr_dot.o

Compiling DDI object: ddi_arr_scalar.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_scalar.o -c ddi_arr_scalar.c endif unset echo Finished compiling: ddi_arr_scalar.o

Compiling DDI object: ddi_arr_select.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_arr_select.o -c ddi_arr_select.c endif unset echo Finished compiling: ddi_arr_select.o

Compiling DDI object: ddi_scattered_data.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_scattered_data.o -c ddi_scattered_data.c endif unset echo Finished compiling: ddi_scattered_data.o

Compiling DDI object: ddi_scatter_acc.o xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -o ddi_scatter_acc.o -c ddi_scatter_acc.c endif unset echo Finished compiling: ddi_scatter_acc.o

Compiling FORTRAN wrappers for DDI xlc_r -O2 -m64 -DIBM64 -I./ -I/gpfs/pkgs/ibm/spectrum_mpi/10.3.0.1/include -DDDI_MPI -DUSE_SYSV -DUSE_SHMEM_SYSV -DMAX_SMP_PROCS=168 -DMAX_NODES=4608 -DINT_SIZE=long -D_UNDERSCORES=0 -o ddi_fortran.o -c ddi_fortran.c unset echo Finished compiling: ddi_fortran.o

DDI compilation ended successfully. Sun Jan 19 20:07:28 HST 2020 5.840u 1.117s 0:07.09 98.0% 0+0k 0+0io 0pf+0w

Output from "make modules"

xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave prec.F90 prec === End of Compilation 1 === 1501-510 Compilation successful for file prec.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mx_limits.F90 mx_limits === End of Compilation 1 === 1501-510 Compilation successful for file mx_limits.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave efpmodule.F90 efp_logical === End of Compilation 1 === dynpol_mol === End of Compilation 2 === dynpol_dist === End of Compilation 3 === efaidis === End of Compilation 4 === efaiexrep === End of Compilation 5 === efaielec === End of Compilation 6 === comm_frginf === End of Compilation 7 === comm_frgstd === End of Compilation 8 === comm_dotmp === End of Compilation 9 === comm_domult === End of Compilation 10 === comm_efmult === End of Compilation 11 === comm_efnrgy === End of Compilation 12 === comm_efpbas === End of Compilation 13 === comm_efpfrc === End of Compilation 14 === comm_efpmp2 === End of Compilation 15 === comm_efpotd === End of Compilation 16 === comm_fgrad === End of Compilation 17 === comm_frgmss === End of Compilation 18 === comm_paulmo === End of Compilation 19 === comm_dntcut === End of Compilation 20 === comm_efpar === End of Compilation 21 === comm_efpar2 === End of Compilation 22 === comm_efpct === End of Compilation 23 === comm_efpges === End of Compilation 24 === comm_efpio === End of Compilation 25 === comm_efppar === End of Compilation 26 === comm_efppbc === End of Compilation 27 === comm_efpscr === End of Compilation 28 === comm_frgljp === End of Compilation 29 === comm_frgtyp === End of Compilation 30 === comm_ieacal === End of Compilation 31 === comm_poltmp === End of Compilation 32 === comm_pcmefp === End of Compilation 33 === comm_frgnrg === End of Compilation 34 === comm_efpfmo === End of Compilation 35 === comm_efpgcx === End of Compilation 36 === comm_efpfm2 === End of Compilation 37 === comm_efptdg === End of Compilation 38 === comm_efpnfo === End of Compilation 39 === fmaxfgrad === End of Compilation 40 === 1501-510 Compilation successful for file efpmodule.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave params.F90 params === End of Compilation 1 === 1501-510 Compilation successful for file params.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave modmnfun.F90 modmnfun === End of Compilation 1 === 1501-510 Compilation successful for file modmnfun.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave modmcpdft.F90 modmcpdft === End of Compilation 1 === comm_maxroots === End of Compilation 2 === comm_rdm_mcpdft === End of Compilation 3 === 1501-510 Compilation successful for file modmcpdft.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mpcdatpm6.F90 mpcdatpm6 === End of Compilation 1 === 1501-510 Compilation successful for file mpcdatpm6.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mpchbond.F90 hp_hbcorr === End of Compilation 1 === 1501-510 Compilation successful for file mpchbond.F90. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave rmddat.f rmddat === End of Compilation 1 === 1501-510 Compilation successful for file rmddat.f. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave secor.F90 semiempirical_corrections === End of Compilation 1 === 1501-510 Compilation successful for file secor.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mod_nosp_basis.F90 mod_nosp_basis === End of Compilation 1 === 1501-510 Compilation successful for file mod_nosp_basis.F90. unset echo xlf2003 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mod_grid_storage.F90 mod_grid_storage === End of Compilation 1 === 1501-510 Compilation successful for file mod_grid_storage.F90. unset echo xlf2003 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mod_dft_partfunc.F90 mod_dft_partfunc === End of Compilation 1 === 1501-510 Compilation successful for file mod_dft_partfunc.F90. unset echo xlf2003 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mod_dft_molgrid.F90 mod_dft_molgrid === End of Compilation 1 === 1501-510 Compilation successful for file mod_dft_molgrid.F90. unset echo xlf90 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mod_dft_fuzzycell.F90 mod_dft_fuzzycell === End of Compilation 1 === 1501-510 Compilation successful for file mod_dft_fuzzycell.F90. unset echo xlf2003 -qsuffix=f=f90 -qfree=f90 -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave mod_dft_gridint.F90 ** mod_dft_gridint === End of Compilation 1 === 1501-510 Compilation successful for file mod_dft_gridint.F90. unset echo

Output from "make"

xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave aldeci.f aldeci === End of Compilation 1 === detinp === End of Compilation 2 === numcsf === End of Compilation 3 === icomb === End of Compilation 4 === detfci === End of Compilation 5 === rdci12 === End of Compilation 6 === detprt === End of Compilation 7 === c1det === End of Compilation 8 === detgrp === End of Compilation 9 === detval === End of Compilation 10 === detdm1 === End of Compilation 11 === detdm2 === End of Compilation 12 === wtdm12 === End of Compilation 13 === binom6 === End of Compilation 14 === davci === End of Compilation 15 === tran === End of Compilation 16 === etran === End of Compilation 17 === getq === End of Compilation 18 === initi === End of Compilation 19 === helem === End of Compilation 20 === retab === End of Compilation 21 === ret1det === End of Compilation 22 === posdet === End of Compilation 23 === advanc === End of Compilation 24 === retspin === End of Compilation 25 === reord === End of Compilation 26 === rinab === End of Compilation 27 === rinab0 === End of Compilation 28 === retab0 === End of Compilation 29 === mosypr === End of Compilation 30 === gajasw === End of Compilation 31 === cortra === End of Compilation 32 === memci === End of Compilation 33 === symwrk === End of Compilation 34 === gtab === End of Compilation 35 === gtab1 === End of Compilation 36 === gmul === End of Compilation 37 === gmul1 === End of Compilation 38 === getsym1 === End of Compilation 39 === sym === End of Compilation 40 === getdata === End of Compilation 41 === detci === End of Compilation 42 === setup === End of Compilation 43 === mempri === End of Compilation 44 === prici1 === End of Compilation 45 === prici2 === End of Compilation 46 === matme1 === End of Compilation 47 === matrd1 === End of Compilation 48 === matr82 === End of Compilation 49 === matme2 === End of Compilation 50 === matrds === End of Compilation 51 === matrd2 === End of Compilation 52 === ecorr === End of Compilation 53 === getdens === End of Compilation 54 === detno === End of Compilation 55 === cinrgy === End of Compilation 56 === geteny2 === End of Compilation 57 === malmq_cidriv === End of Compilation 58 === malmq_getorb === End of Compilation 59 === malmq_torb === End of Compilation 60 === malmq_tchk === End of Compilation 61 === malmq_tfact === End of Compilation 62 === malmq_rinab === End of Compilation 63 === malmq_findet === End of Compilation 64 === malmq_charge_transfer1 === End of Compilation 65 === malmq_charge_transfer2 === End of Compilation 66 === malmq_charge_transfer3 === End of Compilation 67 === malmq_den === End of Compilation 68 === malmq_integrals === End of Compilation 69 === malmq_iswitches === End of Compilation 70 === malmq_charge_transfer4 === End of Compilation 71 === malmq_findstate === End of Compilation 72 === malmq_iwts === End of Compilation 73 === 1501-510 Compilation successful for file aldeci.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave algnci.f algnci === End of Compilation 1 === gciinp === End of Compilation 2 === detgci === End of Compilation 3 === gciprt === End of Compilation 4 === sqrint === End of Compilation 5 === sqwint === End of Compilation 6 === gcisrt === End of Compilation 7 === spchek === End of Compilation 8 === readex === End of Compilation 9 === gcidet === End of Compilation 10 === gcidav === End of Compilation 11 === qget === End of Compilation 12 === gciniti === End of Compilation 13 === grinab0 === End of Compilation 14 === grinab === End of Compilation 15 === gretab0 === End of Compilation 16 === gretab === End of Compilation 17 === gspindet === End of Compilation 18 === gprici === End of Compilation 19 === gcidm1 === End of Compilation 20 === gcime1 === End of Compilation 21 === gcima1 === End of Compilation 22 === gcima2 === End of Compilation 23 === gcidm2 === End of Compilation 24 === gcime2 === End of Compilation 25 === gcisa1 === End of Compilation 26 === gcisa2 === End of Compilation 27 === gcigen === End of Compilation 28 === rspace === End of Compilation 29 === gcisym === End of Compilation 30 === gcicasg === End of Compilation 31 === ecorrg === End of Compilation 32 === 1501-510 Compilation successful for file algnci.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave basccn.f ccnbas === End of Compilation 1 === ccnshl === End of Compilation 2 === ccn001 === End of Compilation 3 === ccn002 === End of Compilation 4 === ccn003 === End of Compilation 5 === ccn004 === End of Compilation 6 === dccn03 === End of Compilation 7 === mccn03 === End of Compilation 8 === accn01 === End of Compilation 9 === accn02 === End of Compilation 10 === accn03 === End of Compilation 11 === accn04 === End of Compilation 12 === cccn02 === End of Compilation 13 === cccn03 === End of Compilation 14 === wcccn02 === End of Compilation 15 === wcccn03 === End of Compilation 16 === dkccn001 === End of Compilation 17 === dkccn002 === End of Compilation 18 === dkccn003 === End of Compilation 19 === dkccn004 === End of Compilation 20 === dkaccn04 === End of Compilation 21 === 1501-510 Compilation successful for file basccn.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave basecp.f hwbas === End of Compilation 1 === hwshl === End of Compilation 2 === hwthr === End of Compilation 3 === hwfor === End of Compilation 4 === hwfiv === End of Compilation 5 === hwsix === End of Compilation 6 === hwtm1 === End of Compilation 7 === hwtm2 === End of Compilation 8 === hwtm3 === End of Compilation 9 === sbkbas === End of Compilation 10 === sbkshl === End of Compilation 11 === sbklan === End of Compilation 12 === sbktwo === End of Compilation 13 === sbkthr === End of Compilation 14 === sbkfou === End of Compilation 15 === sbkfiv === End of Compilation 16 === sbksix === End of Compilation 17 === sbktm1 === End of Compilation 18 === sbktm2 === End of Compilation 19 === sbktm3 === End of Compilation 20 === 1501-510 Compilation successful for file basecp.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave basext.f dzvbas === End of Compilation 1 === dzvshl === End of Compilation 2 === dzv001 === End of Compilation 3 === dzv002 === End of Compilation 4 === dzv003 === End of Compilation 5 === dzv004 === End of Compilation 6 === n311g === End of Compilation 7 === n311xc === End of Compilation 8 === tzvbas === End of Compilation 9 === tzv001 === End of Compilation 10 === tzv002 === End of Compilation 11 === tzv003 === End of Compilation 12 === tzvtm1 === End of Compilation 13 === 1501-510 Compilation successful for file basext.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave basg3l.f ng3l === End of Compilation 1 === g3lone === End of Compilation 2 === g3ltwo === End of Compilation 3 === g3lthr === End of Compilation 4 === g3lfou === End of Compilation 5 === g3lfor === End of Compilation 6 === 1501-510 Compilation successful for file basg3l.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave basg3x.f ng3lx === End of Compilation 1 === g3lxone === End of Compilation 2 === g3lxtwo === End of Compilation 3 === g3lxthr === End of Compilation 4 === g3lxthrx === End of Compilation 5 === g3lxfou === End of Compilation 6 === g3lxfort === End of Compilation 7 === g3lxfor === End of Compilation 8 === 1501-510 Compilation successful for file basg3x.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave bashuz.f bashuz === End of Compilation 1 === hscale === End of Compilation 2 === hshell === End of Compilation 3 === hone === End of Compilation 4 === htwo === End of Compilation 5 === hthree === End of Compilation 6 === hfour === End of Compilation 7 === htmone === End of Compilation 8 === hfive === End of Compilation 9 === htmtwo === End of Compilation 10 === 1501-510 Compilation successful for file bashuz.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave bashz2.f hsix === End of Compilation 1 === hlanth === End of Compilation 2 === htmthr === End of Compilation 3 === 1501-510 Compilation successful for file bashz2.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave baskar.f ktzvbas === End of Compilation 1 === ktzv001 === End of Compilation 2 === ktzv002 === End of Compilation 3 === ktzv003 === End of Compilation 4 === ktzvpbas === End of Compilation 5 === ktzvp001 === End of Compilation 6 === ktzvp002 === End of Compilation 7 === ktzvp003 === End of Compilation 8 === ktzvppbas === End of Compilation 9 === ktzvpp001 === End of Compilation 10 === ktzvpp002 === End of Compilation 11 === ktzvpp003 === End of Compilation 12 === 1501-510 Compilation successful for file baskar.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave basminix.f basminix === End of Compilation 1 === minixone === End of Compilation 2 === minixtwo === End of Compilation 3 === minixthree === End of Compilation 4 === minixfour === End of Compilation 5 === minixtmone === End of Compilation 6 === 1501-510 Compilation successful for file basminix.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave basn21.f n21g === End of Compilation 1 === n21shl === End of Compilation 2 === n21one === End of Compilation 3 === n21two === End of Compilation 4 === n21thr === End of Compilation 5 === n21fou === End of Compilation 6 === n21tm1 === End of Compilation 7 === n21fiv === End of Compilation 8 === n21tm2 === End of Compilation 9 === 1501-510 Compilation successful for file basn21.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave basn31.f n31g === End of Compilation 1 === sfn31 === End of Compilation 2 === n31one === End of Compilation 3 === n31two === End of Compilation 4 === n31thr === End of Compilation 5 === n31fou === End of Compilation 6 === 1501-510 Compilation successful for file basn31.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave baspcn.f pcnbas === End of Compilation 1 === pcnshl === End of Compilation 2 === pcn001 === End of Compilation 3 === pcn002 === End of Compilation 4 === pcn003 === End of Compilation 5 === pcn004 === End of Compilation 6 === apcn01 === End of Compilation 7 === apcn02 === End of Compilation 8 === apcn03 === End of Compilation 9 === apcn04 === End of Compilation 10 === 1501-510 Compilation successful for file baspcn.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave bassto.f stong === End of Compilation 1 === stosf === End of Compilation 2 === stoshl === End of Compilation 3 === s1s === End of Compilation 4 === s2s === End of Compilation 5 === s2p === End of Compilation 6 === s2sp === End of Compilation 7 === s3s === End of Compilation 8 === s3p === End of Compilation 9 === s3d === End of Compilation 10 === s3sp === End of Compilation 11 === s3spd === End of Compilation 12 === s4s === End of Compilation 13 === s4p === End of Compilation 14 === s4d === End of Compilation 15 === s4sp === End of Compilation 16 === s4spd === End of Compilation 17 === s5s === End of Compilation 18 === s5p === End of Compilation 19 === s5d === End of Compilation 20 === s5sp === End of Compilation 21 === 1501-510 Compilation successful for file bassto.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave casino.f casinofile === End of Compilation 1 === 1501-510 Compilation successful for file casino.f. unset echo xlf -c -O2 -q64 -qintsize=8 -g -qarch=pwr8 -qtune=pwr8 -qflag=W:W -qhalt=W -qspillsize=2500 -qnosave ccaux.f transq === End of Compilation 1 === vecadd === End of Compilation 2 === veccop === End of Compilation 3 === vecmul === End of Compilation 4 === vecsub === End of Compilation 5 === tranmd === End of Compilation 6 === insi12 === End of Compilation 7 === insi13 === End of Compilation 8 === insi23 === End of Compilation 9 === insitu === End of Compilation 10 === trt1 === End of Compilation 11 === symetr === End of Compilation 12 === symt21 === End of Compilation 13 === desm21 === End of Compilation 14 === zeroma === End of Compilation 15 === mtrans === End of Compilation 16 === t1sq === End of Compilation 17 === mtra1 === End of Compilation 18 === blkcpy === End of Compilation 19 === drrlen === End of Compilation 20 === newrle === End of Compilation 21 === dodiis === End of Compilation 22 === minv === End of Compilation 23 === trmd === End of Compilation 24 === drrest === End of Compilation 25 === reapakres === End of Compilation 26 === wrires === End of Compilation 27 === reares === End of Compilation 28 === thrshsel === End of Compilation 29 === vripakres === End of Compilation 30 === scattr === End of Compilation 31 === store === End of Compilation 32 === ccmatmul === End of Compilation 33 === ccmatmult === End of Compilation 34 === cctmatmul === End of Compilation 35 === cctmatmult === End of Compilation 36 === fixsym === End of Compilation 37 === indexx === End of Compilation 38 === symetr2 === End of Compilation 39 === wrrle === End of Compilation 40 === rdrle === End of Compilation 41 === wwres === End of Compilation 42 === rrres === End of Compilation 43 === wrve === End of Compilation 44 === rdve === End of Compilation 45 === rdvve === End of Compilation 46 === wrvm === End of Compilation 47 === rdvm === End of Compilation 48 === rdve3 === End of Compilation 49 === rdvem3 === End of Compilation 50 === rdvem3x === End of Compilation 51 === rdvem4 === End of Compilation 52 === rdov4 === End of Compilation 53 === wrvpp === End of Compilation 54 === rdvpp === End of Compilation 55 === rdvppt === End of Compilation 56 === rdvppi === End of Compilation 57 === ro2hpp === End of Compilation 58 === wrt3 === End of Compilation 59 === rdt3 === End of Compilation 60 === wstar === End of Compilation 61 === rstar === End of Compilation 62 === wo1 === End of Compilation 63 === ro1 === End of Compilation 64 === wo2 === End of Compilation 65 === ro2 === End of Compilation 66 === wccfl === End of Compilation 67 === rccfl === End of Compilation 68 === wrexxve === End of Compilation 69 === rexxve === End of Compilation 70 === exwrve === End of Compilation 71 === exrdve === End of Compilation 72 === wrexxvm === End of Compilation 73 === rexxvm === End of Compilation 74 === exwrvm === End of Compilation 75 === exrdvm === End of Compilation 76 === r4c === End of Compilation 77 === w4c === End of Compilation 78 === r4d === End of Compilation 79 === w4d === End of Compilation 80 === r2bd === End of Compilation 81 === w2bd === End of Compilation 82 === r2c === End of Compilation 83 === w2c === End of Compilation 84 === r4b === End of Compilation 85 === w4b === End of Compilation 86 === roo1 === End of Compilation 87 === woo1 === End of Compilation 88 === wrr1 === End of Compilation 89 === rr1 === End of Compilation 90 === wrr2 === End of Compilation 91 === rr2 === End of Compilation 92 === xrf === End of Compilation 93 === rfhh === End of Compilation 94 === rfpp === End of Compilation 95 === wrtab === End of Compilation 96 === rdtab === End of Compilation 97 === wrtab2 === End of Compilation 98 === rdtab2 === End of Compilation 99 === 1501-510 Compilation succ

saromleang commented 4 years ago

It is more than likely that the "norm2.o" linking needs to be guarded by an logical check for threaded GAMESS build.

Changing:

BLAS="$BLAS norm2.o"

to

BLAS="$BLAS"

which is equivalent to not having the logical to add "norm2.o" to the object list allows me to successfully link non-threaded GAMESS on Summit.

@colleeneb could you verify that "norm2.o" is only needed for the threaded build of GAMESS?

colleeneb commented 4 years ago

I think we need norm2 generally, I'll have to double check tonight, though.

colleeneb commented 4 years ago

@saromleang I checked, now I agree with you. I can push the following to PR #301 if you want.

# this can go away once XLF2008 supports the norm2 intrinsic                                                                                                                                                                       
   if ($GMS_FORTRAN == xlf && $GMS_OPENMP == true) then
       set BLAS="$BLAS norm2.o"
   endif
saromleang commented 4 years ago

@colleeneb - I will take care of it. I have added something similar to the saromleang/build-support branch.

https://github.com/gms-bbg/gamess/commit/2fceecd6a40ccacf5378d40fa2cb4ebe6f1bd8cf

tomoppe commented 4 years ago

Dear Sir:

Has anyone investigated using the 64-bit integers, 64-bit pointers version of the ESSL library (-lessl6464) as an alternative to "blaswrap" in "comp"?

This concerns the targets ibm64, ibm64-sp, and ibm-bg.

Tom Oppe

-----Original Message----- From: Colleen Bertoni [mailto:notifications@github.com] Sent: Monday, January 20, 2020 11:12 PM To: gms-bbg/gamess-issues gamess-issues@noreply.github.com Cc: Oppe, Thomas C ERDC-RDE-ITL-MS Contractor Thomas.C.Oppe@erdc.dren.mil; Author author@noreply.github.com Subject: Re: [gms-bbg/gamess-issues] Trouble porting to an IBM Power8 platform running Linux (#15)

@saromleang Blockedhttps://github.com/saromleangBlocked I checked, now I agree with you. I can push the following to PR #301 if you want.

this can go away once XLF2008 supports the norm2 intrinsic

if ($GMS_FORTRAN == xlf && $GMS_OPENMP == true) then set BLAS="$BLAS norm2.o" endif

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub Blockedhttps://github.com/gms-bbg/gamess-issues/issues/15?email_source=notifications&email_token=AMLWS3B2U2AJUBZ3ROOEQF3Q6Z73DA5CNFSM4KIY5ARKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJOQGIQ#issuecomment-576520994Blocked , or unsubscribe Blockedhttps://github.com/notifications/unsubscribe-auth/AMLWS3BPHRT2PLXVA5L3SWLQ6Z73DANCNFSM4KIY5ARABlocked . Blockedhttps://github.com/notifications/beacon/AMLWS3GWVRA2FGPETPLMBETQ6Z73DA5CNFSM4KIY5ARKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJOQGIQ.gifBlocked

colleeneb commented 4 years ago

I haven't, not sure about others. It would be a useful thing to look at, though. If you get it working properly, we can probably add it as an option.