search

shankar1729 / jdftx

JDFTx: software for joint density functional theory
http://jdftx.org
82 stars 54 forks source link

Has anyone compiled JDFTx on Polaris with GPU successfully? #326

Closed ColinBundschu closed 4 months ago

ColinBundschu commented 4 months ago

I have spent several days unsuccessfully attempting to get JDFTx to work with the CRAY environment and nvc++ compilers on ALCF's Polaris supercomputer. These include issues with not recognizing GSL 2.7 and requiring me to modify the CMakeLists.txt file directly (such as changing the cmake directory from the hard coded path to the one CRAY uses). I have been able to build it, but all tests fail. I am kind of at a loss as to how to proceed. Has anyone made this work?

ColinBundschu commented 4 months ago

Excellent, thank you. Also it looks like each compute node has 4 A100s, each with 40GB mem. Given that I plan to run tens of thousands of surface calculations on graphene and oxide like structures, do you have any advice on how to divy up GPU resources? For example, one calculation per GPU at a time, or N calcs per GPU, or N GPUs per calc?

shankar1729 commented 4 months ago

Okay, the tests may be bottlenecked on cuda startup time then: likely the driver does not have persistence mode set for the GPUs. Without it, you can have ~ 3-5 s extra time during gpu_init, which is irrelevant for real jobs, but during the tests that run ~ 20-25 small calcs total in the 10 tests, this adds up.

For the large array of jobs, I'd recommend using 1 GPU or at most 1 node per job to avoid interconnect delays. If each of your jobs fits within one GPU and it is easy to launch single GPU jobs (not always the case in al slurm setups), then that would be the most efficient for overall throughput.

Also, what kind of application are you running solvated calculations for? If it is for catalysis, talk to us about the simulation settings first. We'd like to sync new JDFTx-based efforts on electrocatalysis with our BEAST-DB initiative, which was just made public.

shankar1729 commented 4 months ago

Sounds great: let's talk further on email. Let's close this issue once you check that some more reasonably sized systems run correctly.

ColinBundschu commented 4 months ago

Edit: Trying to fix the links, they seem broken to me

Yes, it is for catalysis. I would love to discuss the settings. I have two projects running right now - one is on ORR for spinel oxides running on about 2000 CPUs on Jetstream2. Its based on the Co3O4 paper here that we published a few months ago, but now we are applying it to the entire space of spinel oxides to find ideal ORR candidates: https://doi.org/10.1021/jacs.3c11977

The project on Polaris is for MNC electrocatalysts, and is a partnership between me and a team at LANL.

I read through your BEAST-DB initiative, and I think the venn diagram of our interests is fairly close to a circle! :)

We should sync up - email work? crb273@cornell.edu

ColinBundschu commented 4 months ago

Sounds good, I will do those tests and report back. I will also open a couple more wish list item issues if that's OK, which have been on my radar but I keep forgetting to ask about.

ColinBundschu commented 4 months ago

How can I specify the GPU to run on?

*************** JDFTx 1.7.0 (git hash aa096dda) ***************

Start date and time: Wed May 15 18:48:08 2024
Executable /home/cbu/jdftx/build/jdftx_gpu with command-line: -i /home/cbu/dft_out/FeNC/clean/661.in -o /home/cbu/dft_out/FeNC/clean/661.out
Running on hosts (process indices):  x3004c0s37b1n0 (0)
Divided in process groups (process indices):  0 (0)
gpuInit: Found compatible cuda device 0 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 1 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 2 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 3 'NVIDIA A100-SXM4-40GB'
gpuInit: Selected device 0
Resource initialization completed at t[s]:      1.69
Run totals: 1 processes, 32 threads, 1 GPUs
shankar1729 commented 4 months ago

This is controlled by CUDA_VISIBLE_DEVICES. If you are running in slurm, use the recommended mechanism on your cluster to control the per-task or per-node GPU allocation, example using --gres, --gpus_per_node etc. This will automatically set that variable and JDFTx will pick it up.

Looks like Polaris is using PBS, not slurm. Hmm, more bad choices.

In that case, just launch 4 jdftx_gpu processes per node; JDFTx will automatically take care of selecting a different GPU from each process on the node.

ColinBundschu commented 4 months ago

jdftx is failing to allocate memory, even when I set export JDFTX_MEMPOOL_SIZE=38000 beforehand (I also tried 20000). See attached zip.

cbu@x3006c0s19b1n0:~/dft_out/FeNC/clean> nvidia-smi
Wed May 15 19:11:02 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.154.05             Driver Version: 535.154.05   CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A100-SXM4-40GB          On  | 00000000:07:00.0 Off |                    0 |
| N/A   30C    P0              53W / 400W |      0MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A100-SXM4-40GB          On  | 00000000:46:00.0 Off |                    0 |
| N/A   31C    P0              51W / 400W |      0MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A100-SXM4-40GB          On  | 00000000:85:00.0 Off |                    0 |
| N/A   32C    P0              58W / 400W |      0MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A100-SXM4-40GB          On  | 00000000:C7:00.0 Off |                    0 |
| N/A   33C    P0              53W / 400W |      0MiB / 40960MiB |      0%      Default |
|                                         |                      |             Disabled |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+
cbu@x3006c0s19b1n0:~/dft_out/FeNC/clean> export JDFTX_MEMPOOL_SIZE=38000
cbu@x3006c0s19b1n0:~/dft_out/FeNC/clean> /home/cbu/jdftx/build/jdftx_gpu -i /home/cbu/dft_out/FeNC/clean/661.in -o /home/cbu/dft_out/FeNC/clean/661.out
GPU memory allocation failed (out of memory)
MPICH ERROR [Rank 0] [job id ] [Wed May 15 19:13:16 2024] [x3006c0s19b1n0] - Abort(1) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

cbu@x3006c0s19b1n0:~/dft_out/FeNC/clean> export JDFTX_MEMPOOL_SIZE=20000
cbu@x3006c0s19b1n0:~/dft_out/FeNC/clean> /home/cbu/jdftx/build/jdftx_gpu -i /home/cbu/dft_out/FeNC/clean/661.in -o /home/cbu/dft_out/FeNC/clean/661.out
GPU memory allocation failed (out of memory)
MPICH ERROR [Rank 0] [job id ] [Wed May 15 19:14:00 2024] [x3006c0s19b1n0] - Abort(1) (rank 0 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0

cbu@x3006c0s19b1n0:~/dft_out/FeNC/clean>

clean.zip

shankar1729 commented 4 months ago

Based on your system size, you're actually running out of memory to store all those wavefunctions on a single GPU. Quick estimate based on bands and basis is that you need 0.83 GB / k-point, per copy of wave function, which is 30 GB for 36 k-points.

Realistically, you need several copy of the wavefunctions during minimize / SCF, at least 5 - 6. So you need around 150 GB for this calculation. Consequently, you'll need to use at least 4 GPUs, if not more to fit this calculation.

ColinBundschu commented 4 months ago

Ok thanks for clarifying that, I am a little uncertain on how to do the estimates for the needed memory (other than guess and check, which has been my go-to thus far), so I apologize for my misunderstanding there, and I typically have not run calculations at this level of kpoint sampling. I ran the same calculation at a more modest 331 kpoint sampling, which seems to be working well. I will run some more tests and then close this out if it all looks correct.

shankar1729 commented 4 months ago

Sounds good: in any case, also make sure the multi-GPU and CUDA-aware MPI is working correctly. This can also be a source of issues.

ColinBundschu commented 4 months ago

It seems that jdftx is crashing when running under mpi, and I am unable to determine why

Job script:

#!/bin/bash
#PBS -l select=2
#PBS -l walltime=01:00:00
#PBS -q debug
#PBS -l filesystems=home
#PBS -A von
#PBS -o /home/cbu/dft_out/FeNC/clean/661_output.log
#PBS -e /home/cbu/dft_out/FeNC/clean/661_error.log

# MPI and OpenMP settings
NNODES=`wc -l < $PBS_NODEFILE`
NRANKS_PER_NODE=$(nvidia-smi -L | wc -l)
NDEPTH=8
NTHREADS=1
NTOTRANKS=$(( NNODES * NRANKS_PER_NODE ))
echo "NUM_OF_NODES= ${NNODES} TOTAL_NUM_RANKS= ${NTOTRANKS} RANKS_PER_NODE= ${NRANKS_PER_NODE} THREADS_PER_RANK= ${NTHREADS}"

cd /home/cbu/dft_out/FeNC/clean/
mpiexec -n ${NTOTRANKS} --ppn ${NRANKS_PER_NODE} --depth=${NDEPTH} --cpu-bind depth --env OMP_NUM_THREADS=${NTHREADS} -env OMP_PLACES=threads /home/cbu/jdftx/build/jdftx_gpu -i /home/cbu/dft_out/FeNC/clean/661.in -o /home/cbu/dft_out/FeNC/clean/661.out

Error output:

process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
/opt/cray/pe/lib64/libmpi_nvidia.so.12(MPL_backtrace_show+0x26) [0x15256021d13b]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1b43354) [0x15255fc7c354]
process_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
/opt/cray/pe/lib64/libmpi_nvidia.so.12(MPL_backtrace_show+0x26) [0x14d83c41d13b]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1b43354) [0x14d83be7c354]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f78a00) [0x14d83c2b1a00]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f942d2) [0x14d83c2cd2d2]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f71265) [0x14d83c2aa265]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0xcf9fe9) [0x14d83b032fe9]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x189ba26) [0x14d83bbd4a26]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x18a776d) [0x14d83bbe076d]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x17d0707) [0x14d83bb09707]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x172730) [0x14d83a4ab730]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1727f1) [0x14d83a4ab7f1]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1defe) [0x14d83bd56efe]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1e42e) [0x14d83bd5742e]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(PMPI_Allreduce+0x2ec) [0x14d83a4f737c]
/soft/perftools/darshan/darshan-3.4.4/lib/libdarshan.so.0(PMPI_Allreduce+0xc0) [0x14d891ff24c0]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil9allReduceIdEEvPT_mNS_8ReduceOpEbPi+0x188) [0x14d8930c8a88]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil13allReduceDataIdEEvR13ManagedMemoryIT_ENS_8ReduceOpEbPi+0x53) [0x14d893433ed3]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN9FieldDataIdE13allReduceDataEPK7MPIUtilNS1_8ReduceOpEbPi+0x3d) [0x14d8934350fd]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt6vectorISt10shared_ptrI15ScalarFieldDataESaIS3_EEPS5_10IncludeTXCPS6_S8_P7matrix3IdE+0x10a5) [0x14d893545265]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt10shared_ptrI15ScalarFieldDataEPS2_10IncludeTXCPS3_S5_P7matrix3IdE+0x1d7) [0x14d8935476d7]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN7IonInfo6updateER8Energies+0x349) [0x14d8935fc289]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN10Everything5setupEv+0xcfa) [0x14d89350a2fa]
/home/cbu/jdftx/build/jdftx_gpu() [0x437a8d]
/lib64/libc.so.6(__libc_start_main+0xef) [0x14d881e3e24d]
/home/cbu/jdftx/build/jd/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f78a00) [0x1525600b1a00]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f942d2) [0x1525600cd2d2]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f71265) [0x1525600aa265]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0xcf9fe9) [0x15255ee32fe9]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x189ba26) [0x15255f9d4a26]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x18a776d) [0x15255f9e076d]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x17d0707) [0x15255f909707]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x172730) [0x15255e2ab730]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1727f1) [0x15255e2ab7f1]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1defe) [0x15255fb56efe]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1e42e) [0x15255fb5742e]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(PMPI_Allreduce+0x2ec) [0x15255e2f737c]
/soft/perftools/darshan/darshan-3.4.4/lib/libdarshan.so.0(PMPI_Allreduce+0xc0) [0x1525b5e614c0]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil9allReduceIdEEvPT_mNS_8ReduceOpEbPi+0x188) [0x1525b6f37a88]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil13allReduceDataIdEEvR13ManagedMemoryIT_ENS_8ReduceOpEbPi+0x53) [0x1525b72a2ed3]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN9FieldDataIdE13allReduceDataEPK7MPIUtilNS1_8ReduceOpEbPi+0x3d) [0x1525b72a40fd]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt6vectorISt10shared_ptrI15ScalarFieldDataESaIS3_EEPS5_10IncludeTXCPS6_S8_P7matrix3IdE+0x10a5) [0x1525b73b4265]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt10shared_ptrI15ScalarFieldDataEPS2_10IncludeTXCPS3_S5_P7matrix3IdE+0x1d7) [0x1525b73b66d7]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN7IonInfo6updateER8Energies+0x349) [0x1525b746b289]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN10Everything5setupEv+0xcfa) [0x1525b73792fa]
/home/cbu/jdftx/build/jdftx_gpu() [0x437a8d]
/lib64/libc.so.6(__libc_start_main+0xef) [0x1525a5c3e24d]
/home/cbu/jdftx/build/jdftx_gpu() [0x43783a]
MPICH ERROR [Rank 4] [job id f42feb4a-cbc2-4566-a429-93e5a3b180de] [Fri May 17 15:26:07 2024] [x3006c0s25b1n0] - Abort(1): Interftx_gpu() [0x43783a]
MPICH ERROR [Rank 5] [job id f42feb4a-cbc2-4566-a429-93e5a3b180de] [Fri May 17 15:26:07 2024] [x3006c0s25b1n0] - Abort(1): Internal error

process_vm_readv: Bad address
/opt/cray/pe/lib64/libmpi_nvidia.so.12(MPL_backtrace_show+0x26) [0x1463cc41d13b]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1b43354) [0x1463cbe7c354]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f78a00) [0x1463cc2b1a00]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f942d2) [0x1463cc2cd2d2]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f71265) [0x1463cc2aa265]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0xcf9fe9) [0x1463cb032fe9]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x189ba26) [0x1463cbbd4a26]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x18a776d) [0x1463cbbe076d]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x17d0707) [0x1463cbb09707]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x172730) [0x1463ca4ab730]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1727f1) [0x1463ca4ab7f1]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1defe) [0x1463cbd56efe]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1e42e) [0x1463cbd5742e]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(PMPI_Allreduce+0x2ec) [0x1463ca4f737c]
/soft/perftools/darshan/darshan-3.4.4/lib/libdarshan.so.0(PMPI_Allreduce+0xc0) [0x146421fbe4c0]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil9allReduceIdEEvPT_mNS_8ReduceOpEbPi+0x188) [0x146423094a88]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil13allReduceDataIdEEvR13ManagedMemoryIT_ENS_8ReduceOpEbPi+0x53) [0x1464233ffed3]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN9FieldDataIdE13allReduceDataEPK7MPIUtilNS1_8ReduceOpEbPi+0x3d) [0x1464234010fd]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt6vectorISt10shared_ptrI15ScalarFieldDataESaIS3_EEPS5_10IncludeTXCPS6_S8_P7matrix3IdE+0x10a5) [0x146423511265]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt10shared_ptrI15ScalarFieldDataEPS2_10IncludeTXCPS3_S5_P7matrix3IdE+0x1d7) [0x1464235136d7]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN7IonInfo6updateER8Energies+0x349) [0x1464235c8289]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN10Everything5setupEv+0xcfa) [0x1464234d62fa]
/home/cbu/jdftx/build/jdftx_gpu() [0x437a8d]
/lib64/libc.so.6(__libc_start_main+0xef) [0x146411e3e24d]
/home/cbu/jdftx/build/jdprocess_vm_readv: Bad address
Assertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
/opt/cray/pe/lib64/libmpi_nvidia.so.12(MPL_backtrace_show+0x26) [0x150e91e1d13b]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1b43354) [0x150e9187c354]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f78a00) [0x150e91cb1a00]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f942d2) [0x150e91ccd2d2]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f71265) [0x150e91caa265]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0xcf9fe9) [0x150e90a32fe9]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x189ba26) [0x150e915d4a26]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x18a776d) [0x150e915e076d]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x17d0707) [0x150e91509707]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x172730) [0x150e8feab730]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1727f1) [0x150e8feab7f1]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1defe) [0x150e91756efe]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1e42e) [0x150e9175742e]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(PMPI_Allreduce+0x2ec) [0x150e8fef737c]
/soft/perftools/darshan/darshan-3.4.4/lib/libdarshan.so.0(PMPI_Allreduce+0xc0) [0x150ee7a3a4c0]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil9allReduceIdEEvPT_mNS_8ReduceOpEbPi+0x188) [0x150ee8b10a88]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil13allReduceDataIdEEvR13ManagedMemoryIT_ENS_8ReduceOpEbPi+0x53) [0x150ee8e7bed3]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN9FieldDataIdE13allReduceDataEPK7MPIUtilNS1_8ReduceOpEbPi+0x3d) [0x150ee8e7d0fd]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt6vectorISt10shared_ptrI15ScalarFieldDataESaIS3_EEPS5_10IncludeTXCPS6_S8_P7matrix3IdE+0x10a5) [0x150ee8f8d265]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt10shared_ptrI15ScalarFieldDataEPS2_10IncludeTXCPS3_S5_P7matrix3IdE+0x1d7) [0x150ee8f8f6d7]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN7IonInfo6updateER8Energies+0x349) [0x150ee9044289]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN10Everything5setupEv+0xcfa) [0x150ee8f522fa]
/home/cbu/jdftx/build/jdftx_gpu() [0x437a8d]
/lib64/libc.so.6(__libc_start_main+0xef) [0x150ed783e24d]
/home/cbu/jdftx/build/jdAssertion failed in file ../src/mpid/ch4/shm/cray_common/cray_common_memops.c at line 461: 0
nal error

ftx_gpu() [0x43783a]
MPICH ERROR [Rank 6] [job id f42feb4a-cbc2-4566-a429-93e5a3b180de] [Fri May 17 15:26:07 2024] [x3006c0s25b1n0] - Abort(1): Internal error

ftx_gpu() [0x43783a]
MPICH ERROR [Rank 7] [job id f42feb4a-cbc2-4566-a429-93e5a3b180de] [Fri May 17 15:26:07 2024] [x3006c0s25b1n0] - Abort(1): Internal error

/opt/cray/pe/lib64/libmpi_nvidia.so.12(MPL_backtrace_show+0x26) [0x14c1a961d13b]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1b43354) [0x14c1a907c354]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f78a00) [0x14c1a94b1a00]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f942d2) [0x14c1a94cd2d2]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f71265) [0x14c1a94aa265]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0xcf9fe9) [0x14c1a8232fe9]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x189ba26) [0x14c1a8dd4a26]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(MPL_backtrace_show+0x26) [0x153e64e1d13b]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1b43354) [0x153e6487c354]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f78a00) [0x153e64cb1a00]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f942d2) [0x153e64ccd2d2]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f71265) [0x153e64caa265]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0xcf9fe9) [0x153e63a32fe9]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x189ba26) [0x153e645d4a26]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x18a776d) [0x153e645e076d]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x17d0707) [0x153e64509707]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x172730) [0x153e62eab730]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1727f1) [0x153e62eab7f1]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1defe) [0x153e64756efe]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1e42e) [0x153e6475742e]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(PMPI_Allreduce+0x2ec) [0x153e62ef737c]
/soft/perftools/darshan/darshan-3.4.4/lib/libdarshan.so.0(PMPI_Allreduce+0xc0) [0x153ebaa2a4c0]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil9allReduceIdEEvPT_mNS_8ReduceOpEbPi+0x188) [0x153ebbb00a88]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil13allReduceDataIdEEvR13ManagedMemoryIT_ENS_8ReduceOpEbPi+0x53) [0x153ebbe6bed3]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN9FieldDataIdE13allReduceDataEPK7MPIUtilNS1_8ReduceOpEbPi+0x3d) [0x153ebbe6d0fd]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt6vectorISt10shared_ptrI15ScalarFieldDataESaIS3_EEPS5_10IncludeTXCPS6_S8_P7matrix3IdE+0x10a5) [0x153ebbf7d265]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt10shared_ptrI15ScalarFieldDataEPS2_10IncludeTXCPS3_S5_P7matrix3IdE+0x1d7) [0x153ebbf7f6d7]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN7IonInfo6updateER8Energies+0x349) [0x153ebc034289]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN10Everything5setupEv+0xcfa) [0x153ebbf422fa]
/home/cbu/jdftx/build/jdftx_gpu() [0x437a8d]
/lib64/libc.so.6(__libc_start_main+0xef) [0x153eaa83e24d]
/home/cbu/jdftx/build/jd/opt/cray/pe/lib64/libmpi_nvidia.so.12(MPL_backtrace_show+0x26) [0x14601d01d13b]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1b43354) [0x14601ca7c354]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f78a00) [0x14601ceb1a00]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f942d2) [0x14601cecd2d2]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f71265) [0x14601ceaa265]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0xcf9fe9) [0x14601bc32fe9]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x189ba26) [0x14601c7d4a26]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x18a776d) [0x14601c7e076d]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x17d0707) [0x14601c709707]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x172730) [0x14601b0ab730]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1727f1) [0x14601b0ab7f1]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1defe) [0x14601c956efe]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1e42e) [0x14601c95742e]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(PMPI_Allreduce+0x2ec) [0x14601b0f737c]
/soft/perftools/darshan/darshan-3.4.4/lib/libdarshan.so.0(PMPI_Allreduce+0xc0) [0x14606da6b4c0]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil9allReduceIdEEvPT_mNS_8ReduceOpEbPi+0x188) [0x146073b60a88]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil13allReduceDataIdEEvR13ManagedMemoryIT_ENS_8ReduceOpEbPi+0x53) [0x146073ecbed3]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN9FieldDataIdE13allReduceDataEPK7MPIUtilNS1_8ReduceOpEbPi+0x3d) [0x146073ecd0fd]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt6vectorISt10shared_ptrI15ScalarFieldDataESaIS3_EEPS5_10IncludeTXCPS6_S8_P7matrix3IdE+0x10a5) [0x146073fdd265]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt10shared_ptrI15ScalarFieldDataEPS2_10IncludeTXCPS3_S5_P7matrix3IdE+0x1d7) [0x146073fdf6d7]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN7IonInfo6updateER8Energies+0x349) [0x146074094289]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN10Everything5setupEv+0xcfa) [0x146073fa22fa]
/home/cbu/jdftx/build/jdftx_gpu() [0x437a8d]
/lib64/libc.so.6(__libc_start_main+0xef) [0x146062a3e24d]
/home/cbu/jdftx/build/jd/opt/cray/pe/lib64/libmpi_nvidia.so.12(MPL_backtrace_show+0x26) [0x14c261e1d13b]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1b43354) [0x14c26187c354]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f78a00) [0x14c261cb1a00]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f942d2) [0x14c261ccd2d2]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1f71265) [0x14c261caa265]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0xcf9fe9) [0x14c260a32fe9]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x189ba26) [0x14c2615d4a26]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x18a776d) [0x14c2615e076d]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x17d0707) [0x14c261509707]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x172730) [0x14c25feab730]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1727f1) [0x14c25feab7f1]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1defe) [0x14c261756efe]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1e42e) [0x14c26175742e]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(PMPI_Allreduce+0x2ec) [0x14c25fef737c]
/soft/perftools/darshan/darshan-3.4.4/lib/libdarshan.so.0(PMPI_Allreduce+0xc0) [0x14c2b7a374c0]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil9allReduceIdEEvPT_mNS_8ReduceOpEbPi+0x188) [0x14c2b8b0da88]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil13allReduceDataIdEEvR13ManagedMemoryIT_ENS_8ReduceOpEbPi+0x53) [0x14c2b8e78ed3]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN9FieldDataIdE13allReduceDataEPK7MPIUtilNS1_8ReduceOpEbPi+0x3d) [0x14c2b8e7a0fd]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt6vectorISt10shared_ptrI15ScalarFieldDataESaIS3_EEPS5_10IncludeTXCPS6_S8_P7matrix3IdE+0x10a5) [0x14c2b8f8a265]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt10shared_ptrI15ScalarFieldDataEPS2_10IncludeTXCPS3_S5_P7matrix3IdE+0x1d7) [0x14c2b8f8c6d7]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN7IonInfo6updateER8Energies+0x349) [0x14c2b9041289]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN10Everything5setupEv+0xcfa) [0x14c2b8f4f2fa]
/home/cbu/jdftx/build/jdftx_gpu() [0x437a8d]
/lib64/libc.so.6(__libc_start_main+0xef) [0x14c2a783e24d]
/home/cbu/jdftx/build/jd/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x18a776d) [0x14c1a8de076d]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x17d0707) [0x14c1a8d09707]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x172730) [0x14c1a76ab730]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1727f1) [0x14c1a76ab7f1]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1defe) [0x14c1a8f56efe]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(+0x1a1e42e) [0x14c1a8f5742e]
/opt/cray/pe/lib64/libmpi_nvidia.so.12(PMPI_Allreduce+0x2ec) [0x14c1a76f737c]
/soft/perftools/darshan/darshan-3.4.4/lib/libdarshan.so.0(PMPI_Allreduce+0xc0) [0x14c1fa0844c0]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil9allReduceIdEEvPT_mNS_8ReduceOpEbPi+0x188) [0x14c2001b4a88]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK7MPIUtil13allReduceDataIdEEvR13ManagedMemoryIT_ENS_8ReduceOpEbPi+0x53) [0x14c20051fed3]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN9FieldDataIdE13allReduceDataEPK7MPIUtilNS1_8ReduceOpEbPi+0x3d) [0x14c2005210fd]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt6vectorISt10shared_ptrI15ScalarFieldDataESaIS3_EEPS5_10IncludeTXCPS6_S8_P7matrix3IdE+0x10a5) [0x14c200631265]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZNK6ExCorrclERKSt10shared_ptrI15ScalarFieldDataEPS2_10IncludeTXCPS3_S5_P7matrix3IdE+0x1d7) [0x14c2006336d7]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN7IonInfo6updateER8Energies+0x349) [0x14c2006e8289]
/home/cbu/jdftx/build/libjdftx_gpu.so(_ZN10Everything5setupEv+0xcfa) [0x14c2005f62fa]
/home/cbu/jdftx/build/jdftx_gpu() [0x437a8d]
/lib64/libc.so.6(__libc_start_main+0xef) [0x14c1ef03e24d]
/home/cbu/jdftx/build/jdftx_gpu() [0x43783a]
MPICH ERROR [Rank 1] [job id f42feb4a-cbc2-4566-a429-93e5a3b180de] [Fri May 17 15:26:07 2024] [x3006c0s13b1n0] - Abort(1): Internal error

ftx_gpu() [0x43783a]
MPICH ERROR [Rank 2] [job id f42feb4a-cbc2-4566-a429-93e5a3b180de] [Fri May 17 15:26:07 2024] [x3006c0s13b1n0] - Abort(1): Internal error

ftx_gpu() [0x43783a]
MPICH ERROR [Rank 3] [job id f42feb4a-cbc2-4566-a429-93e5a3b180de] [Fri May 17 15:26:07 2024] [x3006c0s13b1n0] - Abort(1): Internal error

ftx_gpu() [0x43783a]
MPICH ERROR [Rank 0] [job id f42feb4a-cbc2-4566-a429-93e5a3b180de] [Fri May 17 15:26:07 2024] [x3006c0s13b1n0] - Abort(1): Internal error

x3006c0s25b1n0.hsn.cm.polaris.alcf.anl.gov: rank 5 exited with code 1
shankar1729 commented 4 months ago

You likely need to export MPICH_GPU_SUPPORT_ENABLED=1; see https://docs.alcf.anl.gov/polaris/running-jobs/#running-gpu-enabled-applications

ColinBundschu commented 4 months ago

I tried running the tests with MPI and gpu, and one of the tests keeps failing:

cbu@x3102c0s13b1n0:~/jdftx/build> make test
Running tests...
Test project /home/cbu/jdftx/build
      Start  1: openShell
 1/10 Test  #1: openShell ........................   Passed    0.03 sec
      Start  2: vibrations
 2/10 Test  #2: vibrations .......................   Passed    0.03 sec
      Start  3: moleculeSolvation
 3/10 Test  #3: moleculeSolvation ................   Passed    0.03 sec
      Start  4: ionSolvation
 4/10 Test  #4: ionSolvation .....................   Passed    0.03 sec
      Start  5: latticeOpt
 5/10 Test  #5: latticeOpt .......................   Passed    0.02 sec
      Start  6: metalBulk
 6/10 Test  #6: metalBulk ........................***Failed    7.93 sec
      Start  7: plusU
 7/10 Test  #7: plusU ............................   Passed    0.02 sec
      Start  8: spinOrbit
 8/10 Test  #8: spinOrbit ........................   Passed    0.02 sec
      Start  9: graphene
 9/10 Test  #9: graphene .........................   Passed    0.03 sec
      Start 10: metalSurface
10/10 Test #10: metalSurface .....................   Passed    0.03 sec

90% tests passed, 1 tests failed out of 10

Total Test time (real) =   8.18 sec

The following tests FAILED:
          6 - metalBulk (Failed)
Errors while running CTest
Output from these tests are in: /home/cbu/jdftx/build/Testing/Temporary/LastTest.log
Use "--rerun-failed --output-on-failure" to re-run the failed cases verbosely.
make: *** [Makefile:71: test] Error 8
cbu@x3102c0s13b1n0:~/jdftx/build>
cbu@x3102c0s13b1n0:~/jdftx/build/test/metalBulk> cat totalE.out

*************** JDFTx 1.7.0 (git hash aa096dda) ***************

Start date and time: Sat May 18 19:54:35 2024
Executable /home/cbu/jdftx/build/jdftx_gpu with command-line: -i /home/cbu/jdftx/jdftx-git/jdftx/test/metalBulk/totalE.in -d -o totalE.out
Running on hosts (process indices):  x3102c0s13b1n0 (0-3)
Divided in process groups (process indices):  0 (0)  1 (1)  2 (2)  3 (3)
gpuInit: Found compatible cuda device 0 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 1 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 2 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 3 'NVIDIA A100-SXM4-40GB'
gpuInit: Selected device 0
Resource initialization completed at t[s]:      2.27
Run totals: 4 processes, 32 threads, 4 GPUs

Input parsed successfully to the following command list (including defaults):

basis kpoint-dependent
coords-type Lattice
core-overlap-check vector
coulomb-interaction Periodic
davidson-band-ratio 1.1
dump End None
dump-name $INPUT.$VAR
elec-cutoff 20 100
elec-eigen-algo Davidson
elec-ex-corr lda-PZ
elec-initial-magnetization 3.000000 no
elec-smearing Fermi 0.01
electronic-minimize  \
        dirUpdateScheme      FletcherReeves \
        linminMethod         DirUpdateRecommended \
        nIterations          100 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  1e-08 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
exchange-regularization WignerSeitzTruncated
fluid None
fluid-ex-corr lda-TF lda-PZ
fluid-gummel-loop 10 1.000000e-05
fluid-minimize  \
        dirUpdateScheme      PolakRibiere \
        linminMethod         DirUpdateRecommended \
        nIterations          100 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  0 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
fluid-solvent H2O 55.338 ScalarEOS \
        epsBulk 78.4 \
        pMol 0.92466 \
        epsInf 1.77 \
        Pvap 1.06736e-10 \
        sigmaBulk 4.62e-05 \
        Rvdw 2.61727 \
        Res 1.42 \
        tauNuc 343133 \
        poleEl 15 7 1
forces-output-coords Positions
ion Fe   0.000000000000000   0.000000000000000   0.000000000000000 0
ion-species GBRV/$ID_lda.uspp
ion-width 0
ionic-minimize  \
        dirUpdateScheme      L-BFGS \
        linminMethod         DirUpdateRecommended \
        nIterations          0 \
        history              15 \
        knormThreshold       0.0001 \
        maxThreshold         no \
        energyDiffThreshold  1e-06 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
kpoint   0.000000000000   0.000000000000   0.000000000000  1.00000000000000
kpoint-folding 8 8 8
latt-move-scale 1 1 1
latt-scale 1 1 1
lattice Body-Centered Cubic 5.42
lattice-minimize  \
        dirUpdateScheme      L-BFGS \
        linminMethod         DirUpdateRecommended \
        nIterations          0 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  1e-06 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
lcao-params -1 1e-06 0.01
pcm-variant GLSSA13
perturb-minimize  \
        nIterations            0 \
        algorithm              MINRES \
        residualTol            0.0001 \
        residualDiffThreshold  0.0001 \
        CGBypass               no \
        recomputeResidual      no
spintype z-spin
subspace-rotation-factor 1 yes
symmetries automatic
symmetry-threshold 0.0001

---------- Setting up symmetries ----------

Found 48 point-group symmetries of the bravais lattice
Found 48 space-group symmetries with basis
Applied RMS atom displacement 0 bohrs to make symmetries exact.

---------- Initializing the Grid ----------
R =
[        -2.71         2.71         2.71  ]
[         2.71        -2.71         2.71  ]
[         2.71         2.71        -2.71  ]
unit cell volume = 79.61
G =
[ -2.09176e-17    1.15926    1.15926  ]
[    1.15926 -2.09176e-17    1.15926  ]
[    1.15926    1.15926 -2.09176e-17  ]
Minimum fftbox size, Smin = [  24  24  24  ]
Chosen fftbox size, S = [  24  24  24  ]

---------- Initializing tighter grid for wavefunction operations ----------
R =
[        -2.71         2.71         2.71  ]
[         2.71        -2.71         2.71  ]
[         2.71         2.71        -2.71  ]
unit cell volume = 79.61
G =
[ -2.09176e-17    1.15926    1.15926  ]
[    1.15926 -2.09176e-17    1.15926  ]
[    1.15926    1.15926 -2.09176e-17  ]
Minimum fftbox size, Smin = [  20  20  20  ]
Chosen fftbox size, S = [  20  20  20  ]

---------- Exchange Correlation functional ----------
Initalized Slater LDA exchange.
Initalized Perdew-Zunger LDA correlation.

---------- Setting up pseudopotentials ----------
Width of ionic core gaussian charges (only for fluid interactions / plotting) set to 0

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/fe_lda.uspp':
  Title: Fe.  Created by USPP 7.3.6 on 15-6-15
  Reference state energy: -124.248655.  16 valence electrons in orbitals:
    |300>  occupation: 2  eigenvalue: -3.990939
    |310>  occupation: 6  eigenvalue: -2.749862
    |320>  occupation: 5  eigenvalue: -0.801787
    |400>  occupation: 2  eigenvalue: -0.525653
    |410>  occupation: 0  eigenvalue: -0.319232
  lMax: 2  lLocal: 3  QijEcut: 6
  6 projectors sampled on a log grid with 607 points:
    l: 0  eig: -3.990941  rCut: 1.55
    l: 0  eig: -0.525655  rCut: 1.55
    l: 1  eig: -2.749862  rCut: 1.55
    l: 1  eig: -0.319236  rCut: 1.55
    l: 2  eig: -0.801787  rCut: 1.65
    l: 2  eig: -0.450000  rCut: 1.65
  Partial core density with radius 0.65
  Transforming core density to a uniform radial grid of dG=0.02 with 2415 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 2415 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 2415 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.65 bohrs.

Initialized 1 species with 1 total atoms.

Folded 1 k-points by 8x8x8 to 512 k-points.

---------- Setting up k-points, bands, fillings ----------
Reduced to 29 k-points under symmetry.
Computing the number of bands and number of electrons
Calculating initial fillings.
nElectrons:  16.000000   nBands: 13   nStates: 58

----- Setting up reduced wavefunction bases (one per k-point) -----
average nbasis = 339.904 , ideal nbasis = 340.100

---------- Setting up ewald sum ----------
Optimum gaussian width for ewald sums = 1.692065 bohr.
Real space sum over 2197 unit cells with max indices [  6  6  6  ]
Reciprocal space sum over 2197 terms with max indices [  6  6  6  ]

---------- Allocating electronic variables ----------
Initializing wave functions:  linear combination of atomic orbitals
Fe pseudo-atom occupations:   s ( 2 0 )  p ( 6 0 )  d ( 8 )
        FillingsUpdate:  mu: +1.634587016  nElectrons: 16.000000  magneticMoment: [ Abs: 1.99941  Tot: +1.99941 ]
LCAOMinimize: Iter:   0  F: -121.8193667485396929  |grad|_K:  1.514e-02  alpha:  1.000e+00
        FillingsUpdate:  mu: +1.503652720  nElectrons: 16.000000  magneticMoment: [ Abs: 1.99871  Tot: +1.99871 ]
LCAOMinimize: Iter:   1  F: -122.6469178209512592  |grad|_K:  1.601e-02  alpha:  2.483e-01  linmin: -2.341e-01  cgtest:  6.374e-01  t[s]:      2.88
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 7.447573e-01.
        FillingsUpdate:  mu: +0.627962218  nElectrons: 16.000000  magneticMoment: [ Abs: 2.16764  Tot: +2.15039 ]
LCAOMinimize: Iter:   2  F: -124.3607767369256436  |grad|_K:  1.891e-03  alpha: -2.878e-01  linmin:  3.559e-01  cgtest: -5.658e-01  t[s]:      2.99
LCAOMinimize:   Bad step direction: g.d > 0.
LCAOMinimize:   Undoing step.
LCAOMinimize:   Step failed: resetting search direction.
        FillingsUpdate:  mu: +0.627962218  nElectrons: 16.000000  magneticMoment: [ Abs: 2.16764  Tot: +2.15039 ]
LCAOMinimize: Iter:   3  F: -124.3607767369256436  |grad|_K:  1.891e-03  alpha:  0.000e+00
        FillingsUpdate:  mu: +0.674651783  nElectrons: 16.000000  magneticMoment: [ Abs: 2.21311  Tot: +2.18734 ]
LCAOMinimize: Iter:   4  F: -124.3677272075534290  |grad|_K:  2.682e-04  alpha:  3.091e-01  linmin: -2.387e-01  cgtest:  6.158e-01  t[s]:      3.16
LCAOMinimize: Encountered beta<0, resetting CG.
        FillingsUpdate:  mu: +0.682238941  nElectrons: 16.000000  magneticMoment: [ Abs: 2.25180  Tot: +2.20778 ]
LCAOMinimize: Iter:   5  F: -124.3680308132388603  |grad|_K:  1.830e-04  alpha:  7.450e-01  linmin:  2.297e-03  cgtest:  3.710e-04  t[s]:      3.27
        FillingsUpdate:  mu: +0.677719450  nElectrons: 16.000000  magneticMoment: [ Abs: 2.27986  Tot: +2.22336 ]
LCAOMinimize: Iter:   6  F: -124.3681321983851262  |grad|_K:  4.755e-05  alpha:  5.411e-01  linmin: -2.871e-03  cgtest:  2.934e-02  t[s]:      3.42
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.623439e+00.
        FillingsUpdate:  mu: +0.677836634  nElectrons: 16.000000  magneticMoment: [ Abs: 2.32392  Tot: +2.26418 ]
LCAOMinimize: Iter:   7  F: -124.3681546556754398  |grad|_K:  1.138e-05  alpha:  1.741e+00  linmin: -4.041e-03  cgtest: -7.354e-03  t[s]:      3.60
        FillingsUpdate:  mu: +0.678105500  nElectrons: 16.000000  magneticMoment: [ Abs: 2.32743  Tot: +2.26875 ]
LCAOMinimize: Iter:   8  F: -124.3681553488338096  |grad|_K:  6.841e-06  alpha:  9.378e-01  linmin: -1.175e-04  cgtest: -4.011e-03  t[s]:      3.71
        FillingsUpdate:  mu: +0.677929030  nElectrons: 16.000000  magneticMoment: [ Abs: 2.32804  Tot: +2.26962 ]
LCAOMinimize: Iter:   9  F: -124.3681554570740531  |grad|_K:  3.645e-07  alpha:  4.071e-01  linmin:  1.251e-03  cgtest: -4.593e-03  t[s]:      3.88
LCAOMinimize: Converged (|Delta F|<1.000000e-06 for 2 iters).

---- Citations for features of the code used in this run ----

   Software package:
      R. Sundararaman, K. Letchworth-Weaver, K.A. Schwarz, D. Gunceler, Y. Ozhabes and T.A. Arias, 'JDFTx: software for joint density-functional theory', SoftwareX 6, 278 (2017)

   lda-PZ exchange-correlation functional:
      J.P. Perdew and A. Zunger, Phys. Rev. B 23, 5048 (1981)

   Pseudopotentials:
      KF Garrity, JW Bennett, KM Rabe and D Vanderbilt, Comput. Mater. Sci. 81, 446 (2014)

   Total energy minimization with Auxiliary Hamiltonian:
      C. Freysoldt, S. Boeck, and J. Neugebauer, Phys. Rev. B 79, 241103(R) (2009)

This list may not be complete. Please suggest additional citations or
report any other bugs at https://github.com/shankar1729/jdftx/issues

Initialization completed successfully at t[s]:      3.90

-------- Electronic minimization -----------
        FillingsUpdate:  mu: +0.677929030  nElectrons: 16.000000  magneticMoment: [ Abs: 2.32804  Tot: +2.26962 ]
ElecMinimize: Iter:   0  F: -124.368155457074039  |grad|_K:  7.412e-04  alpha:  1.000e+00
        FillingsUpdate:  mu: +0.595120411  nElectrons: 16.000000  magneticMoment: [ Abs: 2.32476  Tot: +2.26965 ]
        SubspaceRotationAdjust: set factor to 1
ElecMinimize: Iter:   1  F: -124.400899821095464  |grad|_K:  4.459e-04  alpha:  2.272e-01  linmin:  9.930e-06  t[s]:      4.15
        FillingsUpdate:  mu: +0.638083372  nElectrons: 16.000000  magneticMoment: [ Abs: 2.36139  Tot: +2.30377 ]
        SubspaceRotationAdjust: set factor to 0.902
ElecMinimize: Iter:   2  F: -124.424209400648309  |grad|_K:  2.038e-04  alpha:  4.534e-01  linmin:  2.104e-04  t[s]:      4.30
        FillingsUpdate:  mu: +0.629980077  nElectrons: 16.000000  magneticMoment: [ Abs: 2.34319  Tot: +2.28216 ]
        SubspaceRotationAdjust: set factor to 1.09
ElecMinimize: Iter:   3  F: -124.427419669225301  |grad|_K:  1.111e-04  alpha:  3.036e-01  linmin:  9.405e-06  t[s]:      4.43
        FillingsUpdate:  mu: +0.631433925  nElectrons: 16.000000  magneticMoment: [ Abs: 2.32465  Tot: +2.26089 ]
        SubspaceRotationAdjust: set factor to 1.41
ElecMinimize: Iter:   4  F: -124.428331846066271  |grad|_K:  6.489e-05  alpha:  2.827e-01  linmin:  2.880e-05  t[s]:      4.58
        FillingsUpdate:  mu: +0.627956974  nElectrons: 16.000000  magneticMoment: [ Abs: 2.29580  Tot: +2.22980 ]
        SubspaceRotationAdjust: set factor to 1.7
ElecMinimize: Iter:   5  F: -124.428715050627332  |grad|_K:  4.055e-05  alpha:  3.506e-01  linmin:  9.456e-05  t[s]:      4.73
        FillingsUpdate:  mu: +0.628827688  nElectrons: 16.000000  magneticMoment: [ Abs: 2.27590  Tot: +2.20896 ]
        SubspaceRotationAdjust: set factor to 2.16
ElecMinimize: Iter:   6  F: -124.428828052466855  |grad|_K:  1.915e-05  alpha:  2.662e-01  linmin:  2.044e-05  t[s]:      4.86
        FillingsUpdate:  mu: +0.628906047  nElectrons: 16.000000  magneticMoment: [ Abs: 2.25654  Tot: +2.18930 ]
        SubspaceRotationAdjust: set factor to 2.49
ElecMinimize: Iter:   7  F: -124.428861730078708  |grad|_K:  1.468e-05  alpha:  3.516e-01  linmin:  1.846e-05  t[s]:      5.00
        FillingsUpdate:  mu: +0.628992633  nElectrons: 16.000000  magneticMoment: [ Abs: 2.24081  Tot: +2.17374 ]
        SubspaceRotationAdjust: set factor to 3.43
ElecMinimize: Iter:   8  F: -124.428876025577040  |grad|_K:  8.448e-06  alpha:  2.540e-01  linmin:  7.269e-06  t[s]:      5.14
        FillingsUpdate:  mu: +0.628989789  nElectrons: 16.000000  magneticMoment: [ Abs: 2.22891  Tot: +2.16254 ]
        SubspaceRotationAdjust: set factor to 4.21
ElecMinimize: Iter:   9  F: -124.428881926881715  |grad|_K:  5.623e-06  alpha:  3.165e-01  linmin:  2.095e-05  t[s]:      5.27
        FillingsUpdate:  mu: +0.628902355  nElectrons: 16.000000  magneticMoment: [ Abs: 2.22395  Tot: +2.15822 ]
        SubspaceRotationAdjust: set factor to 3.86
ElecMinimize: Iter:  10  F: -124.428883709030202  |grad|_K:  2.891e-06  alpha:  2.160e-01  linmin: -1.111e-06  t[s]:      5.40
        FillingsUpdate:  mu: +0.629097650  nElectrons: 16.000000  magneticMoment: [ Abs: 2.22283  Tot: +2.15760 ]
        SubspaceRotationAdjust: set factor to 3.3
ElecMinimize: Iter:  11  F: -124.428884282926148  |grad|_K:  2.518e-06  alpha:  2.624e-01  linmin:  1.143e-06  t[s]:      5.54
        FillingsUpdate:  mu: +0.628890223  nElectrons: 16.000000  magneticMoment: [ Abs: 2.22252  Tot: +2.15745 ]
        SubspaceRotationAdjust: set factor to 2.48
ElecMinimize: Iter:  12  F: -124.428884490277071  |grad|_K:  1.462e-06  alpha:  1.250e-01  linmin: -2.245e-06  t[s]:      5.68
        FillingsUpdate:  mu: +0.628934359  nElectrons: 16.000000  magneticMoment: [ Abs: 2.22303  Tot: +2.15800 ]
        SubspaceRotationAdjust: set factor to 3.14
ElecMinimize: Iter:  13  F: -124.428884621586278  |grad|_K:  8.239e-07  alpha:  2.345e-01  linmin: -4.513e-06  t[s]:      5.82
        FillingsUpdate:  mu: +0.628975602  nElectrons: 16.000000  magneticMoment: [ Abs: 2.22351  Tot: +2.15845 ]
        SubspaceRotationAdjust: set factor to 3.31
ElecMinimize: Iter:  14  F: -124.428884665461283  |grad|_K:  6.117e-07  alpha:  2.468e-01  linmin: -9.867e-07  t[s]:      5.96
        FillingsUpdate:  mu: +0.628925601  nElectrons: 16.000000  magneticMoment: [ Abs: 2.22375  Tot: +2.15866 ]
        SubspaceRotationAdjust: set factor to 2.63
ElecMinimize: Iter:  15  F: -124.428884680828745  |grad|_K:  3.869e-07  alpha:  1.568e-01  linmin:  4.775e-07  t[s]:      6.10
cbu@x3102c0s13b1n0:~/jdftx/build/test/metalBulk>
shankar1729 commented 4 months ago

Doing the MPI tests through cmake could be a bit tricky in getting all the correct settings. Try your typical solvated slab calculation with multiple GPUs. If that works correctly, you should be set. Not worth getting this test to work.

ColinBundschu commented 4 months ago

Looks like |grad|_K is -nan:

*************** JDFTx 1.7.0 (git hash aa096dda) ***************

Start date and time: Sat May 18 21:41:20 2024
Executable /home/cbu/jdftx/build/jdftx_gpu with command-line: -i /home/cbu/dft_out/FeNC/clean/661.in -o /home/cbu/dft_out/FeNC/clean/661.out
Running on hosts (process indices):  x3005c0s37b0n0 (0-3)  x3005c0s37b1n0 (4-7)
Divided in process groups (process indices):  0 (0)  1 (1)  2 (2)  3 (3)  4 (4)  5 (5)  6 (6)  7 (7)
gpuInit: Found compatible cuda device 0 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 1 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 2 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 3 'NVIDIA A100-SXM4-40GB'
gpuInit: Selected device 0
Resource initialization completed at t[s]:      0.65
Run totals: 8 processes, 64 threads, 8 GPUs

Input parsed successfully to the following command list (including defaults):

add-U   Fe d 0.129
basis kpoint-dependent
coords-type Lattice
core-overlap-check vector
coulomb-interaction Periodic
davidson-band-ratio 1.1
dump End State Stress
dump Ionic IonicPositions Stress
dump-name $INPUT.$VAR
elec-cutoff 20 100
elec-eigen-algo Davidson
elec-ex-corr gga-PBE
electronic-minimize  \
        dirUpdateScheme      FletcherReeves \
        linminMethod         DirUpdateRecommended \
        nIterations          150 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  1e-06 \
        nEnergyDiff          5 \
        alphaTstart          0.1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
exchange-regularization WignerSeitzTruncated
fluid None
fluid-ex-corr lda-TF lda-PZ
fluid-gummel-loop 10 1.000000e-05
fluid-minimize  \
        dirUpdateScheme      PolakRibiere \
        linminMethod         DirUpdateRecommended \
        nIterations          100 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  0 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
fluid-solvent H2O 55.338 ScalarEOS \
        epsBulk 78.4 \
        pMol 0.92466 \
        epsInf 1.77 \
        Pvap 1.06736e-10 \
        sigmaBulk 4.62e-05 \
        Rvdw 2.61727 \
        Res 1.42 \
        tauNuc 343133 \
        poleEl 15 7 1
forces-output-coords Positions
initial-state 661.$VAR
ion C   0.000000000000000   0.999700000000000   0.446000000000000 1
ion C   0.000000000000000   0.499800000000000   0.444300000000000 1
ion C   0.500000000000000   0.000300000000000   0.446200000000000 1
ion C   0.438100000000000   0.127400000000000   0.446100000000000 1
ion C   0.436200000000000   0.618500000000000   0.446400000000000 1
ion C   0.937400000000000   0.124500000000000   0.445400000000000 1
ion C   0.937700000000000   0.625100000000000   0.444700000000000 1
ion C   0.374700000000000   0.252500000000000   0.445900000000000 1
ion C   0.375400000000000   0.747600000000000   0.446100000000000 1
ion C   0.875300000000000   0.249400000000000   0.444800000000000 1
ion C   0.875000000000000   0.750100000000000   0.445300000000000 1
ion C   0.309600000000000   0.375000000000000   0.445400000000000 1
ion C   0.313200000000000   0.874300000000000   0.445900000000000 1
ion C   0.814000000000000   0.374500000000000   0.444600000000000 1
ion C   0.812100000000000   0.874900000000000   0.445600000000000 1
ion C   0.250700000000000   0.999900000000000   0.445700000000000 1
ion C   0.247500000000000   0.499500000000000   0.445100000000000 1
ion C   0.749300000000000   0.999900000000000   0.445600000000000 1
ion C   0.752400000000000   0.499500000000000   0.444900000000000 1
ion C   0.187600000000000   0.124600000000000   0.445300000000000 1
ion C   0.186700000000000   0.625000000000000   0.445100000000000 1
ion C   0.686900000000000   0.125700000000000   0.445700000000000 1
ion C   0.688800000000000   0.624300000000000   0.445600000000000 1
ion C   0.124600000000000   0.249400000000000   0.444800000000000 1
ion C   0.125000000000000   0.750100000000000   0.445500000000000 1
ion C   0.625200000000000   0.252500000000000   0.445900000000000 1
ion C   0.624500000000000   0.747600000000000   0.446000000000000 1
ion C   0.062000000000000   0.374500000000000   0.444400000000000 1
ion C   0.062600000000000   0.875000000000000   0.446000000000000 1
ion C   0.561900000000000   0.873300000000000   0.446100000000000 1
ion C   0.125200000000000   0.999700000000000   0.445800000000000 1
ion C   0.123900000000000   0.499800000000000   0.444500000000000 1
ion C   0.624200000000000   0.000200000000000   0.446000000000000 1
ion C   0.629100000000000   0.499800000000000   0.445900000000000 1
ion C   0.062500000000000   0.124500000000000   0.445400000000000 1
ion C   0.062200000000000   0.625100000000000   0.444900000000000 1
ion C   0.561800000000000   0.127400000000000   0.446100000000000 1
ion C   0.563800000000000   0.618500000000000   0.446300000000000 1
ion C   0.000000000000000   0.249300000000000   0.444800000000000 1
ion C   0.000000000000000   0.750100000000000   0.445500000000000 1
ion C   0.500000000000000   0.256700000000000   0.446300000000000 1
ion C   0.500000000000000   0.745000000000000   0.446300000000000 1
ion C   0.438000000000000   0.873300000000000   0.446200000000000 1
ion C   0.937900000000000   0.374500000000000   0.444300000000000 1
ion C   0.937300000000000   0.875000000000000   0.445900000000000 1
ion C   0.375800000000000   0.000200000000000   0.446000000000000 1
ion C   0.370800000000000   0.499800000000000   0.445900000000000 1
ion C   0.874700000000000   0.999700000000000   0.445700000000000 1
ion C   0.876000000000000   0.499800000000000   0.444400000000000 1
ion C   0.313000000000000   0.125700000000000   0.445700000000000 1
ion C   0.311200000000000   0.624300000000000   0.445700000000000 1
ion C   0.812300000000000   0.124600000000000   0.445300000000000 1
ion C   0.813300000000000   0.625000000000000   0.444900000000000 1
ion C   0.249300000000000   0.249700000000000   0.445200000000000 1
ion C   0.250000000000000   0.749700000000000   0.445700000000000 1
ion C   0.750600000000000   0.249700000000000   0.445200000000000 1
ion C   0.749900000000000   0.749700000000000   0.445500000000000 1
ion C   0.185900000000000   0.374500000000000   0.444700000000000 1
ion C   0.187900000000000   0.874900000000000   0.445700000000000 1
ion C   0.690300000000000   0.375000000000000   0.445400000000000 1
ion C   0.686800000000000   0.874300000000000   0.445800000000000 1
ion C   0.062600000000000   0.041200000000000   0.445800000000000 1
ion C   0.062000000000000   0.541700000000000   0.444500000000000 1
ion C   0.561900000000000   0.042900000000000   0.446100000000000 1
ion C   0.000000000000000   0.166100000000000   0.445200000000000 1
ion C   0.000000000000000   0.666900000000000   0.445000000000000 1
ion C   0.500000000000000   0.171200000000000   0.446200000000000 1
ion C   0.500000000000000   0.659500000000000   0.446400000000000 1
ion C   0.436200000000000   0.297700000000000   0.446200000000000 1
ion C   0.438100000000000   0.788800000000000   0.446300000000000 1
ion C   0.937700000000000   0.291100000000000   0.444600000000000 1
ion C   0.937400000000000   0.791700000000000   0.445600000000000 1
ion C   0.370800000000000   0.416400000000000   0.445900000000000 1
ion C   0.375800000000000   0.916000000000000   0.446000000000000 1
ion C   0.876000000000000   0.416400000000000   0.444400000000000 1
ion C   0.874700000000000   0.916500000000000   0.445800000000000 1
ion C   0.313200000000000   0.041900000000000   0.445800000000000 1
ion C   0.309600000000000   0.541200000000000   0.445500000000000 1
ion C   0.812100000000000   0.041300000000000   0.445500000000000 1
ion C   0.814000000000000   0.541700000000000   0.444700000000000 1
ion C   0.250000000000000   0.166500000000000   0.445400000000000 1
ion C   0.249300000000000   0.666500000000000   0.445500000000000 1
ion C   0.749900000000000   0.166500000000000   0.445400000000000 1
ion C   0.750600000000000   0.666500000000000   0.445300000000000 1
ion C   0.186700000000000   0.291200000000000   0.444800000000000 1
ion C   0.187600000000000   0.791600000000000   0.445700000000000 1
ion C   0.688800000000000   0.291900000000000   0.445500000000000 1
ion C   0.686900000000000   0.790500000000000   0.445800000000000 1
ion C   0.123900000000000   0.416400000000000   0.444500000000000 1
ion C   0.125200000000000   0.916500000000000   0.445800000000000 1
ion C   0.629100000000000   0.416400000000000   0.445800000000000 1
ion C   0.624200000000000   0.916000000000000   0.446000000000000 1
ion C   0.187900000000000   0.041300000000000   0.445500000000000 1
ion C   0.185900000000000   0.541700000000000   0.444900000000000 1
ion C   0.686800000000000   0.041900000000000   0.445700000000000 1
ion C   0.690300000000000   0.541200000000000   0.445400000000000 1
ion C   0.125000000000000   0.166100000000000   0.445200000000000 1
ion C   0.124600000000000   0.666800000000000   0.445200000000000 1
ion C   0.624500000000000   0.168600000000000   0.445900000000000 1
ion C   0.625200000000000   0.663700000000000   0.446000000000000 1
ion C   0.062200000000000   0.291100000000000   0.444600000000000 1
ion C   0.062500000000000   0.791700000000000   0.445700000000000 1
ion C   0.563800000000000   0.297700000000000   0.446200000000000 1
ion C   0.561800000000000   0.788800000000000   0.446200000000000 1
ion C   0.000000000000000   0.416400000000000   0.444300000000000 1
ion C   0.000000000000000   0.916500000000000   0.446100000000000 1
ion C   0.500000000000000   0.915900000000000   0.446200000000000 1
ion C   0.438000000000000   0.042900000000000   0.446100000000000 1
ion C   0.937300000000000   0.041200000000000   0.445800000000000 1
ion C   0.937900000000000   0.541700000000000   0.444400000000000 1
ion C   0.375400000000000   0.168600000000000   0.445900000000000 1
ion C   0.374700000000000   0.663700000000000   0.446100000000000 1
ion C   0.875000000000000   0.166100000000000   0.445200000000000 1
ion C   0.875300000000000   0.666800000000000   0.445000000000000 1
ion C   0.311200000000000   0.291900000000000   0.445500000000000 1
ion C   0.313000000000000   0.790500000000000   0.445900000000000 1
ion C   0.813300000000000   0.291200000000000   0.444800000000000 1
ion C   0.812300000000000   0.791600000000000   0.445500000000000 1
ion C   0.247500000000000   0.416700000000000   0.445000000000000 1
ion C   0.250700000000000   0.916300000000000   0.445800000000000 1
ion C   0.752400000000000   0.416700000000000   0.444900000000000 1
ion C   0.749300000000000   0.916300000000000   0.445700000000000 1
ion N   0.567200000000000   0.378800000000000   0.446300000000000 1
ion N   0.432700000000000   0.378800000000000   0.446300000000000 1
ion N   0.567200000000000   0.537400000000000   0.446300000000000 1
ion N   0.432700000000000   0.537400000000000   0.446400000000000 1
ion Fe   0.500000000000000   0.458100000000000   0.446600000000000 1
ion-species GBRV/$ID_pbe.uspp
ion-width 0
ionic-minimize  \
        dirUpdateScheme      L-BFGS \
        linminMethod         DirUpdateRecommended \
        nIterations          0 \
        history              15 \
        knormThreshold       0.0001 \
        maxThreshold         no \
        energyDiffThreshold  1e-06 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
kpoint   0.500000000000   0.500000000000   0.000000000000  1.00000000000000
kpoint-folding 6 6 1
latt-move-scale 1 1 1
latt-scale 1 1 1
lattice Orthorhombic 37.2276 32.3143 41.574
lattice-minimize  \
        dirUpdateScheme      L-BFGS \
        linminMethod         DirUpdateRecommended \
        nIterations          0 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  1e-06 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
lcao-params 150 1e-06 0.001
pcm-variant GLSSA13
perturb-minimize  \
        nIterations            0 \
        algorithm              MINRES \
        residualTol            0.0001 \
        residualDiffThreshold  0.0001 \
        CGBypass               no \
        recomputeResidual      no
spintype z-spin
subspace-rotation-factor 1 yes
symmetries automatic
symmetry-threshold 0.0001

---------- Setting up symmetries ----------

Found 8 point-group symmetries of the bravais lattice
Found 1 space-group symmetries with basis
Applied RMS atom displacement 0 bohrs to make symmetries exact.

---------- Initializing the Grid ----------
R =
[      37.2276            0            0  ]
[            0      32.3143            0  ]
[            0            0       41.574  ]
unit cell volume = 50012.9
G =
[   0.168778          0          0  ]
[          0    0.19444          0  ]
[          0          0   0.151133  ]
Minimum fftbox size, Smin = [  168  148  188  ]
Chosen fftbox size, S = [  168  150  192  ]

---------- Initializing tighter grid for wavefunction operations ----------
R =
[      37.2276            0            0  ]
[            0      32.3143            0  ]
[            0            0       41.574  ]
unit cell volume = 50012.9
G =
[   0.168778          0          0  ]
[          0    0.19444          0  ]
[          0          0   0.151133  ]
Minimum fftbox size, Smin = [  152  132  168  ]
Chosen fftbox size, S = [  160  140  168  ]

---------- Exchange Correlation functional ----------
Initalized PBE GGA exchange.
Initalized PBE GGA correlation.

---------- Setting up pseudopotentials ----------
Width of ionic core gaussian charges (only for fluid interactions / plotting) set to 0

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/c_pbe.uspp':
  Title: C.  Created by USPP 7.3.6 on 3-2-2014
  Reference state energy: -5.406344.  4 valence electrons in orbitals:
    |200>  occupation: 2  eigenvalue: -0.504890
    |210>  occupation: 2  eigenvalue: -0.194356
  lMax: 1  lLocal: 2  QijEcut: 5
  4 projectors sampled on a log grid with 503 points:
    l: 0  eig: -0.504890  rCut: 1.3
    l: 0  eig: 0.000000  rCut: 1.3
    l: 1  eig: -0.194357  rCut: 1.3
    l: 1  eig: 0.000000  rCut: 1.3
  Partial core density with radius 1.1
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.30 bohrs.

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/n_pbe.uspp':
  Title: N.  Created by USPP 7.3.6 on 3-2-2014
  Reference state energy: -9.763716.  5 valence electrons in orbitals:
    |200>  occupation: 2  eigenvalue: -0.681964
    |210>  occupation: 3  eigenvalue: -0.260726
  lMax: 1  lLocal: 2  QijEcut: 6
  4 projectors sampled on a log grid with 491 points:
    l: 0  eig: -0.681964  rCut: 1.15
    l: 0  eig: 0.000000  rCut: 1.15
    l: 1  eig: -0.260729  rCut: 1.2
    l: 1  eig: 0.500000  rCut: 1.2
  Partial core density with radius 0.8
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.20 bohrs.

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/fe_pbe.uspp':
  Title: Fe.  Created by USPP 7.3.6 on 15-6-15
  Reference state energy: -124.316937.  16 valence electrons in orbitals:
    |300>  occupation: 2  eigenvalue: -4.008817
    |310>  occupation: 6  eigenvalue: -2.753096
    |320>  occupation: 5  eigenvalue: -0.791264
    |400>  occupation: 2  eigenvalue: -0.516384
    |410>  occupation: 0  eigenvalue: -0.312222
  lMax: 2  lLocal: 3  QijEcut: 6
  6 projectors sampled on a log grid with 607 points:
    l: 0  eig: -4.008819  rCut: 1.55
    l: 0  eig: -0.516386  rCut: 1.55
    l: 1  eig: -2.753096  rCut: 1.55
    l: 1  eig: -0.312226  rCut: 1.55
    l: 2  eig: -0.791264  rCut: 1.65
    l: 2  eig: -0.450000  rCut: 1.65
  Partial core density with radius 0.65
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.65 bohrs.

Initialized 3 species with 127 total atoms.

Folded 1 k-points by 6x6x1 to 36 k-points.

---------- Setting up k-points, bands, fillings ----------
Adding inversion symmetry to k-mesh for non-inversion-symmetric unit cell.
Reduced to 18 k-points under symmetry.
Computing the number of bands and number of electrons
Calculating initial fillings.
nElectrons: 524.000000   nBands: 262   nStates: 36

----- Setting up reduced wavefunction bases (one per k-point) -----
average nbasis = 213652.444 , ideal nbasis = 213658.714

---------- Setting up ewald sum ----------
Optimum gaussian width for ewald sums = 6.555965 bohr.
Real space sum over 343 unit cells with max indices [  3  3  3  ]
Reciprocal space sum over 9177 terms with max indices [  10  9  11  ]

---------- Allocating electronic variables ----------
Initializing wave functions:  linear combination of atomic orbitals
C pseudo-atom occupations:   s ( 2 )  p ( 2 )
N pseudo-atom occupations:   s ( 2 )  p ( 3 )
Fe pseudo-atom occupations:   s ( 2 0 )  p ( 6 0 )  d ( 8 )
        FillingsUpdate:  mu: -0.200546791  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00301  Tot: -0.00000 ]
LCAOMinimize: Iter:   0  Etot: -836.1900001330973282  |grad|_K:  2.578e-03  alpha:  1.000e+00
        FillingsUpdate:  mu: -0.239012044  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00300  Tot: -0.00000 ]
LCAOMinimize: Iter:   1  Etot: -840.1981605887019668  |grad|_K:  1.408e-03  alpha:  8.797e-02  linmin: -1.147e-01  cgtest:  1.087e+00  t[s]:     54.27
LCAOMinimize: Encountered beta<0, resetting CG.
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 2.638958e-01.
        FillingsUpdate:  mu: -0.251775431  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00300  Tot: -0.00000 ]
LCAOMinimize: Iter:   2  Etot: -841.9010627750307094  |grad|_K:  1.450e-03  alpha: -3.862e+00  linmin: -1.559e-01  cgtest:  7.906e-01  t[s]:     68.25
LCAOMinimize:   Step increased Etot by 2.610224e+00, reducing alpha to 1.969309e-02.
        FillingsUpdate:  mu: -0.254365212  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00301  Tot: +0.00000 ]
LCAOMinimize: Iter:   3  Etot: -842.5367290380218037  |grad|_K:  1.783e-03  alpha:  1.969e-02  linmin: -1.586e-01  cgtest:  7.695e-01  t[s]:     91.29
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 5.907928e-02.
        FillingsUpdate:  mu: -0.252922317  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00312  Tot: +0.00000 ]
LCAOMinimize: Iter:   4  Etot: -850.1436035357298806  |grad|_K:  5.292e-03  alpha: -1.613e-03  linmin: -2.033e-01  cgtest:  1.624e-01  t[s]:    105.25
LCAOMinimize:   Step increased Etot by 1.716467e+02, reducing alpha to 4.884919e-02.
LCAOMinimize:   Step increased Etot by 3.845617e+01, reducing alpha to 4.884919e-03.
LCAOMinimize:   Step increased Etot by 7.303828e+00, reducing alpha to 4.884919e-04.
LCAOMinimize:   Step failed to reduce Etot after 3 attempts. Quitting step.
LCAOMinimize:   Undoing step.
LCAOMinimize:   Step failed: resetting search direction.
        FillingsUpdate:  mu: -0.165736625  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00301  Tot: -0.00000 ]
LCAOMinimize: Iter:   5  Etot: -842.9991513441606230  |grad|_K:  2.345e-03  alpha:  4.885e-04
        FillingsUpdate:  mu: -0.180898727  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00303  Tot: +0.00000 ]
LCAOMinimize: Iter:   6  Etot: -850.5886285733228078  |grad|_K:  2.046e-03  alpha:  4.401e-02  linmin: -1.514e-01  cgtest:  5.380e-01  t[s]:    160.94
        FillingsUpdate:  mu: -0.200430338  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00304  Tot: +0.00000 ]
LCAOMinimize: Iter:   7  Etot: -851.2977269890628804  |grad|_K:  3.372e-03  alpha:  6.784e-03  linmin: -1.437e-01  cgtest:  5.108e-01  t[s]:    174.55
        FillingsUpdate:  mu: -0.234056557  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00308  Tot: +0.00000 ]
LCAOMinimize: Iter:   8  Etot: -852.9851686957016454  |grad|_K:  7.650e-04  alpha:  6.725e-03  linmin:  1.202e-01  cgtest: -5.408e-01  t[s]:    188.38
LCAOMinimize:   Bad step direction: g.d > 0.
LCAOMinimize:   Undoing step.
LCAOMinimize:   Step failed: resetting search direction.
        FillingsUpdate:  mu: -0.234056557  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00308  Tot: +0.00000 ]
LCAOMinimize: Iter:   9  Etot: -852.9851686957016454  |grad|_K:  7.650e-04  alpha:  0.000e+00
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.017623e-02.
        FillingsUpdate:  mu: -0.211377672  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00308  Tot: +0.00000 ]
LCAOMinimize: Iter:  10  Etot: -853.0546119021586264  |grad|_K:  2.782e-04  alpha:  2.362e-02  linmin: -6.009e-02  cgtest:  6.822e-01  t[s]:    215.90
LCAOMinimize: Encountered beta<0, resetting CG.
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 7.086069e-02.
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.125821e-01.
        FillingsUpdate:  mu: -0.203668714  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00312  Tot: +0.00000 ]
LCAOMinimize: Iter:  11  Etot: -853.1418900666448053  |grad|_K:  1.305e-04  alpha:  2.332e-01  linmin: -6.344e-04  cgtest:  2.420e-02  t[s]:    238.49
        FillingsUpdate:  mu: -0.230562002  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00314  Tot: +0.00000 ]
LCAOMinimize: Iter:  12  Etot: -853.1474554894697349  |grad|_K:  7.175e-05  alpha:  5.839e-02  linmin: -3.812e-02  cgtest:  3.335e-01  t[s]:    252.36
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.751808e-01.
        FillingsUpdate:  mu: -0.219858020  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00321  Tot: +0.00000 ]
LCAOMinimize: Iter:  13  Etot: -853.1551342328848477  |grad|_K:  4.474e-05  alpha:  2.860e-01  linmin: -9.727e-03  cgtest: -7.244e-02  t[s]:    270.57
        FillingsUpdate:  mu: -0.227966251  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00325  Tot: +0.00000 ]
LCAOMinimize: Iter:  14  Etot: -853.1566177379488636  |grad|_K:  4.766e-05  alpha:  1.461e-01  linmin:  1.720e-04  cgtest: -3.259e-02  t[s]:    284.35
        FillingsUpdate:  mu: -0.227081884  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00335  Tot: -0.00000 ]
LCAOMinimize: Iter:  15  Etot: -853.1584875545073601  |grad|_K:  1.270e-05  alpha:  1.703e-01  linmin: -6.893e-03  cgtest:  1.597e-01  t[s]:    298.11
        FillingsUpdate:  mu: -0.225802672  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00345  Tot: +0.00000 ]
LCAOMinimize: Iter:  16  Etot: -853.1587569867698448  |grad|_K:  6.717e-06  alpha:  3.435e-01  linmin: -9.258e-04  cgtest:  5.796e-03  t[s]:    311.81
        FillingsUpdate:  mu: -0.226995749  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00350  Tot: -0.00000 ]
LCAOMinimize: Iter:  17  Etot: -853.1587771921241483  |grad|_K:  4.657e-06  alpha:  9.312e-02  linmin:  8.685e-04  cgtest: -1.167e-02  t[s]:    325.65
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.793552e-01.
        FillingsUpdate:  mu: -0.226827490  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00370  Tot: -0.00000 ]
LCAOMinimize: Iter:  18  Etot: -853.1588075401823517  |grad|_K:  2.805e-06  alpha:  2.915e-01  linmin: -2.952e-04  cgtest:  4.398e-03  t[s]:    343.76
        FillingsUpdate:  mu: -0.226668169  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00410  Tot: +0.00000 ]
LCAOMinimize: Iter:  19  Etot: -853.1588235148224157  |grad|_K:  1.228e-06  alpha:  4.180e-01  linmin: -3.157e-03  cgtest: -8.332e-03  t[s]:    357.68
        FillingsUpdate:  mu: -0.226894647  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00435  Tot: +0.00000 ]
LCAOMinimize: Iter:  20  Etot: -853.1588248546177056  |grad|_K:  1.196e-06  alpha:  1.825e-01  linmin:  1.295e-04  cgtest: -1.873e-02  t[s]:    371.50
        FillingsUpdate:  mu: -0.226782994  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00466  Tot: -0.00000 ]
LCAOMinimize: Iter:  21  Etot: -853.1588255970707451  |grad|_K:  3.131e-07  alpha:  1.078e-01  linmin:  1.719e-04  cgtest:  3.557e-02  t[s]:    385.34
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 3.233069e-01.
        FillingsUpdate:  mu: -0.226742961  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00569  Tot: -0.00000 ]
LCAOMinimize: Iter:  22  Etot: -853.1588258265810509  |grad|_K:  3.687e-07  alpha:  4.847e-01  linmin: -8.805e-05  cgtest: -1.778e-03  t[s]:    403.53
LCAOMinimize: Converged (|Delta Etot|<1.000000e-06 for 2 iters).

---- Citations for features of the code used in this run ----

   Software package:
      R. Sundararaman, K. Letchworth-Weaver, K.A. Schwarz, D. Gunceler, Y. Ozhabes and T.A. Arias, 'JDFTx: software for joint density-functional theory', SoftwareX 6, 278 (2017)

   Simplified rotationally-invariant DFT+U:
      S. L. Dudarev et al., Phys. Rev. B 57, 1505 (1998)

   gga-PBE exchange-correlation functional:
      J.P. Perdew, K. Burke and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996)

   Pseudopotentials:
      KF Garrity, JW Bennett, KM Rabe and D Vanderbilt, Comput. Mater. Sci. 81, 446 (2014)

   Total energy minimization:
      T.A. Arias, M.C. Payne and J.D. Joannopoulos, Phys. Rev. Lett. 69, 1077 (1992)

This list may not be complete. Please suggest additional citations or
report any other bugs at https://github.com/shankar1729/jdftx/issues

Initialization completed successfully at t[s]:    404.85

-------- Electronic minimization -----------
ElecMinimize: Iter:   0  Etot: -853.158814195010450  |grad|_K:       -nan  alpha:  1.000e-01
ElecMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 3.000000e-01.
ElecMinimize: Iter:   1  Etot: -860.126768894767110  |grad|_K:       -nan  alpha:  6.646e-01  linmin: -3.184e-02  t[s]:    428.30
ElecMinimize: Iter:   2  Etot: -860.535720082702596  |grad|_K:       -nan  alpha:  2.466e-01  linmin: -2.781e-02  t[s]:    441.36
ElecMinimize: Iter:   3  Etot: -860.839426000230560  |grad|_K:       -nan  alpha:  5.964e-01  linmin:  4.448e-03  t[s]:    454.69
ElecMinimize: Iter:   4  Etot: -860.903514655694721  |grad|_K:       -nan  alpha:  7.473e-01  linmin: -4.785e-04  t[s]:    467.91
ElecMinimize: Iter:   5  Etot: -860.919294704711660  |grad|_K:       -nan  alpha:  2.900e-01  linmin: -1.549e-03  t[s]:    481.30
ElecMinimize: Iter:   6  Etot: -860.932856224269926  |grad|_K:       -nan  alpha:  7.115e-01  linmin: -8.372e-04  t[s]:    494.61
ElecMinimize: Iter:   7  Etot: -860.936285409196103  |grad|_K:       -nan  alpha:  5.130e-01  linmin: -9.076e-06  t[s]:    507.79
ElecMinimize: Iter:   8  Etot: -860.938176838846402  |grad|_K:       -nan  alpha:  6.343e-01  linmin: -6.049e-04  t[s]:    520.89
ElecMinimize: Iter:   9  Etot: -860.939425571850620  |grad|_K:       -nan  alpha:  4.036e-01  linmin: -9.194e-05  t[s]:    534.03
ElecMinimize: Iter:  10  Etot: -860.940488652698150  |grad|_K:       -nan  alpha:  6.079e-01  linmin: -8.496e-04  t[s]:    547.12
ElecMinimize: Iter:  11  Etot: -860.941202884638528  |grad|_K:       -nan  alpha:  3.837e-01  linmin: -3.701e-05  t[s]:    560.29
ElecMinimize: Iter:  12  Etot: -860.941897863199074  |grad|_K:       -nan  alpha:  4.063e-01  linmin:  1.597e-04  t[s]:    573.74
ElecMinimize: Iter:  13  Etot: -860.942320968067861  |grad|_K:       -nan  alpha:  5.281e-01  linmin: -1.069e-03  t[s]:    587.33
ElecMinimize: Iter:  14  Etot: -860.942610815933563  |grad|_K:       -nan  alpha:  3.899e-01  linmin:  1.339e-04  t[s]:    600.83
ElecMinimize: Iter:  15  Etot: -860.942819161593093  |grad|_K:       -nan  alpha:  4.955e-01  linmin: -4.685e-04  t[s]:    614.56
ElecMinimize: Iter:  16  Etot: -860.942965324400234  |grad|_K:       -nan  alpha:  5.544e-01  linmin: -1.161e-04  t[s]:    628.24
ElecMinimize: Iter:  17  Etot: -860.943079714542478  |grad|_K:       -nan  alpha:  3.722e-01  linmin: -1.836e-06  t[s]:    641.96
ElecMinimize: Iter:  18  Etot: -860.943177905790094  |grad|_K:       -nan  alpha:  5.355e-01  linmin: -2.306e-04  t[s]:    655.27
ElecMinimize: Iter:  19  Etot: -860.943263661583615  |grad|_K:       -nan  alpha:  6.000e-01  linmin:  6.665e-05  t[s]:    668.95
ElecMinimize: Iter:  20  Etot: -860.943324004911460  |grad|_K:       -nan  alpha:  4.077e-01  linmin: -1.384e-06  t[s]:    682.39
ElecMinimize: Iter:  21  Etot: -860.943370442354649  |grad|_K:       -nan  alpha:  4.058e-01  linmin: -4.647e-05  t[s]:    695.81
ElecMinimize: Iter:  22  Etot: -860.943411328148841  |grad|_K:       -nan  alpha:  5.031e-01  linmin:  5.003e-05  t[s]:    709.31
ElecMinimize: Iter:  23  Etot: -860.943438587161836  |grad|_K:       -nan  alpha:  4.369e-01  linmin:  3.554e-06  t[s]:    722.70
ElecMinimize: Iter:  24  Etot: -860.943463345898863  |grad|_K:       -nan  alpha:  4.155e-01  linmin: -1.323e-05  t[s]:    736.04
ElecMinimize: Iter:  25  Etot: -860.943489840625034  |grad|_K:       -nan  alpha:  5.926e-01  linmin:  4.896e-05  t[s]:    749.43
ElecMinimize: Iter:  26  Etot: -860.943509150914565  |grad|_K:       -nan  alpha:  4.958e-01  linmin: -1.077e-05  t[s]:    762.94
ElecMinimize: Iter:  27  Etot: -860.943532114248683  |grad|_K:       -nan  alpha:  4.963e-01  linmin: -5.324e-07  t[s]:    776.26
ElecMinimize: Iter:  28  Etot: -860.943565807664072  |grad|_K:       -nan  alpha:  6.939e-01  linmin:  9.932e-05  t[s]:    789.85
ElecMinimize: Iter:  29  Etot: -860.943629908744356  |grad|_K:       -nan  alpha:  7.511e-01  linmin:  3.266e-05  t[s]:    803.57
ElecMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.253197e+00.
ElecMinimize: Iter:  30  Etot: -860.944091472008836  |grad|_K:       -nan  alpha:  2.184e+00  linmin:  5.442e-03  t[s]:    821.24
ElecMinimize:   Step increased Etot by 1.126865e-02, reducing alpha to 3.495264e-03.
ElecMinimize:   Step increased Etot by 1.144622e-02, reducing alpha to 3.495264e-04.
ElecMinimize:   Step increased Etot by 1.147192e-02, reducing alpha to 3.495264e-05.
ElecMinimize:   Step failed to reduce Etot after 3 attempts. Quitting step.
ElecMinimize:   Undoing step.
ElecMinimize:   Step failed: resetting search direction.
ElecMinimize: Iter:  31  Etot: -860.932619261507625  |grad|_K:       -nan  alpha:  3.495e-05
ElecMinimize: Iter:  32  Etot: -860.941860520131399  |grad|_K:       -nan  alpha:  2.362e-01  linmin: -1.207e-01  t[s]:    872.88
ElecMinimize: Iter:  33  Etot: -860.948586491640071  |grad|_K:       -nan  alpha:  4.920e-01  linmin:  1.646e-03  t[s]:    886.20
ElecMinimize: Iter:  34  Etot: -860.951352173658051  |grad|_K:       -nan  alpha:  4.354e-01  linmin:  2.081e-05  t[s]:    899.62
ElecMinimize: Iter:  35  Etot: -860.955693358787016  |grad|_K:       -nan  alpha:  5.349e-01  linmin:  2.874e-04  t[s]:    913.29
ElecMinimize: Iter:  36  Etot: -860.960161673671905  |grad|_K:       -nan  alpha:  7.449e-01  linmin:  1.753e-03  t[s]:    926.55
ElecMinimize: Iter:  37  Etot: -860.963188429329875  |grad|_K:       -nan  alpha:  3.703e-01  linmin: -1.737e-03  t[s]:    939.89
ElecMinimize: Iter:  38  Etot: -860.965834047569615  |grad|_K:       -nan  alpha:  5.107e-01  linmin:  1.005e-03  t[s]:    953.38
ElecMinimize: Iter:  39  Etot: -860.968169607308596  |grad|_K:       -nan  alpha:  5.055e-01  linmin:  1.987e-04  t[s]:    966.51
ElecMinimize: Iter:  40  Etot: -860.969822109628581  |grad|_K:       -nan  alpha:  3.548e-01  linmin: -7.369e-05  t[s]:    979.91
ElecMinimize: Iter:  41  Etot: -860.971119759735416  |grad|_K:       -nan  alpha:  4.897e-01  linmin: -2.510e-04  t[s]:    993.23
ElecMinimize: Iter:  42  Etot: -860.972188097216304  |grad|_K:       -nan  alpha:  4.404e-01  linmin: -4.069e-05  t[s]:   1006.53
ElecMinimize: Iter:  43  Etot: -860.973017529614140  |grad|_K:       -nan  alpha:  4.626e-01  linmin:  5.834e-04  t[s]:   1020.29
shankar1729 commented 4 months ago

Are you missing smearing?

ColinBundschu commented 4 months ago

Yes I was missing it, good callout, but the issue persists with smearing:

*************** JDFTx 1.7.0 (git hash aa096dda) ***************

Start date and time: Sun May 19 14:56:55 2024
Executable /home/cbu/jdftx/build/jdftx_gpu with command-line: -i /home/cbu/dft_out/FeNC/clean/661.in -o /home/cbu/dft_out/FeNC/clean/661.out
Running on hosts (process indices):  x3002c0s31b0n0 (0-3)  x3002c0s31b1n0 (4-7)
Divided in process groups (process indices):  0 (0)  1 (1)  2 (2)  3 (3)  4 (4)  5 (5)  6 (6)  7 (7)
gpuInit: Found compatible cuda device 0 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 1 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 2 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 3 'NVIDIA A100-SXM4-40GB'
gpuInit: Selected device 0
Resource initialization completed at t[s]:      0.67
Run totals: 8 processes, 64 threads, 8 GPUs

Input parsed successfully to the following command list (including defaults):

add-U   Fe d 0.129
basis kpoint-dependent
coords-type Lattice
core-overlap-check vector
coulomb-interaction Periodic
davidson-band-ratio 1.1
dump End State Stress
dump Ionic IonicPositions Stress
dump-name $INPUT.$VAR
elec-cutoff 20 100
elec-eigen-algo Davidson
elec-ex-corr gga-PBE
elec-smearing Fermi 0.01
electronic-minimize  \
        dirUpdateScheme      FletcherReeves \
        linminMethod         DirUpdateRecommended \
        nIterations          150 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  1e-06 \
        nEnergyDiff          5 \
        alphaTstart          0.1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
exchange-regularization WignerSeitzTruncated
fluid None
fluid-ex-corr lda-TF lda-PZ
fluid-gummel-loop 10 1.000000e-05
fluid-minimize  \
        dirUpdateScheme      PolakRibiere \
        linminMethod         DirUpdateRecommended \
        nIterations          100 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  0 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
fluid-solvent H2O 55.338 ScalarEOS \
        epsBulk 78.4 \
        pMol 0.92466 \
        epsInf 1.77 \
        Pvap 1.06736e-10 \
        sigmaBulk 4.62e-05 \
        Rvdw 2.61727 \
        Res 1.42 \
        tauNuc 343133 \
        poleEl 15 7 1
forces-output-coords Positions
initial-state 661.$VAR
ion C   0.000000000000000   0.999700000000000   0.446000000000000 1
ion C   0.000000000000000   0.499800000000000   0.444300000000000 1
ion C   0.500000000000000   0.000300000000000   0.446200000000000 1
ion C   0.438100000000000   0.127400000000000   0.446100000000000 1
ion C   0.436200000000000   0.618500000000000   0.446400000000000 1
ion C   0.937400000000000   0.124500000000000   0.445400000000000 1
ion C   0.937700000000000   0.625100000000000   0.444700000000000 1
ion C   0.374700000000000   0.252500000000000   0.445900000000000 1
ion C   0.375400000000000   0.747600000000000   0.446100000000000 1
ion C   0.875300000000000   0.249400000000000   0.444800000000000 1
ion C   0.875000000000000   0.750100000000000   0.445300000000000 1
ion C   0.309600000000000   0.375000000000000   0.445400000000000 1
ion C   0.313200000000000   0.874300000000000   0.445900000000000 1
ion C   0.814000000000000   0.374500000000000   0.444600000000000 1
ion C   0.812100000000000   0.874900000000000   0.445600000000000 1
ion C   0.250700000000000   0.999900000000000   0.445700000000000 1
ion C   0.247500000000000   0.499500000000000   0.445100000000000 1
ion C   0.749300000000000   0.999900000000000   0.445600000000000 1
ion C   0.752400000000000   0.499500000000000   0.444900000000000 1
ion C   0.187600000000000   0.124600000000000   0.445300000000000 1
ion C   0.186700000000000   0.625000000000000   0.445100000000000 1
ion C   0.686900000000000   0.125700000000000   0.445700000000000 1
ion C   0.688800000000000   0.624300000000000   0.445600000000000 1
ion C   0.124600000000000   0.249400000000000   0.444800000000000 1
ion C   0.125000000000000   0.750100000000000   0.445500000000000 1
ion C   0.625200000000000   0.252500000000000   0.445900000000000 1
ion C   0.624500000000000   0.747600000000000   0.446000000000000 1
ion C   0.062000000000000   0.374500000000000   0.444400000000000 1
ion C   0.062600000000000   0.875000000000000   0.446000000000000 1
ion C   0.561900000000000   0.873300000000000   0.446100000000000 1
ion C   0.125200000000000   0.999700000000000   0.445800000000000 1
ion C   0.123900000000000   0.499800000000000   0.444500000000000 1
ion C   0.624200000000000   0.000200000000000   0.446000000000000 1
ion C   0.629100000000000   0.499800000000000   0.445900000000000 1
ion C   0.062500000000000   0.124500000000000   0.445400000000000 1
ion C   0.062200000000000   0.625100000000000   0.444900000000000 1
ion C   0.561800000000000   0.127400000000000   0.446100000000000 1
ion C   0.563800000000000   0.618500000000000   0.446300000000000 1
ion C   0.000000000000000   0.249300000000000   0.444800000000000 1
ion C   0.000000000000000   0.750100000000000   0.445500000000000 1
ion C   0.500000000000000   0.256700000000000   0.446300000000000 1
ion C   0.500000000000000   0.745000000000000   0.446300000000000 1
ion C   0.438000000000000   0.873300000000000   0.446200000000000 1
ion C   0.937900000000000   0.374500000000000   0.444300000000000 1
ion C   0.937300000000000   0.875000000000000   0.445900000000000 1
ion C   0.375800000000000   0.000200000000000   0.446000000000000 1
ion C   0.370800000000000   0.499800000000000   0.445900000000000 1
ion C   0.874700000000000   0.999700000000000   0.445700000000000 1
ion C   0.876000000000000   0.499800000000000   0.444400000000000 1
ion C   0.313000000000000   0.125700000000000   0.445700000000000 1
ion C   0.311200000000000   0.624300000000000   0.445700000000000 1
ion C   0.812300000000000   0.124600000000000   0.445300000000000 1
ion C   0.813300000000000   0.625000000000000   0.444900000000000 1
ion C   0.249300000000000   0.249700000000000   0.445200000000000 1
ion C   0.250000000000000   0.749700000000000   0.445700000000000 1
ion C   0.750600000000000   0.249700000000000   0.445200000000000 1
ion C   0.749900000000000   0.749700000000000   0.445500000000000 1
ion C   0.185900000000000   0.374500000000000   0.444700000000000 1
ion C   0.187900000000000   0.874900000000000   0.445700000000000 1
ion C   0.690300000000000   0.375000000000000   0.445400000000000 1
ion C   0.686800000000000   0.874300000000000   0.445800000000000 1
ion C   0.062600000000000   0.041200000000000   0.445800000000000 1
ion C   0.062000000000000   0.541700000000000   0.444500000000000 1
ion C   0.561900000000000   0.042900000000000   0.446100000000000 1
ion C   0.000000000000000   0.166100000000000   0.445200000000000 1
ion C   0.000000000000000   0.666900000000000   0.445000000000000 1
ion C   0.500000000000000   0.171200000000000   0.446200000000000 1
ion C   0.500000000000000   0.659500000000000   0.446400000000000 1
ion C   0.436200000000000   0.297700000000000   0.446200000000000 1
ion C   0.438100000000000   0.788800000000000   0.446300000000000 1
ion C   0.937700000000000   0.291100000000000   0.444600000000000 1
ion C   0.937400000000000   0.791700000000000   0.445600000000000 1
ion C   0.370800000000000   0.416400000000000   0.445900000000000 1
ion C   0.375800000000000   0.916000000000000   0.446000000000000 1
ion C   0.876000000000000   0.416400000000000   0.444400000000000 1
ion C   0.874700000000000   0.916500000000000   0.445800000000000 1
ion C   0.313200000000000   0.041900000000000   0.445800000000000 1
ion C   0.309600000000000   0.541200000000000   0.445500000000000 1
ion C   0.812100000000000   0.041300000000000   0.445500000000000 1
ion C   0.814000000000000   0.541700000000000   0.444700000000000 1
ion C   0.250000000000000   0.166500000000000   0.445400000000000 1
ion C   0.249300000000000   0.666500000000000   0.445500000000000 1
ion C   0.749900000000000   0.166500000000000   0.445400000000000 1
ion C   0.750600000000000   0.666500000000000   0.445300000000000 1
ion C   0.186700000000000   0.291200000000000   0.444800000000000 1
ion C   0.187600000000000   0.791600000000000   0.445700000000000 1
ion C   0.688800000000000   0.291900000000000   0.445500000000000 1
ion C   0.686900000000000   0.790500000000000   0.445800000000000 1
ion C   0.123900000000000   0.416400000000000   0.444500000000000 1
ion C   0.125200000000000   0.916500000000000   0.445800000000000 1
ion C   0.629100000000000   0.416400000000000   0.445800000000000 1
ion C   0.624200000000000   0.916000000000000   0.446000000000000 1
ion C   0.187900000000000   0.041300000000000   0.445500000000000 1
ion C   0.185900000000000   0.541700000000000   0.444900000000000 1
ion C   0.686800000000000   0.041900000000000   0.445700000000000 1
ion C   0.690300000000000   0.541200000000000   0.445400000000000 1
ion C   0.125000000000000   0.166100000000000   0.445200000000000 1
ion C   0.124600000000000   0.666800000000000   0.445200000000000 1
ion C   0.624500000000000   0.168600000000000   0.445900000000000 1
ion C   0.625200000000000   0.663700000000000   0.446000000000000 1
ion C   0.062200000000000   0.291100000000000   0.444600000000000 1
ion C   0.062500000000000   0.791700000000000   0.445700000000000 1
ion C   0.563800000000000   0.297700000000000   0.446200000000000 1
ion C   0.561800000000000   0.788800000000000   0.446200000000000 1
ion C   0.000000000000000   0.416400000000000   0.444300000000000 1
ion C   0.000000000000000   0.916500000000000   0.446100000000000 1
ion C   0.500000000000000   0.915900000000000   0.446200000000000 1
ion C   0.438000000000000   0.042900000000000   0.446100000000000 1
ion C   0.937300000000000   0.041200000000000   0.445800000000000 1
ion C   0.937900000000000   0.541700000000000   0.444400000000000 1
ion C   0.375400000000000   0.168600000000000   0.445900000000000 1
ion C   0.374700000000000   0.663700000000000   0.446100000000000 1
ion C   0.875000000000000   0.166100000000000   0.445200000000000 1
ion C   0.875300000000000   0.666800000000000   0.445000000000000 1
ion C   0.311200000000000   0.291900000000000   0.445500000000000 1
ion C   0.313000000000000   0.790500000000000   0.445900000000000 1
ion C   0.813300000000000   0.291200000000000   0.444800000000000 1
ion C   0.812300000000000   0.791600000000000   0.445500000000000 1
ion C   0.247500000000000   0.416700000000000   0.445000000000000 1
ion C   0.250700000000000   0.916300000000000   0.445800000000000 1
ion C   0.752400000000000   0.416700000000000   0.444900000000000 1
ion C   0.749300000000000   0.916300000000000   0.445700000000000 1
ion N   0.567200000000000   0.378800000000000   0.446300000000000 1
ion N   0.432700000000000   0.378800000000000   0.446300000000000 1
ion N   0.567200000000000   0.537400000000000   0.446300000000000 1
ion N   0.432700000000000   0.537400000000000   0.446400000000000 1
ion Fe   0.500000000000000   0.458100000000000   0.446600000000000 1
ion-species GBRV/$ID_pbe.uspp
ion-width 0
ionic-minimize  \
        dirUpdateScheme      L-BFGS \
        linminMethod         DirUpdateRecommended \
        nIterations          0 \
        history              15 \
        knormThreshold       0.0001 \
        maxThreshold         no \
        energyDiffThreshold  1e-06 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
kpoint   0.500000000000   0.500000000000   0.000000000000  1.00000000000000
kpoint-folding 4 4 1
latt-move-scale 1 1 1
latt-scale 1 1 1
lattice Orthorhombic 37.2276 32.3143 41.574
lattice-minimize  \
        dirUpdateScheme      L-BFGS \
        linminMethod         DirUpdateRecommended \
        nIterations          0 \
        history              15 \
        knormThreshold       0 \
        maxThreshold         no \
        energyDiffThreshold  1e-06 \
        nEnergyDiff          2 \
        alphaTstart          1 \
        alphaTmin            1e-10 \
        updateTestStepSize   yes \
        alphaTreduceFactor   0.1 \
        alphaTincreaseFactor 3 \
        nAlphaAdjustMax      3 \
        wolfeEnergy          0.0001 \
        wolfeGradient        0.9 \
        fdTest               no
lcao-params 150 1e-06 0.01
pcm-variant GLSSA13
perturb-minimize  \
        nIterations            0 \
        algorithm              MINRES \
        residualTol            0.0001 \
        residualDiffThreshold  0.0001 \
        CGBypass               no \
        recomputeResidual      no
spintype z-spin
subspace-rotation-factor 1 yes
symmetries automatic
symmetry-threshold 0.0001

---------- Setting up symmetries ----------

Found 8 point-group symmetries of the bravais lattice
Found 1 space-group symmetries with basis
Applied RMS atom displacement 0 bohrs to make symmetries exact.

---------- Initializing the Grid ----------
R =
[      37.2276            0            0  ]
[            0      32.3143            0  ]
[            0            0       41.574  ]
unit cell volume = 50012.9
G =
[   0.168778          0          0  ]
[          0    0.19444          0  ]
[          0          0   0.151133  ]
Minimum fftbox size, Smin = [  168  148  188  ]
Chosen fftbox size, S = [  168  150  192  ]

---------- Initializing tighter grid for wavefunction operations ----------
R =
[      37.2276            0            0  ]
[            0      32.3143            0  ]
[            0            0       41.574  ]
unit cell volume = 50012.9
G =
[   0.168778          0          0  ]
[          0    0.19444          0  ]
[          0          0   0.151133  ]
Minimum fftbox size, Smin = [  152  132  168  ]
Chosen fftbox size, S = [  160  140  168  ]

---------- Exchange Correlation functional ----------
Initalized PBE GGA exchange.
Initalized PBE GGA correlation.

---------- Setting up pseudopotentials ----------
Width of ionic core gaussian charges (only for fluid interactions / plotting) set to 0

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/c_pbe.uspp':
  Title: C.  Created by USPP 7.3.6 on 3-2-2014
  Reference state energy: -5.406344.  4 valence electrons in orbitals:
    |200>  occupation: 2  eigenvalue: -0.504890
    |210>  occupation: 2  eigenvalue: -0.194356
  lMax: 1  lLocal: 2  QijEcut: 5
  4 projectors sampled on a log grid with 503 points:
    l: 0  eig: -0.504890  rCut: 1.3
    l: 0  eig: 0.000000  rCut: 1.3
    l: 1  eig: -0.194357  rCut: 1.3
    l: 1  eig: 0.000000  rCut: 1.3
  Partial core density with radius 1.1
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.30 bohrs.

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/n_pbe.uspp':
  Title: N.  Created by USPP 7.3.6 on 3-2-2014
  Reference state energy: -9.763716.  5 valence electrons in orbitals:
    |200>  occupation: 2  eigenvalue: -0.681964
    |210>  occupation: 3  eigenvalue: -0.260726
  lMax: 1  lLocal: 2  QijEcut: 6
  4 projectors sampled on a log grid with 491 points:
    l: 0  eig: -0.681964  rCut: 1.15
    l: 0  eig: 0.000000  rCut: 1.15
    l: 1  eig: -0.260729  rCut: 1.2
    l: 1  eig: 0.500000  rCut: 1.2
  Partial core density with radius 0.8
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.20 bohrs.

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/fe_pbe.uspp':
  Title: Fe.  Created by USPP 7.3.6 on 15-6-15
  Reference state energy: -124.316937.  16 valence electrons in orbitals:
    |300>  occupation: 2  eigenvalue: -4.008817
    |310>  occupation: 6  eigenvalue: -2.753096
    |320>  occupation: 5  eigenvalue: -0.791264
    |400>  occupation: 2  eigenvalue: -0.516384
    |410>  occupation: 0  eigenvalue: -0.312222
  lMax: 2  lLocal: 3  QijEcut: 6
  6 projectors sampled on a log grid with 607 points:
    l: 0  eig: -4.008819  rCut: 1.55
    l: 0  eig: -0.516386  rCut: 1.55
    l: 1  eig: -2.753096  rCut: 1.55
    l: 1  eig: -0.312226  rCut: 1.55
    l: 2  eig: -0.791264  rCut: 1.65
    l: 2  eig: -0.450000  rCut: 1.65
  Partial core density with radius 0.65
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.65 bohrs.

Initialized 3 species with 127 total atoms.

Folded 1 k-points by 4x4x1 to 16 k-points.

---------- Setting up k-points, bands, fillings ----------
Adding inversion symmetry to k-mesh for non-inversion-symmetric unit cell.
Reduced to 8 k-points under symmetry.
Computing the number of bands and number of electrons
Calculating initial fillings.
nElectrons: 524.000000   nBands: 517   nStates: 16

----- Setting up reduced wavefunction bases (one per k-point) -----
average nbasis = 213660.750 , ideal nbasis = 213658.714

---------- Setting up ewald sum ----------
Optimum gaussian width for ewald sums = 6.555965 bohr.
Real space sum over 343 unit cells with max indices [  3  3  3  ]
Reciprocal space sum over 9177 terms with max indices [  10  9  11  ]

---------- Allocating electronic variables ----------
Initializing wave functions:  linear combination of atomic orbitals
C pseudo-atom occupations:   s ( 2 )  p ( 2 )
N pseudo-atom occupations:   s ( 2 )  p ( 3 )
Fe pseudo-atom occupations:   s ( 2 0 )  p ( 6 0 )  d ( 8 )
        FillingsUpdate:  mu: -0.199252942  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00415  Tot: +0.00000 ]
LCAOMinimize: Iter:   0  F: -836.1805908503083629  |grad|_K:  3.928e-03  alpha:  1.000e+00
        FillingsUpdate:  mu: -0.235192551  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00415  Tot: +0.00000 ]
LCAOMinimize: Iter:   1  F: -840.2669621067253729  |grad|_K:  2.119e-03  alpha:  9.079e-02  linmin: -1.409e-01  cgtest:  1.088e+00  t[s]:     22.45
LCAOMinimize: Encountered beta<0, resetting CG.
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 2.723795e-01.
        FillingsUpdate:  mu: -0.248271590  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00415  Tot: -0.00000 ]
LCAOMinimize: Iter:   2  F: -842.0717796668276378  |grad|_K:  2.258e-03  alpha: -1.458e+00  linmin: -1.675e-01  cgtest:  7.477e-01  t[s]:     28.03
LCAOMinimize:   Step increased F by 1.054948e+01, reducing alpha to 2.103098e-02.
        FillingsUpdate:  mu: -0.251851445  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00416  Tot: -0.00000 ]
LCAOMinimize: Iter:   3  F: -843.0099803439525203  |grad|_K:  3.541e-03  alpha:  2.103e-02  linmin: -2.148e-01  cgtest:  5.914e-01  t[s]:     37.57
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 6.309295e-02.
        FillingsUpdate:  mu: -0.170119754  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00417  Tot: -0.00000 ]
LCAOMinimize: Iter:   4  F: -843.2335550372346233  |grad|_K:  5.364e-03  alpha:  2.494e-02  linmin:  1.788e-01  cgtest: -6.022e-01  t[s]:     44.98
LCAOMinimize:   Bad step direction: g.d > 0.
LCAOMinimize:   Undoing step.
LCAOMinimize:   Step failed: resetting search direction.
        FillingsUpdate:  mu: -0.170119754  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00417  Tot: -0.00000 ]
LCAOMinimize: Iter:   5  F: -843.2335550372346233  |grad|_K:  5.364e-03  alpha:  0.000e+00
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 7.483144e-02.
        FillingsUpdate:  mu: -0.176306301  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00418  Tot: +0.00000 ]
LCAOMinimize: Iter:   6  F: -849.7156321835368544  |grad|_K:  3.602e-03  alpha: -1.130e-02  linmin: -2.535e-01  cgtest:  8.777e-01  t[s]:     54.51
LCAOMinimize: Encountered beta<0, resetting CG.
LCAOMinimize:   Step increased F by 5.907787e+00, reducing alpha to 8.263169e-03.
        FillingsUpdate:  mu: -0.178781016  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00418  Tot: +0.00000 ]
LCAOMinimize: Iter:   7  F: -850.1519945926452237  |grad|_K:  3.465e-03  alpha:  8.263e-03  linmin: -2.466e-01  cgtest:  9.669e-01  t[s]:     64.08
LCAOMinimize: Encountered beta<0, resetting CG.
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 2.478951e-02.
        FillingsUpdate:  mu: -0.185762370  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00418  Tot: -0.00000 ]
LCAOMinimize: Iter:   8  F: -850.6680078289901985  |grad|_K:  4.036e-03  alpha: -1.954e-02  linmin: -2.963e-01  cgtest:  7.771e-01  t[s]:     69.69
LCAOMinimize:   Step increased F by 1.056669e+01, reducing alpha to 1.115090e-02.
        FillingsUpdate:  mu: -0.194651035  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00419  Tot: +0.00000 ]
LCAOMinimize: Iter:   9  F: -852.2327275838514424  |grad|_K:  3.565e-03  alpha:  1.115e-02  linmin: -3.292e-01  cgtest:  6.576e-01  t[s]:     79.33
        FillingsUpdate:  mu: -0.231323065  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00422  Tot: +0.00000 ]
LCAOMinimize: Iter:  10  F: -852.9315340385956006  |grad|_K:  1.308e-03  alpha:  2.032e-02  linmin:  1.795e-01  cgtest: -6.793e-01  t[s]:     85.06
LCAOMinimize:   Bad step direction: g.d > 0.
LCAOMinimize:   Undoing step.
LCAOMinimize:   Step failed: resetting search direction.
        FillingsUpdate:  mu: -0.231323065  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00422  Tot: +0.00000 ]
LCAOMinimize: Iter:  11  F: -852.9315340385956006  |grad|_K:  1.308e-03  alpha:  0.000e+00
        FillingsUpdate:  mu: -0.199597337  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00421  Tot: +0.00000 ]
LCAOMinimize: Iter:  12  F: -853.0281232410086432  |grad|_K:  5.310e-04  alpha:  2.598e-02  linmin: -1.980e-02  cgtest:  7.957e-02  t[s]:     94.71
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 7.794284e-02.
        FillingsUpdate:  mu: -0.237084121  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00426  Tot: +0.00000 ]
LCAOMinimize: Iter:  13  F: -853.1475565341165748  |grad|_K:  1.943e-04  alpha:  2.060e-01  linmin:  7.109e-02  cgtest: -4.571e-01  t[s]:    102.17
        FillingsUpdate:  mu: -0.213976300  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00428  Tot: -0.00000 ]
LCAOMinimize: Iter:  14  F: -853.1507992923070560  |grad|_K:  2.218e-04  alpha:  6.711e-02  linmin:  8.639e-03  cgtest:  3.344e-01  t[s]:    107.90
        FillingsUpdate:  mu: -0.243172808  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00433  Tot: +0.00000 ]
LCAOMinimize: Iter:  15  F: -853.1611704667942604  |grad|_K:  2.208e-04  alpha:  1.043e-01  linmin:  7.209e-03  cgtest: -2.968e-01  t[s]:    113.65
        FillingsUpdate:  mu: -0.225007548  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00439  Tot: +0.00000 ]
LCAOMinimize: Iter:  16  F: -853.1665834570842435  |grad|_K:  9.723e-05  alpha:  5.987e-02  linmin:  4.331e-03  cgtest:  4.163e-01  t[s]:    119.39
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.796171e-01.
        FillingsUpdate:  mu: -0.219109087  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00447  Tot: +0.00000 ]
LCAOMinimize: Iter:  17  F: -853.1732120096525023  |grad|_K:  8.336e-05  alpha:  3.204e-01  linmin: -9.620e-03  cgtest:  7.979e-02  t[s]:    126.84
        FillingsUpdate:  mu: -0.229235517  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00450  Tot: -0.00000 ]
LCAOMinimize: Iter:  18  F: -853.1739927450065579  |grad|_K:  8.031e-05  alpha:  5.597e-02  linmin:  1.908e-02  cgtest: -9.721e-02  t[s]:    132.56
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.679035e-01.
        FillingsUpdate:  mu: -0.233635872  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00471  Tot: +0.00000 ]
LCAOMinimize: Iter:  19  F: -853.1772144680605834  |grad|_K:  1.273e-04  alpha:  2.593e-01  linmin: -8.108e-04  cgtest:  9.725e-01  t[s]:    140.08
        FillingsUpdate:  mu: -0.217339424  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00479  Tot: +0.00000 ]
LCAOMinimize: Iter:  20  F: -853.1785488626690039  |grad|_K:  8.669e-05  alpha:  6.831e-02  linmin:  9.082e-02  cgtest: -7.240e-01  t[s]:    145.77
        FillingsUpdate:  mu: -0.215245262  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00497  Tot: +0.00000 ]
LCAOMinimize: Iter:  21  F: -853.1791494838466861  |grad|_K:  1.021e-04  alpha:  1.139e-01  linmin: -5.283e-03  cgtest:  1.008e+00  t[s]:    151.50
        FillingsUpdate:  mu: -0.224952821  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00501  Tot: +0.00000 ]
LCAOMinimize: Iter:  22  F: -853.1801205885554964  |grad|_K:  1.977e-05  alpha:  4.421e-02  linmin:  2.412e-02  cgtest: -1.349e-01  t[s]:    157.13
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.326288e-01.
        FillingsUpdate:  mu: -0.224757872  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00528  Tot: -0.00000 ]
LCAOMinimize: Iter:  23  F: -853.1803857919030634  |grad|_K:  1.085e-05  alpha:  3.225e-01  linmin: -4.691e-03  cgtest:  7.594e-02  t[s]:    164.63
        FillingsUpdate:  mu: -0.223659910  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00577  Tot: +0.00000 ]
LCAOMinimize: Iter:  24  F: -853.1804842116893042  |grad|_K:  1.160e-05  alpha:  3.841e-01  linmin: -2.685e-03  cgtest: -1.381e-02  t[s]:    170.36
        FillingsUpdate:  mu: -0.224825224  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00599  Tot: -0.00000 ]
LCAOMinimize: Iter:  25  F: -853.1805044938911351  |grad|_K:  7.316e-06  alpha:  7.182e-02  linmin:  9.125e-03  cgtest: -9.095e-02  t[s]:    176.07
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.154587e-01.
        FillingsUpdate:  mu: -0.224798520  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00680  Tot: -0.00000 ]
LCAOMinimize: Iter:  26  F: -853.1805321719591575  |grad|_K:  3.158e-06  alpha:  2.511e-01  linmin: -2.450e-03  cgtest:  3.304e-01  t[s]:    183.48
        FillingsUpdate:  mu: -0.224420471  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00711  Tot: +0.00000 ]
LCAOMinimize: Iter:  27  F: -853.1805344163470863  |grad|_K:  3.028e-06  alpha:  1.048e-01  linmin:  2.582e-04  cgtest: -4.732e-03  t[s]:    189.23
        FillingsUpdate:  mu: -0.224575055  nElectrons: 524.000000  magneticMoment: [ Abs: 0.00790  Tot: +0.00000 ]
LCAOMinimize: Iter:  28  F: -853.1805370788463279  |grad|_K:  1.424e-06  alpha:  1.354e-01  linmin: -5.694e-04  cgtest:  1.849e-02  t[s]:    194.95
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 4.062054e-01.
        FillingsUpdate:  mu: -0.224687346  nElectrons: 524.000000  magneticMoment: [ Abs: 0.01129  Tot: -0.00000 ]
LCAOMinimize: Iter:  29  F: -853.1805398611076043  |grad|_K:  1.647e-06  alpha:  6.378e-01  linmin: -4.473e-04  cgtest: -8.379e-03  t[s]:    202.46
        FillingsUpdate:  mu: -0.224434058  nElectrons: 524.000000  magneticMoment: [ Abs: 0.01373  Tot: -0.00000 ]
LCAOMinimize: Iter:  30  F: -853.1805407952858786  |grad|_K:  2.576e-06  alpha:  1.604e-01  linmin:  1.981e-04  cgtest: -1.041e-03  t[s]:    208.17
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 4.812434e-01.
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.443730e+00.
        FillingsUpdate:  mu: -0.223588635  nElectrons: 524.000000  magneticMoment: [ Abs: 0.15430  Tot: -0.00000 ]
LCAOMinimize: Iter:  31  F: -853.1805730409050739  |grad|_K:  3.084e-05  alpha:  2.906e+00  linmin:  5.547e-03  cgtest:  2.172e-01  t[s]:    217.39
        FillingsUpdate:  mu: -0.223563388  nElectrons: 524.000000  magneticMoment: [ Abs: 0.15726  Tot: -0.00000 ]
LCAOMinimize: Iter:  32  F: -853.1805739195883689  |grad|_K:  3.144e-05  alpha:  4.408e-04  linmin: -5.386e-03  cgtest:  1.000e+00  t[s]:    223.11
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.322387e-03.
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 3.967162e-03.
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.190148e-02.
        FillingsUpdate:  mu: -0.229675725  nElectrons: 524.000000  magneticMoment: [ Abs: 0.34632  Tot: +0.00000 ]
LCAOMinimize: Iter:  33  F: -853.1819041880086161  |grad|_K:  8.382e-05  alpha:  6.577e-01  linmin:  7.182e-03  cgtest: -2.258e-01  t[s]:    232.39
        FillingsUpdate:  mu: -0.227829624  nElectrons: 524.000000  magneticMoment: [ Abs: 0.57257  Tot: -0.00000 ]
LCAOMinimize: Iter:  34  F: -853.1840369573932321  |grad|_K:  1.158e-04  alpha:  9.878e-02  linmin: -3.830e-02  cgtest:  7.839e-01  t[s]:    238.04
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.963446e-01.
        FillingsUpdate:  mu: -0.201955008  nElectrons: 524.000000  magneticMoment: [ Abs: 1.27826  Tot: -0.00000 ]
LCAOMinimize: Iter:  35  F: -853.2027495234381149  |grad|_K:  2.020e-04  alpha:  6.287e-01  linmin:  1.791e-02  cgtest:  3.918e-02  t[s]:    245.52
        FillingsUpdate:  mu: -0.221226128  nElectrons: 524.000000  magneticMoment: [ Abs: 1.46050  Tot: +0.00000 ]
LCAOMinimize: Iter:  36  F: -853.2101644446980799  |grad|_K:  7.961e-05  alpha:  8.741e-02  linmin: -4.744e-02  cgtest:  5.720e-01  t[s]:    251.23
LCAOMinimize: Encountered beta<0, resetting CG.
        FillingsUpdate:  mu: -0.227096715  nElectrons: 524.000000  magneticMoment: [ Abs: 1.51719  Tot: +0.00000 ]
LCAOMinimize: Iter:  37  F: -853.2129957479132827  |grad|_K:  6.238e-05  alpha:  2.124e-01  linmin:  8.815e-03  cgtest: -3.773e-02  t[s]:    256.95
        FillingsUpdate:  mu: -0.224310506  nElectrons: 524.000000  magneticMoment: [ Abs: 1.52775  Tot: +0.00000 ]
LCAOMinimize: Iter:  38  F: -853.2140836837246525  |grad|_K:  3.671e-05  alpha:  1.406e-01  linmin: -3.871e-04  cgtest:  1.471e-01  t[s]:    262.69
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 4.219384e-01.
        FillingsUpdate:  mu: -0.223908148  nElectrons: 524.000000  magneticMoment: [ Abs: 1.62127  Tot: -0.00000 ]
LCAOMinimize: Iter:  39  F: -853.2153378323487232  |grad|_K:  3.264e-05  alpha:  4.345e-01  linmin:  1.325e-02  cgtest: -1.796e-01  t[s]:    270.16
        FillingsUpdate:  mu: -0.223404637  nElectrons: 524.000000  magneticMoment: [ Abs: 1.72953  Tot: -0.00000 ]
LCAOMinimize: Iter:  40  F: -853.2163747486874854  |grad|_K:  2.722e-05  alpha:  4.822e-01  linmin: -1.146e-02  cgtest:  5.119e-01  t[s]:    275.87
        FillingsUpdate:  mu: -0.224327403  nElectrons: 524.000000  magneticMoment: [ Abs: 1.78263  Tot: -0.00000 ]
LCAOMinimize: Iter:  41  F: -853.2170379531492017  |grad|_K:  2.341e-05  alpha:  4.099e-01  linmin: -4.278e-04  cgtest:  2.138e-03  t[s]:    281.58
        FillingsUpdate:  mu: -0.222865287  nElectrons: 524.000000  magneticMoment: [ Abs: 1.86348  Tot: -0.00000 ]
LCAOMinimize: Iter:  42  F: -853.2177352551783542  |grad|_K:  2.183e-05  alpha:  5.780e-01  linmin: -1.358e-02  cgtest:  5.803e-02  t[s]:    287.28
        FillingsUpdate:  mu: -0.224480824  nElectrons: 524.000000  magneticMoment: [ Abs: 1.91752  Tot: -0.00000 ]
LCAOMinimize: Iter:  43  F: -853.2180452778361541  |grad|_K:  2.153e-05  alpha:  2.899e-01  linmin:  1.510e-03  cgtest: -9.190e-02  t[s]:    293.04
        FillingsUpdate:  mu: -0.223484156  nElectrons: 524.000000  magneticMoment: [ Abs: 1.92900  Tot: -0.00000 ]
LCAOMinimize: Iter:  44  F: -853.2182355214135896  |grad|_K:  1.417e-05  alpha:  1.965e-01  linmin: -6.006e-04  cgtest:  1.008e-01  t[s]:    298.78
        FillingsUpdate:  mu: -0.222983975  nElectrons: 524.000000  magneticMoment: [ Abs: 1.95216  Tot: +0.00000 ]
LCAOMinimize: Iter:  45  F: -853.2184515615687133  |grad|_K:  1.367e-05  alpha:  4.998e-01  linmin: -1.437e-04  cgtest: -4.019e-02  t[s]:    304.50
        FillingsUpdate:  mu: -0.224063204  nElectrons: 524.000000  magneticMoment: [ Abs: 1.97970  Tot: +0.00000 ]
LCAOMinimize: Iter:  46  F: -853.2185578434454101  |grad|_K:  1.587e-05  alpha:  2.711e-01  linmin:  1.901e-03  cgtest: -3.556e-02  t[s]:    310.25
        FillingsUpdate:  mu: -0.223320898  nElectrons: 524.000000  magneticMoment: [ Abs: 1.99242  Tot: +0.00000 ]
LCAOMinimize: Iter:  47  F: -853.2187303587761562  |grad|_K:  8.004e-06  alpha:  3.235e-01  linmin:  4.888e-03  cgtest: -4.774e-02  t[s]:    315.93
        FillingsUpdate:  mu: -0.222666804  nElectrons: 524.000000  magneticMoment: [ Abs: 1.98915  Tot: +0.00000 ]
LCAOMinimize: Iter:  48  F: -853.2187953011800801  |grad|_K:  1.169e-05  alpha:  4.807e-01  linmin:  1.969e-03  cgtest:  2.511e-02  t[s]:    321.67
        FillingsUpdate:  mu: -0.223404104  nElectrons: 524.000000  magneticMoment: [ Abs: 2.00113  Tot: -0.00000 ]
LCAOMinimize: Iter:  49  F: -853.2188593172882065  |grad|_K:  9.505e-06  alpha:  2.216e-01  linmin:  5.437e-04  cgtest:  2.511e-03  t[s]:    327.38
        FillingsUpdate:  mu: -0.223577110  nElectrons: 524.000000  magneticMoment: [ Abs: 2.00433  Tot: -0.00000 ]
LCAOMinimize: Iter:  50  F: -853.2189873215894522  |grad|_K:  8.314e-06  alpha:  6.636e-01  linmin:  1.304e-03  cgtest:  3.004e-02  t[s]:    333.05
        FillingsUpdate:  mu: -0.222922529  nElectrons: 524.000000  magneticMoment: [ Abs: 1.99820  Tot: -0.00000 ]
LCAOMinimize: Iter:  51  F: -853.2190139920132879  |grad|_K:  7.216e-06  alpha:  1.841e-01  linmin:  3.380e-03  cgtest: -1.788e-02  t[s]:    338.77
        FillingsUpdate:  mu: -0.223074523  nElectrons: 524.000000  magneticMoment: [ Abs: 2.00140  Tot: -0.00000 ]
LCAOMinimize: Iter:  52  F: -853.2190547067200441  |grad|_K:  5.771e-06  alpha:  3.716e-01  linmin:  2.853e-03  cgtest:  3.714e-03  t[s]:    344.48
        FillingsUpdate:  mu: -0.223442342  nElectrons: 524.000000  magneticMoment: [ Abs: 1.99976  Tot: -0.00000 ]
LCAOMinimize: Iter:  53  F: -853.2190818984204270  |grad|_K:  5.254e-06  alpha:  3.840e-01  linmin:  1.748e-04  cgtest:  2.166e-02  t[s]:    350.28
        FillingsUpdate:  mu: -0.223102967  nElectrons: 524.000000  magneticMoment: [ Abs: 1.99366  Tot: +0.00000 ]
LCAOMinimize: Iter:  54  F: -853.2190926255292425  |grad|_K:  3.784e-06  alpha:  1.816e-01  linmin:  3.605e-04  cgtest: -1.232e-02  t[s]:    355.97
        FillingsUpdate:  mu: -0.223130903  nElectrons: 524.000000  magneticMoment: [ Abs: 1.99311  Tot: +0.00000 ]
LCAOMinimize: Iter:  55  F: -853.2191007166378540  |grad|_K:  1.929e-06  alpha:  2.634e-01  linmin: -8.211e-04  cgtest:  2.342e-03  t[s]:    361.67
        FillingsUpdate:  mu: -0.223308391  nElectrons: 524.000000  magneticMoment: [ Abs: 1.99118  Tot: -0.00000 ]
LCAOMinimize: Iter:  56  F: -853.2191045279538457  |grad|_K:  2.732e-06  alpha:  4.765e-01  linmin:  9.996e-05  cgtest: -4.483e-03  t[s]:    367.38
        FillingsUpdate:  mu: -0.223153507  nElectrons: 524.000000  magneticMoment: [ Abs: 1.98458  Tot: +0.00000 ]
LCAOMinimize: Iter:  57  F: -853.2191081882534718  |grad|_K:  2.371e-06  alpha:  2.284e-01  linmin: -1.338e-04  cgtest:  4.467e-03  t[s]:    373.10
        FillingsUpdate:  mu: -0.223127254  nElectrons: 524.000000  magneticMoment: [ Abs: 1.97438  Tot: +0.00000 ]
LCAOMinimize: Iter:  58  F: -853.2191161578977017  |grad|_K:  2.064e-06  alpha:  6.602e-01  linmin:  1.677e-04  cgtest: -3.069e-02  t[s]:    378.73
        FillingsUpdate:  mu: -0.223307607  nElectrons: 524.000000  magneticMoment: [ Abs: 1.97330  Tot: -0.00000 ]
LCAOMinimize: Iter:  59  F: -853.2191184927492031  |grad|_K:  2.388e-06  alpha:  2.524e-01  linmin: -1.258e-03  cgtest:  1.990e-02  t[s]:    384.46
        FillingsUpdate:  mu: -0.223232383  nElectrons: 524.000000  magneticMoment: [ Abs: 1.96647  Tot: +0.00000 ]
LCAOMinimize: Iter:  60  F: -853.2191231179250508  |grad|_K:  2.069e-06  alpha:  3.731e-01  linmin: -2.614e-03  cgtest:  5.624e-03  t[s]:    390.20
        FillingsUpdate:  mu: -0.223080722  nElectrons: 524.000000  magneticMoment: [ Abs: 1.96024  Tot: +0.00000 ]
LCAOMinimize: Iter:  61  F: -853.2191298395363219  |grad|_K:  2.862e-06  alpha:  7.166e-01  linmin: -2.546e-03  cgtest: -4.431e-02  t[s]:    395.85
        FillingsUpdate:  mu: -0.223285807  nElectrons: 524.000000  magneticMoment: [ Abs: 1.96052  Tot: +0.00000 ]
LCAOMinimize: Iter:  62  F: -853.2191337500629515  |grad|_K:  2.683e-06  alpha:  2.049e-01  linmin: -9.630e-03  cgtest:  6.709e-02  t[s]:    401.60
        FillingsUpdate:  mu: -0.223278291  nElectrons: 524.000000  magneticMoment: [ Abs: 1.95691  Tot: +0.00000 ]
LCAOMinimize: Iter:  63  F: -853.2191412101236665  |grad|_K:  2.661e-06  alpha:  4.548e-01  linmin: -4.650e-03  cgtest: -1.899e-02  t[s]:    407.29
        FillingsUpdate:  mu: -0.223044416  nElectrons: 524.000000  magneticMoment: [ Abs: 1.95522  Tot: +0.00000 ]
LCAOMinimize: Iter:  64  F: -853.2191510073769223  |grad|_K:  3.962e-06  alpha:  6.223e-01  linmin: -3.560e-03  cgtest: -4.349e-02  t[s]:    413.05
        FillingsUpdate:  mu: -0.223263118  nElectrons: 524.000000  magneticMoment: [ Abs: 1.95956  Tot: +0.00000 ]
LCAOMinimize: Iter:  65  F: -853.2191580156489863  |grad|_K:  2.855e-06  alpha:  1.831e-01  linmin: -1.396e-02  cgtest:  1.203e-01  t[s]:    418.75
        FillingsUpdate:  mu: -0.223248037  nElectrons: 524.000000  magneticMoment: [ Abs: 1.95991  Tot: -0.00000 ]
LCAOMinimize: Iter:  66  F: -853.2191633027995294  |grad|_K:  1.461e-06  alpha:  2.831e-01  linmin: -1.071e-03  cgtest: -1.304e-01  t[s]:    424.47
        FillingsUpdate:  mu: -0.223132523  nElectrons: 524.000000  magneticMoment: [ Abs: 1.95932  Tot: -0.00000 ]
LCAOMinimize: Iter:  67  F: -853.2191651113481612  |grad|_K:  1.982e-06  alpha:  3.934e-01  linmin:  1.950e-04  cgtest:  1.630e-02  t[s]:    430.23
        FillingsUpdate:  mu: -0.223237584  nElectrons: 524.000000  magneticMoment: [ Abs: 1.96186  Tot: +0.00000 ]
LCAOMinimize: Iter:  68  F: -853.2191669627698047  |grad|_K:  1.339e-06  alpha:  2.204e-01  linmin:  2.170e-04  cgtest:  1.325e-02  t[s]:    435.92
        FillingsUpdate:  mu: -0.223253029  nElectrons: 524.000000  magneticMoment: [ Abs: 1.96363  Tot: -0.00000 ]
LCAOMinimize: Iter:  69  F: -853.2191688719732383  |grad|_K:  1.155e-06  alpha:  4.965e-01  linmin: -2.447e-04  cgtest: -9.060e-03  t[s]:    441.68
        FillingsUpdate:  mu: -0.223178107  nElectrons: 524.000000  magneticMoment: [ Abs: 1.96298  Tot: -0.00000 ]
LCAOMinimize: Iter:  70  F: -853.2191694048785848  |grad|_K:  9.472e-07  alpha:  1.850e-01  linmin: -6.325e-04  cgtest:  8.233e-03  t[s]:    447.44
        FillingsUpdate:  mu: -0.223223156  nElectrons: 524.000000  magneticMoment: [ Abs: 1.96367  Tot: -0.00000 ]
LCAOMinimize: Iter:  71  F: -853.2191700113370416  |grad|_K:  7.933e-07  alpha:  3.140e-01  linmin:  2.490e-05  cgtest: -4.664e-03  t[s]:    453.14
LCAOMinimize: Converged (|Delta F|<1.000000e-06 for 2 iters).

---- Citations for features of the code used in this run ----

   Software package:
      R. Sundararaman, K. Letchworth-Weaver, K.A. Schwarz, D. Gunceler, Y. Ozhabes and T.A. Arias, 'JDFTx: software for joint density-functional theory', SoftwareX 6, 278 (2017)

   Simplified rotationally-invariant DFT+U:
      S. L. Dudarev et al., Phys. Rev. B 57, 1505 (1998)

   gga-PBE exchange-correlation functional:
      J.P. Perdew, K. Burke and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996)

   Pseudopotentials:
      KF Garrity, JW Bennett, KM Rabe and D Vanderbilt, Comput. Mater. Sci. 81, 446 (2014)

   Total energy minimization with Auxiliary Hamiltonian:
      C. Freysoldt, S. Boeck, and J. Neugebauer, Phys. Rev. B 79, 241103(R) (2009)

This list may not be complete. Please suggest additional citations or
report any other bugs at https://github.com/shankar1729/jdftx/issues

Initialization completed successfully at t[s]:    453.81

-------- Electronic minimization -----------
        FillingsUpdate:  mu: -0.223223156  nElectrons: 524.000000  magneticMoment: [ Abs: 1.96367  Tot: -0.00000 ]
ElecMinimize: Iter:   0  F: -853.219170011342044  |grad|_K:       -nan  alpha:  1.000e-01
ElecMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 3.000000e-01.
        FillingsUpdate:  mu: -0.133094817  nElectrons: 524.000000  magneticMoment: [ Abs: 2.02980  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 1
ElecMinimize: Iter:   1  F: -860.232275882298836  |grad|_K:       -nan  alpha:  6.706e-01  linmin: -1.888e-03  t[s]:    470.91
        FillingsUpdate:  mu: -0.092468003  nElectrons: 524.000000  magneticMoment: [ Abs: 1.59987  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.91
ElecMinimize: Iter:   2  F: -860.464253998061622  |grad|_K:       -nan  alpha:  8.242e-02  linmin: -7.090e-04  t[s]:    479.53
ElecMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.472616e-01.
        FillingsUpdate:  mu: -0.062623866  nElectrons: 524.000000  magneticMoment: [ Abs: 1.36239  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.649
ElecMinimize: Iter:   3  F: -860.650373645047125  |grad|_K:       -nan  alpha:  2.554e-01  linmin:  2.652e-06  t[s]:    491.38
        FillingsUpdate:  mu: -0.065453955  nElectrons: 524.000000  magneticMoment: [ Abs: 1.35918  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.931
ElecMinimize: Iter:   4  F: -860.797083334653166  |grad|_K:       -nan  alpha:  1.028e-01  linmin:  1.519e-05  t[s]:    500.33
        FillingsUpdate:  mu: -0.097400394  nElectrons: 524.000000  magneticMoment: [ Abs: 1.42773  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 1.04
ElecMinimize: Iter:   5  F: -860.863083055195375  |grad|_K:       -nan  alpha:  9.391e-02  linmin: -1.398e-05  t[s]:    509.31
        FillingsUpdate:  mu: -0.109740661  nElectrons: 524.000000  magneticMoment: [ Abs: 1.42798  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.878
ElecMinimize: Iter:   6  F: -860.899909553289262  |grad|_K:       -nan  alpha:  1.214e-01  linmin:  2.490e-06  t[s]:    518.23
        FillingsUpdate:  mu: -0.093853505  nElectrons: 524.000000  magneticMoment: [ Abs: 1.33348  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 1.12
ElecMinimize: Iter:   7  F: -860.927908136729684  |grad|_K:       -nan  alpha:  1.100e-01  linmin: -2.193e-05  t[s]:    526.89
        FillingsUpdate:  mu: -0.081949943  nElectrons: 524.000000  magneticMoment: [ Abs: 1.25613  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.879
ElecMinimize: Iter:   8  F: -860.950882013324645  |grad|_K:       -nan  alpha:  1.385e-01  linmin: -9.827e-05  t[s]:    536.13
        FillingsUpdate:  mu: -0.079665759  nElectrons: 524.000000  magneticMoment: [ Abs: 1.20982  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.66
ElecMinimize: Iter:   9  F: -860.970199981508586  |grad|_K:       -nan  alpha:  1.521e-01  linmin:  1.600e-05  t[s]:    545.43
        FillingsUpdate:  mu: -0.089299662  nElectrons: 524.000000  magneticMoment: [ Abs: 1.20935  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.851
ElecMinimize: Iter:  10  F: -860.977798120410853  |grad|_K:       -nan  alpha:  1.327e-01  linmin: -2.406e-05  t[s]:    554.74
        FillingsUpdate:  mu: -0.093358983  nElectrons: 524.000000  magneticMoment: [ Abs: 1.20155  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.635
ElecMinimize: Iter:  11  F: -860.984556389825457  |grad|_K:       -nan  alpha:  2.525e-01  linmin: -2.902e-05  t[s]:    563.42
        FillingsUpdate:  mu: -0.089848844  nElectrons: 524.000000  magneticMoment: [ Abs: 1.18514  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.674
ElecMinimize: Iter:  12  F: -860.989542259777750  |grad|_K:       -nan  alpha:  1.748e-01  linmin:  1.100e-06  t[s]:    572.38
        FillingsUpdate:  mu: -0.087624534  nElectrons: 524.000000  magneticMoment: [ Abs: 1.17505  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.716
ElecMinimize: Iter:  13  F: -860.991264083347460  |grad|_K:       -nan  alpha:  2.014e-01  linmin: -8.141e-07  t[s]:    581.51
        FillingsUpdate:  mu: -0.087759142  nElectrons: 524.000000  magneticMoment: [ Abs: 1.16698  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.682
ElecMinimize: Iter:  14  F: -860.992450645403551  |grad|_K:       -nan  alpha:  2.433e-01  linmin:  9.507e-07  t[s]:    590.58
        FillingsUpdate:  mu: -0.088828418  nElectrons: 524.000000  magneticMoment: [ Abs: 1.16431  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.841
ElecMinimize: Iter:  15  F: -860.992960537694785  |grad|_K:       -nan  alpha:  1.870e-01  linmin:  3.008e-07  t[s]:    599.59
        FillingsUpdate:  mu: -0.089544738  nElectrons: 524.000000  magneticMoment: [ Abs: 1.16147  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.844
ElecMinimize: Iter:  16  F: -860.993345080954441  |grad|_K:       -nan  alpha:  2.760e-01  linmin:  3.488e-07  t[s]:    608.73
        FillingsUpdate:  mu: -0.089122250  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15854  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.739
ElecMinimize: Iter:  17  F: -860.993609166086003  |grad|_K:       -nan  alpha:  2.026e-01  linmin:  5.368e-07  t[s]:    617.78
        FillingsUpdate:  mu: -0.088591835  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15730  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.875
ElecMinimize: Iter:  18  F: -860.993725002769679  |grad|_K:       -nan  alpha:  1.899e-01  linmin:  4.303e-08  t[s]:    626.55
        FillingsUpdate:  mu: -0.088497511  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15695  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.801
ElecMinimize: Iter:  19  F: -860.993789762496476  |grad|_K:       -nan  alpha:  2.268e-01  linmin:  1.930e-07  t[s]:    635.89
        FillingsUpdate:  mu: -0.088728992  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15648  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.701
ElecMinimize: Iter:  20  F: -860.993845175660795  |grad|_K:       -nan  alpha:  2.486e-01  linmin:  4.770e-07  t[s]:    645.20
        FillingsUpdate:  mu: -0.088852362  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15608  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.675
ElecMinimize: Iter:  21  F: -860.993878160093686  |grad|_K:       -nan  alpha:  2.055e-01  linmin:  3.292e-07  t[s]:    654.06
        FillingsUpdate:  mu: -0.088863090  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15607  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.857
ElecMinimize: Iter:  22  F: -860.993900912147751  |grad|_K:       -nan  alpha:  2.643e-01  linmin:  2.441e-07  t[s]:    662.70
        FillingsUpdate:  mu: -0.088853206  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15644  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.881
ElecMinimize: Iter:  23  F: -860.993914346602651  |grad|_K:       -nan  alpha:  2.255e-01  linmin:  7.726e-08  t[s]:    671.73
        FillingsUpdate:  mu: -0.088833765  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15688  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.998
ElecMinimize: Iter:  24  F: -860.993926298297879  |grad|_K:       -nan  alpha:  2.817e-01  linmin:  7.548e-08  t[s]:    680.96
        FillingsUpdate:  mu: -0.088831983  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15709  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.907
ElecMinimize: Iter:  25  F: -860.993935368467646  |grad|_K:       -nan  alpha:  2.060e-01  linmin:  6.810e-08  t[s]:    689.58
        FillingsUpdate:  mu: -0.088874825  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15727  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 1.06
ElecMinimize: Iter:  26  F: -860.993941123639388  |grad|_K:       -nan  alpha:  2.157e-01  linmin:  1.121e-07  t[s]:    698.88
        FillingsUpdate:  mu: -0.088911414  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15761  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.843
ElecMinimize: Iter:  27  F: -860.993946701957043  |grad|_K:       -nan  alpha:  2.485e-01  linmin:  8.540e-07  t[s]:    708.14
        FillingsUpdate:  mu: -0.088898419  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15769  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.708
ElecMinimize: Iter:  28  F: -860.993949101053204  |grad|_K:       -nan  alpha:  6.529e-02  linmin: -7.775e-06  t[s]:    717.25
        FillingsUpdate:  mu: -0.088837814  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15773  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.527
ElecMinimize: Iter:  29  F: -860.993952981078905  |grad|_K:       -nan  alpha:  1.633e-01  linmin:  7.974e-06  t[s]:    726.44
        FillingsUpdate:  mu: -0.088828830  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15771  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.4
ElecMinimize: Iter:  30  F: -860.993953362402067  |grad|_K:       -nan  alpha:  4.344e-02  linmin: -1.720e-06  t[s]:    735.72
        FillingsUpdate:  mu: -0.088816918  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15764  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.272
ElecMinimize: Iter:  31  F: -860.993954761981058  |grad|_K:       -nan  alpha:  1.192e-01  linmin:  2.966e-06  t[s]:    744.98
        FillingsUpdate:  mu: -0.088818640  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15760  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.194
ElecMinimize: Iter:  32  F: -860.993955450799035  |grad|_K:       -nan  alpha:  8.220e-02  linmin:  1.398e-08  t[s]:    754.03
        FillingsUpdate:  mu: -0.088828756  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15756  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.151
ElecMinimize: Iter:  33  F: -860.993956270161107  |grad|_K:       -nan  alpha:  1.119e-01  linmin:  1.149e-06  t[s]:    763.14
        FillingsUpdate:  mu: -0.088840970  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15754  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.123
        SubspaceRotationAdjust: resetting CG because factor has changed by 0.123454
ElecMinimize:   State modified externally: resetting search direction.
ElecMinimize: Iter:  34  F: -860.993956865533278  |grad|_K:       -nan  alpha:  1.040e-01
ElecMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 3.120007e-01.
        FillingsUpdate:  mu: -0.088853087  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15759  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.199
ElecMinimize: Iter:  35  F: -860.993960674960476  |grad|_K:       -nan  alpha:  8.250e-01  linmin: -8.511e-06  t[s]:    788.81
        FillingsUpdate:  mu: -0.088847626  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15755  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.242
ElecMinimize: Iter:  36  F: -860.993962451721814  |grad|_K:       -nan  alpha:  4.969e-01  linmin:  3.406e-07  t[s]:    797.97
        FillingsUpdate:  mu: -0.088862551  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15759  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.313
ElecMinimize: Iter:  37  F: -860.993964114890673  |grad|_K:       -nan  alpha:  5.796e-01  linmin: -3.424e-06  t[s]:    807.13
        FillingsUpdate:  mu: -0.088855800  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15756  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.41
ElecMinimize: Iter:  38  F: -860.993965023431201  |grad|_K:       -nan  alpha:  6.381e-01  linmin: -7.923e-07  t[s]:    816.02
        FillingsUpdate:  mu: -0.088843281  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15754  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.487
ElecMinimize: Iter:  39  F: -860.993965638887175  |grad|_K:       -nan  alpha:  6.287e-01  linmin: -1.814e-07  t[s]:    825.17
        FillingsUpdate:  mu: -0.088851726  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15756  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.537
ElecMinimize: Iter:  40  F: -860.993966234829259  |grad|_K:       -nan  alpha:  6.132e-01  linmin:  6.860e-08  t[s]:    833.84
        FillingsUpdate:  mu: -0.088851608  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15755  Tot: -0.00000 ]
        SubspaceRotationAdjust: set factor to 0.741
ElecMinimize: Iter:  41  F: -860.993966462220442  |grad|_K:       -nan  alpha:  1.079e-01  linmin: -3.064e-06  t[s]:    842.49
        FillingsUpdate:  mu: -0.088847824  nElectrons: 524.000000  magneticMoment: [ Abs: 1.15754  Tot: +0.00000 ]
        SubspaceRotationAdjust: set factor to 0.698
ElecMinimize: Iter:  42  F: -860.993966826246492  |grad|_K:       -nan  alpha:  2.438e-01  linmin:  9.508e-07  t[s]:    851.63
ElecMinimize: Converged (|Delta F|<1.000000e-06 for 5 iters).
Setting wave functions to eigenvectors of Hamiltonian

# Stress tensor in Cartesian coordinates [Eh/a0^3]:
[ -2.01881e-05   1.7499e-09 -2.02393e-09  ]
[   1.7499e-09   2.6804e-05 -4.48833e-09  ]
[ -2.02393e-09 -4.48833e-09  1.77343e-06  ]

# Ionic positions in lattice coordinates:
ion C   0.000000000000000   0.999700000000000   0.446000000000000 1
ion C   0.000000000000000   0.499800000000000   0.444300000000000 1
ion C   0.500000000000000   0.000300000000000   0.446200000000000 1
ion C   0.438100000000000   0.127400000000000   0.446100000000000 1
ion C   0.436200000000000   0.618500000000000   0.446400000000000 1
ion C   0.937400000000000   0.124500000000000   0.445400000000000 1
ion C   0.937700000000000   0.625100000000000   0.444700000000000 1
ion C   0.374700000000000   0.252500000000000   0.445900000000000 1
ion C   0.375400000000000   0.747600000000000   0.446100000000000 1
ion C   0.875300000000000   0.249400000000000   0.444800000000000 1
ion C   0.875000000000000   0.750100000000000   0.445300000000000 1
ion C   0.309600000000000   0.375000000000000   0.445400000000000 1
ion C   0.313200000000000   0.874300000000000   0.445900000000000 1
ion C   0.814000000000000   0.374500000000000   0.444600000000000 1
ion C   0.812100000000000   0.874900000000000   0.445600000000000 1
ion C   0.250700000000000   0.999900000000000   0.445700000000000 1
ion C   0.247500000000000   0.499500000000000   0.445100000000000 1
ion C   0.749300000000000   0.999900000000000   0.445600000000000 1
ion C   0.752400000000000   0.499500000000000   0.444900000000000 1
ion C   0.187600000000000   0.124600000000000   0.445300000000000 1
ion C   0.186700000000000   0.625000000000000   0.445100000000000 1
ion C   0.686900000000000   0.125700000000000   0.445700000000000 1
ion C   0.688800000000000   0.624300000000000   0.445600000000000 1
ion C   0.124600000000000   0.249400000000000   0.444800000000000 1
ion C   0.125000000000000   0.750100000000000   0.445500000000000 1
ion C   0.625200000000000   0.252500000000000   0.445900000000000 1
ion C   0.624500000000000   0.747600000000000   0.446000000000000 1
ion C   0.062000000000000   0.374500000000000   0.444400000000000 1
ion C   0.062600000000000   0.875000000000000   0.446000000000000 1
ion C   0.561900000000000   0.873300000000000   0.446100000000000 1
ion C   0.125200000000000   0.999700000000000   0.445800000000000 1
ion C   0.123900000000000   0.499800000000000   0.444500000000000 1
ion C   0.624200000000000   0.000200000000000   0.446000000000000 1
ion C   0.629100000000000   0.499800000000000   0.445900000000000 1
ion C   0.062500000000000   0.124500000000000   0.445400000000000 1
ion C   0.062200000000000   0.625100000000000   0.444900000000000 1
ion C   0.561800000000000   0.127400000000000   0.446100000000000 1
ion C   0.563800000000000   0.618500000000000   0.446300000000000 1
ion C   0.000000000000000   0.249300000000000   0.444800000000000 1
ion C   0.000000000000000   0.750100000000000   0.445500000000000 1
ion C   0.500000000000000   0.256700000000000   0.446300000000000 1
ion C   0.500000000000000   0.745000000000000   0.446300000000000 1
ion C   0.438000000000000   0.873300000000000   0.446200000000000 1
ion C   0.937900000000000   0.374500000000000   0.444300000000000 1
ion C   0.937300000000000   0.875000000000000   0.445900000000000 1
ion C   0.375800000000000   0.000200000000000   0.446000000000000 1
ion C   0.370800000000000   0.499800000000000   0.445900000000000 1
ion C   0.874700000000000   0.999700000000000   0.445700000000000 1
ion C   0.876000000000000   0.499800000000000   0.444400000000000 1
ion C   0.313000000000000   0.125700000000000   0.445700000000000 1
ion C   0.311200000000000   0.624300000000000   0.445700000000000 1
ion C   0.812300000000000   0.124600000000000   0.445300000000000 1
ion C   0.813300000000000   0.625000000000000   0.444900000000000 1
ion C   0.249300000000000   0.249700000000000   0.445200000000000 1
ion C   0.250000000000000   0.749700000000000   0.445700000000000 1
ion C   0.750600000000000   0.249700000000000   0.445200000000000 1
ion C   0.749900000000000   0.749700000000000   0.445500000000000 1
ion C   0.185900000000000   0.374500000000000   0.444700000000000 1
ion C   0.187900000000000   0.874900000000000   0.445700000000000 1
ion C   0.690300000000000   0.375000000000000   0.445400000000000 1
ion C   0.686800000000000   0.874300000000000   0.445800000000000 1
ion C   0.062600000000000   0.041200000000000   0.445800000000000 1
ion C   0.062000000000000   0.541700000000000   0.444500000000000 1
ion C   0.561900000000000   0.042900000000000   0.446100000000000 1
ion C   0.000000000000000   0.166100000000000   0.445200000000000 1
ion C   0.000000000000000   0.666900000000000   0.445000000000000 1
ion C   0.500000000000000   0.171200000000000   0.446200000000000 1
ion C   0.500000000000000   0.659500000000000   0.446400000000000 1
ion C   0.436200000000000   0.297700000000000   0.446200000000000 1
ion C   0.438100000000000   0.788800000000000   0.446300000000000 1
ion C   0.937700000000000   0.291100000000000   0.444600000000000 1
ion C   0.937400000000000   0.791700000000000   0.445600000000000 1
ion C   0.370800000000000   0.416400000000000   0.445900000000000 1
ion C   0.375800000000000   0.916000000000000   0.446000000000000 1
ion C   0.876000000000000   0.416400000000000   0.444400000000000 1
ion C   0.874700000000000   0.916500000000000   0.445800000000000 1
ion C   0.313200000000000   0.041900000000000   0.445800000000000 1
ion C   0.309600000000000   0.541200000000000   0.445500000000000 1
ion C   0.812100000000000   0.041300000000000   0.445500000000000 1
ion C   0.814000000000000   0.541700000000000   0.444700000000000 1
ion C   0.250000000000000   0.166500000000000   0.445400000000000 1
ion C   0.249300000000000   0.666500000000000   0.445500000000000 1
ion C   0.749900000000000   0.166500000000000   0.445400000000000 1
ion C   0.750600000000000   0.666500000000000   0.445300000000000 1
ion C   0.186700000000000   0.291200000000000   0.444800000000000 1
ion C   0.187600000000000   0.791600000000000   0.445700000000000 1
ion C   0.688800000000000   0.291900000000000   0.445500000000000 1
ion C   0.686900000000000   0.790500000000000   0.445800000000000 1
ion C   0.123900000000000   0.416400000000000   0.444500000000000 1
ion C   0.125200000000000   0.916500000000000   0.445800000000000 1
ion C   0.629100000000000   0.416400000000000   0.445800000000000 1
ion C   0.624200000000000   0.916000000000000   0.446000000000000 1
ion C   0.187900000000000   0.041300000000000   0.445500000000000 1
ion C   0.185900000000000   0.541700000000000   0.444900000000000 1
ion C   0.686800000000000   0.041900000000000   0.445700000000000 1
ion C   0.690300000000000   0.541200000000000   0.445400000000000 1
ion C   0.125000000000000   0.166100000000000   0.445200000000000 1
ion C   0.124600000000000   0.666800000000000   0.445200000000000 1
ion C   0.624500000000000   0.168600000000000   0.445900000000000 1
ion C   0.625200000000000   0.663700000000000   0.446000000000000 1
ion C   0.062200000000000   0.291100000000000   0.444600000000000 1
ion C   0.062500000000000   0.791700000000000   0.445700000000000 1
ion C   0.563800000000000   0.297700000000000   0.446200000000000 1
ion C   0.561800000000000   0.788800000000000   0.446200000000000 1
ion C   0.000000000000000   0.416400000000000   0.444300000000000 1
ion C   0.000000000000000   0.916500000000000   0.446100000000000 1
ion C   0.500000000000000   0.915900000000000   0.446200000000000 1
ion C   0.438000000000000   0.042900000000000   0.446100000000000 1
ion C   0.937300000000000   0.041200000000000   0.445800000000000 1
ion C   0.937900000000000   0.541700000000000   0.444400000000000 1
ion C   0.375400000000000   0.168600000000000   0.445900000000000 1
ion C   0.374700000000000   0.663700000000000   0.446100000000000 1
ion C   0.875000000000000   0.166100000000000   0.445200000000000 1
ion C   0.875300000000000   0.666800000000000   0.445000000000000 1
ion C   0.311200000000000   0.291900000000000   0.445500000000000 1
ion C   0.313000000000000   0.790500000000000   0.445900000000000 1
ion C   0.813300000000000   0.291200000000000   0.444800000000000 1
ion C   0.812300000000000   0.791600000000000   0.445500000000000 1
ion C   0.247500000000000   0.416700000000000   0.445000000000000 1
ion C   0.250700000000000   0.916300000000000   0.445800000000000 1
ion C   0.752400000000000   0.416700000000000   0.444900000000000 1
ion C   0.749300000000000   0.916300000000000   0.445700000000000 1
ion N   0.567200000000000   0.378800000000000   0.446300000000000 1
ion N   0.432700000000000   0.378800000000000   0.446300000000000 1
ion N   0.567200000000000   0.537400000000000   0.446300000000000 1
ion N   0.432700000000000   0.537400000000000   0.446400000000000 1
ion Fe   0.500000000000000   0.458100000000000   0.446600000000000 1

# Forces in Lattice coordinates:
force C  -0.036097386784225  -0.012503623163075  -0.003772081904265 1
force C  -0.035873894699910  -0.012504932294282   0.007375468172502 1
force C  -0.028028589893123  -0.077137048828192  -0.009761203876510 1
force C  -0.017573262724174  -0.043734875068353  -0.000055342885653 1
force C  -0.071041898171815  -0.120796463317057  -0.015992496926845 1
force C   0.037517060902757  -0.034611214185032   0.004790092912465 1
force C   0.018514022316299  -0.028097801141182   0.013424040062001 1
force C   0.000360959111735  -0.053239295018969  -0.003952881134491 1
force C  -0.035483826085421  -0.000988893407658   0.005290951634862 1
force C   0.048818413836186  -0.016502554288980   0.007754933225403 1
force C  -0.039105974900389  -0.005680415101136   0.016459609794697 1
force C  -0.063321559792616  -0.019688754887696   0.005943319734783 1
force C   0.006257522204443  -0.043148697445442   0.001485312701575 1
force C  -0.002433229240504  -0.015586716111945   0.008874913265070 1
force C  -0.026281363045459  -0.016023343634304   0.017861948280451 1
force C  -0.035096507426636  -0.034455774501825  -0.004387505743586 1
force C   0.024078461848368   0.053905125559188  -0.003894367644155 1
force C   0.024861802092846  -0.037214232358461   0.003749394661732 1
force C  -0.025395956807879   0.053256951531179   0.008131190935610 1
force C   0.028749434237349  -0.031549259019042   0.011637120461223 1
force C  -0.015003306182148  -0.028273436229434   0.017585539487069 1
force C  -0.012922019371060   0.001039417582070  -0.009657216660319 1
force C  -0.065235433732405  -0.017068234265641  -0.010670208349956 1
force C   0.003626486635803  -0.011154999702205   0.005915262643500 1
force C  -0.046595756737180  -0.008803311742135   0.011383457742866 1
force C   0.064617478084510  -0.053663694285107  -0.003236865696079 1
force C   0.040742067591685   0.000190143848378   0.004732213519668 1
force C  -0.020006062699454  -0.054965058141328   0.001781871401040 1
force C  -0.010759419315673  -0.047349148438009  -0.017515839906763 1
force C   0.055391083688216  -0.070437096061042   0.015189674980115 1
force C   0.048422362002726   0.009112368541472  -0.021791785143250 1
force C   0.025539079772126  -0.019684827237635   0.020022997372754 1
force C  -0.006552505788595  -0.044742296127230  -0.014437184340244 1
force C  -0.198878301760213   0.138719645859399  -0.015405324683375 1
force C   0.020478108541744  -0.031422319404903   0.004279116375097 1
force C   0.018225282896658  -0.031539272385308  -0.007197340623151 1
force C   0.044197578738486  -0.052905474683701  -0.001073491301682 1
force C   0.027958103149670  -0.127224158071307  -0.007878440087737 1
force C  -0.036393706590931   0.028089102224928  -0.008300501417182 1
force C  -0.027342660337998   0.023480302467191  -0.008426237682990 1
force C  -0.005634086522251  -0.104577900297168  -0.006408876348479 1
force C  -0.034558687777656  -0.073112185568470   0.011473638075696 1
force C   0.007839720049560  -0.074372699328789  -0.002465767361538 1
force C   0.050413433331266  -0.049163028310544   0.016985550681363 1
force C   0.045334698788214  -0.035735151615916   0.001153586371516 1
force C  -0.024171855782339  -0.046857326683328  -0.006889685177400 1
force C   0.205280693184667   0.138963057626908   0.011159095557654 1
force C  -0.004652373266775   0.006174125144057   0.008738767747161 1
force C  -0.019467410796525  -0.021950608251537   0.008436153419004 1
force C   0.028430018979317  -0.000783938485020   0.000061370652906 1
force C  -0.008741840191669  -0.018488364193607  -0.002315719975447 1
force C   0.019795973808260  -0.020627068222810   0.013040856262681 1
force C  -0.067349159385127  -0.026453053995249   0.013263327838819 1
force C   0.007995675680757  -0.001599306251679   0.000609720977068 1
force C   0.017547274274465   0.005531486508184   0.001917503114254 1
force C   0.054263526394573   0.000368587337162   0.002421525232527 1
force C  -0.014240065540952   0.006445679397477   0.005800499819499 1
force C   0.019007287235732  -0.013015297305793   0.000777195146734 1
force C  -0.022875546858226  -0.000495080648626   0.016079137380285 1
force C   0.078415665410892  -0.017945236394775  -0.010487532454550 1
force C  -0.025201567407128  -0.042030906836958   0.009686207199005 1
force C  -0.010929242303030   0.046827661777939  -0.002017575576204 1
force C  -0.019979296269648   0.054724067355677   0.000492439797845 1
force C   0.055496182932279   0.070463320660032   0.006480457080191 1
force C  -0.027329048239077  -0.023787441269512   0.003219260259954 1
force C  -0.036474812132952  -0.028487060404234   0.003621818762462 1
force C  -0.034611677466029   0.073683111468864   0.000051138862165 1
force C  -0.005515096566741   0.104760344973144   0.003199827851956 1
force C  -0.070818483989507   0.121051398636629   0.000460163697152 1
force C  -0.017616107781601   0.043493659447055  -0.016188870379476 1
force C   0.018434862880969   0.028985289484984  -0.003490730724155 1
force C   0.037370002489574   0.035025452538021  -0.002692059872000 1
force C   0.204893521482534  -0.138683932913423  -0.007062714947035 1
force C  -0.023988628706185   0.046720899306969   0.012069720942651 1
force C  -0.019248245830440   0.022225834126258  -0.013534410126989 1
force C  -0.004714265245073  -0.005863534549432  -0.011875528255511 1
force C   0.006314412612736   0.042700245746807  -0.001810256100985 1
force C  -0.063498831792564   0.019009598740275   0.009048172553269 1
force C  -0.026312333402679   0.015349867853470   0.001915858198347 1
force C  -0.002087258106011   0.014958007936220  -0.009790589458283 1
force C   0.017711175582702  -0.005852391769549  -0.007440615148765 1
force C   0.007837285462678   0.001314996719395  -0.003385481197399 1
force C  -0.014224742788586  -0.006678740568987  -0.005982241942565 1
force C   0.054211946791801  -0.000602297262604   0.001211788022840 1
force C  -0.014839766577493   0.028931032445900   0.008523221217621 1
force C   0.028631377421218   0.031952622330538  -0.017512052654544 1
force C  -0.065278486157235   0.017560630909904  -0.000371454174188 1
force C  -0.012896453917895  -0.000496626638239  -0.008306586759378 1
force C   0.025145479029640   0.020120980720367  -0.010265884113082 1
force C   0.048158441713945  -0.008681869341191   0.017763215527527 1
force C  -0.199705231274098  -0.138415942952218   0.020788497981252 1
force C  -0.006728703452080   0.044667438511479  -0.006378284510126 1
force C  -0.022958185163282  -0.000287830572408   0.021446725073374 1
force C   0.018547452469214   0.012394411030923  -0.014013751916423 1
force C  -0.025378167601818   0.041519650478866   0.015700012385604 1
force C   0.078038701376773   0.017081026590981   0.010728837028028 1
force C  -0.046680444602538   0.008973777949270  -0.015036266549444 1
force C   0.003511314964655   0.010331144422929  -0.012702684579983 1
force C   0.040692849650993  -0.000708116609257   0.004190360585501 1
force C   0.064638946170282   0.054194360344507  -0.000431706090774 1
force C   0.018167499204079   0.031943664051070   0.006034268542693 1
force C   0.020418900292609   0.031771691293395   0.002501721144232 1
force C   0.027844390202349   0.127626628842346   0.000589211534269 1
force C   0.044226244865677   0.052611799588268  -0.011712741515229 1
force C  -0.035931313694614   0.012843278201109  -0.010127358927352 1
force C  -0.036146239055969   0.012949167522110  -0.010125906978639 1
force C  -0.027993245994786   0.076854535791331  -0.001133372303244 1
force C   0.007877640576171   0.074426117276883   0.005779368910271 1
force C   0.045379861006147   0.035198650617194  -0.013723163212577 1
force C   0.050441045848274   0.048642798295208  -0.001732082334545 1
force C  -0.035425947472575   0.000451846570922   0.002838720350656 1
force C   0.000300025570067   0.053771052309696  -0.001049279632994 1
force C  -0.039136544432975   0.005979659066186  -0.013259325871945 1
force C   0.048935116012416   0.015759527445782  -0.020029132501009 1
force C  -0.008863571836545   0.018997142081593  -0.003108597899592 1
force C   0.028531953540629   0.001365698108242   0.001223282731763 1
force C  -0.067392214212154   0.027208097256176   0.000898414794163 1
force C   0.019912438730635   0.021114547945457  -0.012801763343784 1
force C   0.024136255438392  -0.053560965789876   0.001574214909321 1
force C  -0.035040797438599   0.034622529303151  -0.010282715293992 1
force C  -0.025133499997746  -0.052938732405841   0.000263386728093 1
force C   0.024851726285830   0.037310467606237  -0.012242189397644 1
force N   0.309044250440172  -0.213349486654035  -0.016811898392880 1
force N  -0.280970348316565  -0.211504019736010  -0.006808319818319 1
force N   0.308717813019658   0.210079273457236   0.003954941241846 1
force N  -0.281010006937855   0.208163269812653  -0.006298634238608 1
force Fe  -0.032008443659494   0.003325007333651   0.003476082195820 1

# Energy components:
   Eewald =     3622.1040525149132918
       EH =     4313.6658415792844607
     Eloc =    -9120.8009111300088989
      Enl =      123.2693971432083799
      Exc =     -246.9034426694223043
 Exc_core =        5.2485462687340156
       KE =      442.4068107028130044
        U =        0.0593774951773265
-------------------------------------
     Etot =     -860.9503280953010744
       TS =        0.0436387309454157
-------------------------------------
        F =     -860.9939668262464920

Dumping '661.ionpos' ... done
Dumping '661.stress' ... done
IonicMinimize: Iter:   0  F: -860.993966826246492  |grad|_K:  1.570e-03  t[s]:    856.57
IonicMinimize: None of the convergence criteria satisfied after 0 iterations.

#--- Lowdin population analysis ---
# oxidation-state C +0.042 +0.043 +0.040 +0.036 +0.162 +0.042 +0.040 +0.028 +0.043 +0.043 +0.043 +0.030 +0.043 +0.046 +0.042 +0.041 +0.038 +0.041 +0.038 +0.042 +0.041 +0.040 +0.051 +0.043 +0.043 +0.027 +0.043 +0.043 +0.043 +0.042 +0.042 +0.043 +0.042 +0.145 +0.042 +0.040 +0.036 +0.162 +0.044 +0.041 -0.011 +0.049 +0.042 +0.043 +0.043 +0.042 +0.145 +0.042 +0.043 +0.040 +0.051 +0.042 +0.041 +0.037 +0.045 +0.037 +0.045 +0.046 +0.042 +0.030 +0.042 +0.043 +0.043 +0.042 +0.041 +0.044 +0.049 -0.011 +0.162 +0.036 +0.040 +0.042 +0.145 +0.042 +0.043 +0.042 +0.043 +0.030 +0.042 +0.046 +0.045 +0.037 +0.045 +0.037 +0.041 +0.042 +0.051 +0.040 +0.043 +0.042 +0.145 +0.042 +0.042 +0.046 +0.042 +0.030 +0.043 +0.043 +0.043 +0.027 +0.040 +0.042 +0.162 +0.036 +0.043 +0.042 +0.040 +0.042 +0.043 +0.043 +0.043 +0.028 +0.043 +0.043 +0.051 +0.040 +0.041 +0.042 +0.038 +0.041 +0.038 +0.041
# magnetic-moments C +0.000 +0.000 +0.000 +0.000 +0.000 -0.000 +0.000 +0.000 -0.000 -0.000 +0.000 -0.000 +0.000 -0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 -0.000 +0.000 -0.000 +0.000 -0.000 -0.000 +0.000 +0.000 -0.000 +0.000 +0.000 +0.000 +0.001 -0.000 +0.000 +0.000 -0.000 -0.000 +0.000 +0.000 -0.000 -0.000 +0.000 +0.000 +0.000 +0.001 +0.000 -0.000 -0.000 +0.000 +0.000 +0.000 -0.000 +0.000 -0.000 +0.000 -0.000 +0.000 -0.000 +0.000 +0.000 +0.000 +0.000 +0.000 -0.000 +0.000 +0.001 -0.000 +0.000 +0.000 +0.000 +0.001 +0.000 +0.000 +0.000 +0.000 -0.000 +0.000 -0.000 +0.000 -0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.001 +0.000 +0.000 -0.000 +0.000 -0.000 +0.000 -0.000 +0.000 +0.000 +0.000 -0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 -0.000 +0.000 -0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000 +0.000
# oxidation-state N -0.137 -0.137 -0.137 -0.137
# magnetic-moments N +0.000 +0.000 +0.000 +0.000
# oxidation-state Fe +0.202
# magnetic-moments Fe -0.007

Dumping '661.fillings' ... done
Dumping '661.wfns' ... done
Dumping '661.eigenvals' ... done
Dumping '661.stress' ... done
End date and time: Sun May 19 15:11:27 2024  (Duration: 0-0:14:31.55)
Done!

PROFILER:             augmentDensityGrid     0.021368 +/-     0.025112 s,  738 calls,     15.769669 s total
PROFILER:         augmentDensityGridGrad     0.166410 +/-     0.201705 s,  363 calls,     60.406814 s total
PROFILER:        augmentDensitySpherical     0.005125 +/-     0.006784 s, 1476 calls,      7.564255 s total
PROFILER:    augmentDensitySphericalGrad     0.004469 +/-     0.005899 s,  726 calls,      3.244775 s total
PROFILER:                 augmentOverlap     0.051761 +/-     0.069630 s, 1062 calls,     54.970294 s total
PROFILER:                     changeGrid     0.000337 +/-     0.000420 s,  972 calls,      0.327116 s total
PROFILER:        ColumnBundle::randomize     1.063495 +/-     0.036999 s,    2 calls,      2.126989 s total
PROFILER:                     diagouterI     0.517767 +/-     0.081313 s,  492 calls,    254.741297 s total
PROFILER:              EdensityAndVscloc     0.365623 +/-     0.010856 s,  247 calls,     90.308770 s total
PROFILER:                     EnlAndGrad     0.059926 +/-     0.091796 s,  546 calls,     32.719536 s total
PROFILER:            ExCorrCommunication     0.055855 +/-     0.049240 s, 1501 calls,     83.838750 s total
PROFILER:               ExCorrFunctional     0.000402 +/-     0.000978 s,  252 calls,      0.101314 s total
PROFILER:                    ExCorrTotal     0.351424 +/-     0.031971 s,  252 calls,     88.558796 s total
PROFILER:                   Idag_DiagV_I     0.682800 +/-     0.032498 s,  240 calls,    163.871955 s total
PROFILER:                    inv(matrix)     0.013077 +/-     0.002065 s,  174 calls,      2.275331 s total
PROFILER:            matrix::diagonalize     0.010427 +/-     0.003113 s,  724 calls,      7.549204 s total
PROFILER:                    matrix::set     0.000010 +/-     0.000010 s, 54382 calls,      0.552553 s total
PROFILER:            orthoMatrix(matrix)     0.010938 +/-     0.006524 s,  178 calls,      1.946932 s total
PROFILER:     RadialFunctionR::transform     0.009799 +/-     0.085326 s,  101 calls,      0.989742 s total
PROFILER:                    reduceKmesh     0.000256 +/-     0.000000 s,    1 calls,      0.000256 s total
PROFILER:                   rhoAtom_calc     0.002044 +/-     0.003016 s,  738 calls,      1.508114 s total
PROFILER:                   rhoAtom_grad     0.001802 +/-     0.002547 s,  714 calls,      1.286689 s total
PROFILER:               WavefunctionDrag     0.640644 +/-     0.000000 s,    1 calls,      0.640644 s total
PROFILER:                            Y*M     0.020006 +/-     0.017399 s, 3033 calls,     60.676770 s total
PROFILER:                          Y1^Y2     0.017627 +/-     0.025617 s, 2382 calls,     41.987971 s total

MEMUSAGE:                   ColumnBundle    17.921206 GB
MEMUSAGE:        complexScalarFieldTilde     0.056076 GB
MEMUSAGE:                    IndexArrays     0.050941 GB
MEMUSAGE:                         matrix     0.128838 GB
MEMUSAGE:                           misc     0.116134 GB
MEMUSAGE:                    ScalarField     0.757027 GB
MEMUSAGE:               ScalarFieldTilde     0.546366 GB
MEMUSAGE:                          Total    18.423792 GB
shankar1729 commented 4 months ago

The grad K nan may be a separate algorithm issue; see other discussions. Instead compare to same calculation on different tested hardware to be sure.

The calculation itself seems to be running fine across multiple GPUs and nodes. Don't forget to add the mempool variable for further speed up.

ColinBundschu commented 4 months ago

Could you point me to the relevant discussions? I tried searching for them and I was not able to locate anything that seemed relevant. I will try it on a system I know works in the meantime.

Should the mempool be set to the memory of a single GPU or the collection of all GPUs?

ColinBundschu commented 4 months ago

221_nlpcm.zip Also it looks like everything works with nonlinear pcm fluid on a different calc

shankar1729 commented 4 months ago

Hi Colin,

See the last comment here for the grad_k=nan discussion: https://github.com/shankar1729/jdftx/issues/242

The memory pool size is per process and GPU. So don't exceed around 38000 for the A100s. For the calculation you just shared, note that only 4 GPUs are actually active due to nStates=4, so rerun with single node with 4 GPUs and memory pool set. You should hopefully see a modest speedup from that.

Best, Shankar

ColinBundschu commented 4 months ago

It was ridiculously fast. It took 5 minutes 30 seconds to run.

shankar1729 commented 4 months ago

Great, maybe you can post the final compilation flags and job example that worked at the end of this thread before closing the issue? That would be very useful for future reference for anyone looking.

ColinBundschu commented 4 months ago

clean.zip

Here is the complete setup:

# Set up cblas
mkdir -p ~/local/lib
ln -s /usr/lib64/libcblas.so.3 ~/local/lib/libcblas.so

# Set up GSL
wget https://ftp.gnu.org/gnu/gsl/gsl-latest.tar.gz
tar -xzf gsl-latest.tar.gz
cd gsl-*/
./configure --prefix=$HOME/gsl
make
make install

#cmake and nvidia modules
module use /soft/modulefiles
module load spack-pe-base cmake
module load craype-accel-nvidia80
export MPICH_GPU_SUPPORT_ENABLED=1

#compilers
export CC=cc
export CXX=CC

# run from jdftx/build after "rm -rf *"
cmake\
 -D CBLAS_PATH=/home/cbu/local/lib\
 -D GSL_PATH=/home/cbu/gsl\
 -D FFTW3_PATH=/opt/cray/pe/fftw/3.3.10.6/x86_milan\
 -D CMAKE_PREFIX_PATH=/opt/nvidia/hpc_sdk/Linux_x86_64/23.9/math_libs/12.2/lib64\
 -D CMAKE_CUDA_ARCHITECTURES="80"\
 -D EnableProfiling=yes\
 -D EnableCUDA=yes\
 -D EnableCuSolver=yes\
 -D CudaAwareMPI=yes\
 -D PinnedHostMemory=yes\
 -D CMAKE_LIBRARY_PATH="${LD_LIBRARY_PATH//:/;}"\
 -D CMAKE_CXX_FLAGS="-Wl,--no-warn-execstack --diag_suppress=unsigned_compare_with_zero"\
 ../jdftx-git/jdftx