search

shankar1729 / jdftx

JDFTx: software for joint density functional theory
http://jdftx.org
83 stars 54 forks source link

MPI calculation fails on Polaris when split over 2 nodes with fluid (nan Energies) #334

Closed ColinBundschu closed 4 weeks ago

ColinBundschu commented 5 months ago

Complete input and outputs here: test.zip

It seems like there is an issue with the MPI encoding/decoding leading to nan

# Energy components:
   A_diel =                      -nan
   Eewald =     3622.1040525149132918
       EH =                      -nan
     Eloc =                      -nan
      Enl =                      -nan
      Exc =                      -nan
 Exc_core =        5.2485462687340156
       KE =                      -nan
        U =                      -nan
-------------------------------------
     Etot =                      -nan
       TS =        0.3917036407203660
-------------------------------------
        F =                      -nan
shankar1729 commented 5 months ago

Is this only happening for fluid jobs, or for all calculations, given that all the energy components are turning NAN?

Does it happen only for GPU jobs? This could be a cuda-aware MPI issue if limited to GPU jobs.

ColinBundschu commented 5 months ago

It does not happen on fluid jobs or vacuum jobs run on multiple GPUs on a single node (to be clear, these jobs run correctly). Only when MPI is used between two nodes w/ GPU does this happen. I am not actually sure that the fluid part is relevant - I did not check it without fluid.

ColinBundschu commented 5 months ago

Is this relevant?

ElecMinimize: Iter:  27  F: -861.081904643185226  |grad|_K:        nan  alpha:  7.362e-07
ElecMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 3.000000e-01.
    FillingsUpdate:  mu: -0.092921451  nElectrons: 524.000000  magneticMoment: [ Abs: 0.01684  Tot: -0.00134 ]
    SubspaceRotationAdjust: set factor to 0.28
ElecMinimize: Iter:  28  F: -861.082133209565541  |grad|_K:        nan  alpha:  5.120e-01  linmin: -1.634e-07  t[s]:    363.49
ElecMinimize:   Step increased F by 1.579920e-04, reducing alpha to 1.052135e-02.
WARNING: 499 elements failed to converge in cusolverDn eigenvalue routine Zheevd; falling back to CPU LAPACK.
WARNING: 500 elements failed to converge in cusolverDn eigenvalue routine Zheevd; falling back to CPU LAPACK.
ElecMinimize:   Step failed with F = -nan, reducing alpha to 1.052135e-03.
ColinBundschu commented 5 months ago

This example seems diagnostic as well:


*************** JDFTx 1.7.0 (git hash 7de33975) ***************

Start date and time: Thu May 30 19:46:49 2024
Executable /home/cbu/jdftx/build/jdftx_gpu with command-line: -i 441_nlpcm.in -o 441_nlpcm.out
Running on hosts (process indices):  x3109c0s7b0n0 (0-3)  x3109c0s7b1n0 (4-7)
Divided in process groups (process indices):  0 (0)  1 (1)  2 (2)  3 (3)  4 (4)  5 (5)  6 (6)  7 (7)
gpuInit: Found compatible cuda device 0 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 1 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 2 'NVIDIA A100-SXM4-40GB'
gpuInit: Found compatible cuda device 3 'NVIDIA A100-SXM4-40GB'
gpuInit: Selected device 0
Resource initialization completed at t[s]:      0.67
Run totals: 8 processes, 64 threads, 8 GPUs
Memory pool size: 38000 MB (per process)

Input parsed successfully to the following command list (including defaults):

add-U   Fe d 0.129
basis kpoint-dependent
coords-type Lattice
core-overlap-check vector
coulomb-interaction Periodic
davidson-band-ratio 1.1
dump End State Stress
dump Ionic IonicPositions Stress
dump-name $INPUT.$VAR
elec-cutoff 20 100
elec-eigen-algo Davidson
elec-ex-corr gga-PBE
elec-smearing Fermi 0.02
electronic-minimize  \
    dirUpdateScheme      FletcherReeves \
    linminMethod         DirUpdateRecommended \
    nIterations          150 \
    history              15 \
    knormThreshold       0 \
    maxThreshold         no \
    energyDiffThreshold  1e-06 \
    nEnergyDiff          5 \
    alphaTstart          0.1 \
    alphaTmin            1e-10 \
    updateTestStepSize   yes \
    alphaTreduceFactor   0.1 \
    alphaTincreaseFactor 3 \
    nAlphaAdjustMax      3 \
    wolfeEnergy          0.0001 \
    wolfeGradient        0.9 \
    fdTest               no
exchange-regularization WignerSeitzTruncated
fluid NonlinearPCM 298.000000 1.013250
fluid-anion F- 1 MeanFieldLJ \
    epsBulk 1 \
    pMol 0 \
    epsInf 1 \
    Pvap 0 \
    sigmaBulk 0 \
    Rvdw 2.24877 \
    Res 0 \
    tauNuc 343133
fluid-cation Na+ 1 MeanFieldLJ \
    epsBulk 1 \
    pMol 0 \
    epsInf 1 \
    Pvap 0 \
    sigmaBulk 0 \
    Rvdw 2.19208 \
    Res 0 \
    tauNuc 343133
fluid-ex-corr lda-TF lda-PZ
fluid-gummel-loop 10 1.000000e-05
fluid-minimize  \
    dirUpdateScheme      PolakRibiere \
    linminMethod         DirUpdateRecommended \
    nIterations          222 \
    history              15 \
    knormThreshold       0 \
    maxThreshold         no \
    energyDiffThreshold  1e-10 \
    nEnergyDiff          2 \
    alphaTstart          0.007 \
    alphaTmin            1e-10 \
    updateTestStepSize   yes \
    alphaTreduceFactor   0.1 \
    alphaTincreaseFactor 3 \
    nAlphaAdjustMax      6 \
    wolfeEnergy          0.0001 \
    wolfeGradient        0.9 \
    fdTest               no
fluid-solvent H2O 55.338 ScalarEOS \
    epsBulk 78.4 \
    pMol 0.92466 \
    epsInf 1.77 \
    Pvap 1.06736e-10 \
    sigmaBulk 4.62e-05 \
    Rvdw 2.61727 \
    Res 1.42 \
    tauNuc 343133 \
    poleEl 15 7 1
forces-output-coords Positions
initial-state 441_nlpcm.$VAR
ion C  -0.000029707957535   0.999734075373656   0.445928226069441 1
ion C  -0.000030660203837   0.499844907911305   0.444322186596028 1
ion C   0.499970372367738   0.000280714830126   0.446154934120106 1
ion C   0.438015305987727   0.127431817301940   0.446133786332907 1
ion C   0.435820836742936   0.618613767676327   0.445833776102915 1
ion C   0.937452161300365   0.124476331537161   0.445388090202350 1
ion C   0.937724355040379   0.625130877346753   0.444740465576478 1
ion C   0.374526832562430   0.252276673076671   0.445828109043488 1
ion C   0.375273370438041   0.747794284053398   0.446149721831786 1
ion C   0.875366000999245   0.249386918048184   0.444811996267458 1
ion C   0.874994427215820   0.750167816094456   0.445332331271550 1
ion C   0.309551550942670   0.374875987351058   0.445489572828667 1
ion C   0.313150509895064   0.874337377267287   0.445905891444084 1
ion C   0.813960227052455   0.374527353147648   0.444658056041645 1
ion C   0.812099469165706   0.874970944221219   0.445642451566290 1
ion C   0.250603457195469   0.999892226291721   0.445693193186235 1
ion C   0.247611804433649   0.499671157297559   0.445225706704760 1
ion C   0.749342065784283   0.999894361541899   0.445627922441240 1
ion C   0.752330127890262   0.499678373962203   0.445054796180746 1
ion C   0.187587971865206   0.124579311674992   0.445341658164641 1
ion C   0.186686814053406   0.625051996155078   0.445139452774535 1
ion C   0.686964046400407   0.125644470510937   0.445698587716742 1
ion C   0.688814237612989   0.624416798255218   0.445556728165458 1
ion C   0.124572628291732   0.249391664033795   0.444818734697330 1
ion C   0.124947681092763   0.750165095121656   0.445511251936340 1
ion C   0.625373840867768   0.252286237468929   0.445808603648308 1
ion C   0.624646980381510   0.747788946014781   0.446041101655852 1
ion C   0.062007133961267   0.374499878653393   0.444385061057204 1
ion C   0.062578350545449   0.875008154040588   0.445923393957856 1
ion C   0.561977715777834   0.873249178502399   0.446132772248917 1
ion C   0.125225357659535   0.999749782341358   0.445755187171603 1
ion C   0.123965402731164   0.499830043804282   0.444543827421787 1
ion C   0.624206175584290   0.000200254746304   0.445946847267422 1
ion C   0.628849169560916   0.500199213027952   0.445178198972218 1
ion C   0.062489127927124   0.124477827397682   0.445390155345229 1
ion C   0.062216478130419   0.625128359913442   0.444865777737386 1
ion C   0.561910178089229   0.127431419525230   0.446133108704057 1
ion C   0.564033023167032   0.618642800770310   0.445773610784156 1
ion C  -0.000030916517966   0.249371385965329   0.444772375090896 1
ion C  -0.000028939857070   0.750187134694269   0.445459262751891 1
ion C   0.499947202057239   0.256681546043961   0.446157175193274 1
ion C   0.499960330485616   0.744856014781870   0.446378659811442 1
ion C   0.437963315791167   0.873246940231938   0.446176236049555 1
ion C   0.937933248977612   0.374500804682541   0.444335224671986 1
ion C   0.937366608000279   0.875011158256017   0.445863223270998 1
ion C   0.375737916349473   0.000198004275686   0.445976748073236 1
ion C   0.371087557659699   0.500217543714721   0.445310332270646 1
ion C   0.874719480395426   0.999749434033385   0.445712277918779 1
ion C   0.875975729556790   0.499833594597233   0.444402785504607 1
ion C   0.312966925438063   0.125643892723006   0.445716129801589 1
ion C   0.311122106676233   0.624397264046538   0.445714329093068 1
ion C   0.812349574365409   0.124576107130399   0.445335962125694 1
ion C   0.813255859005537   0.625058890093383   0.444929114922949 1
ion C   0.249215281853863   0.249673316041915   0.445208288845139 1
ion C   0.249971151390751   0.749814911392065   0.445685077785866 1
ion C   0.750718700906891   0.249664077026149   0.445207015162407 1
ion C   0.749969856130867   0.749824751720888   0.445506173635731 1
ion C   0.185984222507043   0.374536733637164   0.444741389740409 1
ion C   0.187846439478187   0.874968850569466   0.445757398758088 1
ion C   0.690402766833897   0.374855805760445   0.445398413468850 1
ion C   0.686794155101733   0.874338057164529   0.445810795947367 1
ion C   0.062578252559700   0.041187935828431   0.445753489402645 1
ion C   0.062007159373190   0.541697378677913   0.444510094254811 1
ion C   0.561977522154194   0.042945428160908   0.446092496408946 1
ion C  -0.000028720931097   0.166009496261780   0.445198040753436 1
ion C  -0.000031456785661   0.666825131090442   0.445003747493612 1
ion C   0.499959193870288   0.171338662282984   0.446258445641774 1
ion C   0.499947756190214   0.659514185073599   0.446287999234636 1
ion C   0.435824900974686   0.297594827763685   0.445663621800177 1
ion C   0.438014970687155   0.788761749068432   0.446276296560717 1
ion C   0.937724355658082   0.291066364715505   0.444594972468324 1
ion C   0.937451776549768   0.791721051919954   0.445584046857019 1
ion C   0.371095236325109   0.415975601693076   0.445242788509375 1
ion C   0.375737578134043   0.915996994320677   0.446023127067455 1
ion C   0.875976734583145   0.416363659032656   0.444371807837518 1
ion C   0.874719219172485   0.916447270404974   0.445773823922606 1
ion C   0.313150343412540   0.041857728203515   0.445784683992839 1
ion C   0.309552497304054   0.541316950477765   0.445605191982716 1
ion C   0.812099480136245   0.041224162104735   0.445540019126919 1
ion C   0.813959222220814   0.541669159899156   0.444719411187060 1
ion C   0.249971068047741   0.166380587618098   0.445381622484253 1
ion C   0.249215575450700   0.666521707004682   0.445505943479218 1
ion C   0.749970247280581   0.166369966833721   0.445380783794484 1
ion C   0.750716357808668   0.666531380815207   0.445315188971194 1
ion C   0.186687488858668   0.291144316489885   0.444843113487566 1
ion C   0.187587326407784   0.791617150275487   0.445657923188448 1
ion C   0.688816097315192   0.291777924710189   0.445472765785234 1
ion C   0.686962945359704   0.790550592012889   0.445802958260784 1
ion C   0.123965970430202   0.416366785487216   0.444470244775231 1
ion C   0.125224918439458   0.916447325238592   0.445844976282335 1
ion C   0.628845947293803   0.415996038480234   0.445158324182246 1
ion C   0.624206026326245   0.915994328111704   0.445970882753547 1
ion C   0.187846245047716   0.041226229234834   0.445577763821630 1
ion C   0.185984968797380   0.541659656351465   0.444904437613512 1
ion C   0.686794131724760   0.041856228275513   0.445740212079925 1
ion C   0.690399093862252   0.541337392340211   0.445443198898003 1
ion C   0.124947707352462   0.166031148718795   0.445162986608232 1
ion C   0.124572318884695   0.666804287607905   0.445158329410321 1
ion C   0.624646559072846   0.168404605266302   0.445942603516031 1
ion C   0.625372400142354   0.663911457622512   0.445904336010831 1
ion C   0.062216706855987   0.291068373077357   0.444612860557117 1
ion C   0.062488484457463   0.791719465959812   0.445678612007524 1
ion C   0.564030021969983   0.297561897132438   0.445688774394897 1
ion C   0.561909832600928   0.788761728913306   0.446217398341270 1
ion C  -0.000030838243679   0.416352361580263   0.444277775164100 1
ion C  -0.000030130492913   0.916463508061080   0.446005686649703 1
ion C   0.499970091625480   0.915913447038932   0.446179752741310 1
ion C   0.437962547101906   0.042947684813139   0.446098956776782 1
ion C   0.937366484108135   0.041185020384516   0.445739088473453 1
ion C   0.937932656728495   0.541696490027580   0.444410054418785 1
ion C   0.375273216462532   0.168399965303887   0.445953582197257 1
ion C   0.374525826761453   0.663922804369501   0.446025889218876 1
ion C   0.874994540275412   0.166028512978554   0.445166132015055 1
ion C   0.875365238026728   0.666809909495606   0.444959118556613 1
ion C   0.311121345734956   0.291797780477080   0.445499570110804 1
ion C   0.312966624765426   0.790551938488303   0.445933039306181 1
ion C   0.813256466578631   0.291137701674143   0.444821697482967 1
ion C   0.812349159019483   0.791620366282151   0.445482903196133 1
ion C   0.247612153297023   0.416524751675460   0.445153640713813 1
ion C   0.250603348791376   0.916303676623925   0.445773578241327 1
ion C   0.752331534809626   0.416517853532229   0.445029057716477 1
ion C   0.749341833246069   0.916300678674924   0.445678645949204 1
ion N   0.567824723640165   0.378411454019348   0.444462488104865 1
ion N   0.431928410032153   0.378301497433238   0.444325030101871 1
ion N   0.567836387517307   0.537805623992100   0.444476742065708 1
ion N   0.431924098466622   0.537911834829578   0.444499847359895 1
ion Fe   0.499803939860738   0.458211757446663   0.450834969008521 1
ion O   0.498458244590283   0.457838499544820   0.533949429396895 1
ion H   0.546214242780432   0.456126955822647   0.546043316697304 1
ion-species GBRV/$ID_pbe.uspp
ion-width Ecut
ionic-minimize  \
    dirUpdateScheme      L-BFGS \
    linminMethod         DirUpdateRecommended \
    nIterations          0 \
    history              15 \
    knormThreshold       0.0001 \
    maxThreshold         no \
    energyDiffThreshold  1e-06 \
    nEnergyDiff          2 \
    alphaTstart          1 \
    alphaTmin            1e-10 \
    updateTestStepSize   yes \
    alphaTreduceFactor   0.1 \
    alphaTincreaseFactor 3 \
    nAlphaAdjustMax      3 \
    wolfeEnergy          0.0001 \
    wolfeGradient        0.9 \
    fdTest               no
kpoint   0.500000000000   0.500000000000   0.000000000000  1.00000000000000
kpoint-folding 4 4 1 
latt-move-scale 1 1 1
latt-scale 1 1 1 
lattice Orthorhombic 37.2276 32.3143 41.574
lattice-minimize  \
    dirUpdateScheme      L-BFGS \
    linminMethod         DirUpdateRecommended \
    nIterations          100 \
    history              15 \
    knormThreshold       0 \
    maxThreshold         no \
    energyDiffThreshold  1e-05 \
    nEnergyDiff          5 \
    alphaTstart          0.1 \
    alphaTmin            1e-10 \
    updateTestStepSize   yes \
    alphaTreduceFactor   0.1 \
    alphaTincreaseFactor 3 \
    nAlphaAdjustMax      3 \
    wolfeEnergy          0.0001 \
    wolfeGradient        0.9 \
    fdTest               no
lcao-params 150 1e-06 0.02
pcm-variant GLSSA13
perturb-minimize  \
    nIterations            0 \
    algorithm              MINRES \
    residualTol            0.0001 \
    residualDiffThreshold  0.0001 \
    CGBypass               no \
    recomputeResidual      no
spintype z-spin
subspace-rotation-factor 1 yes
symmetries automatic
symmetry-threshold 0.0001

---------- Setting up symmetries ----------

Found 8 point-group symmetries of the bravais lattice
Found 1 space-group symmetries with basis
Applied RMS atom displacement 0 bohrs to make symmetries exact.

---------- Initializing the Grid ----------
R = 
[      37.2276            0            0  ]
[            0      32.3143            0  ]
[            0            0       41.574  ]
unit cell volume = 50012.9
G =
[   0.168778          0          0  ]
[          0    0.19444          0  ]
[          0          0   0.151133  ]
Minimum fftbox size, Smin = [  168  148  188  ]
Chosen fftbox size, S = [  168  150  192  ]

---------- Initializing tighter grid for wavefunction operations ----------
R = 
[      37.2276            0            0  ]
[            0      32.3143            0  ]
[            0            0       41.574  ]
unit cell volume = 50012.9
G =
[   0.168778          0          0  ]
[          0    0.19444          0  ]
[          0          0   0.151133  ]
Minimum fftbox size, Smin = [  152  132  168  ]
Chosen fftbox size, S = [  160  140  168  ]

---------- Exchange Correlation functional ----------
Initalized PBE GGA exchange.
Initalized PBE GGA correlation.

---------- Setting up pseudopotentials ----------
Width of ionic core gaussian charges (only for fluid interactions / plotting) set to 0.397384

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/c_pbe.uspp':
  Title: C.  Created by USPP 7.3.6 on 3-2-2014
  Reference state energy: -5.406344.  4 valence electrons in orbitals:
    |200>  occupation: 2  eigenvalue: -0.504890
    |210>  occupation: 2  eigenvalue: -0.194356
  lMax: 1  lLocal: 2  QijEcut: 5
  4 projectors sampled on a log grid with 503 points:
    l: 0  eig: -0.504890  rCut: 1.3
    l: 0  eig: 0.000000  rCut: 1.3
    l: 1  eig: -0.194357  rCut: 1.3
    l: 1  eig: 0.000000  rCut: 1.3
  Partial core density with radius 1.1
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.30 bohrs.

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/n_pbe.uspp':
  Title: N.  Created by USPP 7.3.6 on 3-2-2014
  Reference state energy: -9.763716.  5 valence electrons in orbitals:
    |200>  occupation: 2  eigenvalue: -0.681964
    |210>  occupation: 3  eigenvalue: -0.260726
  lMax: 1  lLocal: 2  QijEcut: 6
  4 projectors sampled on a log grid with 491 points:
    l: 0  eig: -0.681964  rCut: 1.15
    l: 0  eig: 0.000000  rCut: 1.15
    l: 1  eig: -0.260729  rCut: 1.2
    l: 1  eig: 0.500000  rCut: 1.2
  Partial core density with radius 0.8
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.20 bohrs.

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/fe_pbe.uspp':
  Title: Fe.  Created by USPP 7.3.6 on 15-6-15
  Reference state energy: -124.316937.  16 valence electrons in orbitals:
    |300>  occupation: 2  eigenvalue: -4.008817
    |310>  occupation: 6  eigenvalue: -2.753096
    |320>  occupation: 5  eigenvalue: -0.791264
    |400>  occupation: 2  eigenvalue: -0.516384
    |410>  occupation: 0  eigenvalue: -0.312222
  lMax: 2  lLocal: 3  QijEcut: 6
  6 projectors sampled on a log grid with 607 points:
    l: 0  eig: -4.008819  rCut: 1.55
    l: 0  eig: -0.516386  rCut: 1.55
    l: 1  eig: -2.753096  rCut: 1.55
    l: 1  eig: -0.312226  rCut: 1.55
    l: 2  eig: -0.791264  rCut: 1.65
    l: 2  eig: -0.450000  rCut: 1.65
  Partial core density with radius 0.65
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.65 bohrs.

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/o_pbe.uspp':
  Title: O.  Created by USPP 7.3.6 on 3-2-2014
  Reference state energy: -15.894388.  6 valence electrons in orbitals:
    |200>  occupation: 2  eigenvalue: -0.878823
    |210>  occupation: 4  eigenvalue: -0.332131
  lMax: 2  lLocal: 2  QijEcut: 6
  5 projectors sampled on a log grid with 511 points:
    l: 0  eig: -0.878823  rCut: 1.25
    l: 0  eig: 0.000000  rCut: 1.25
    l: 1  eig: -0.332132  rCut: 1.25
    l: 1  eig: 0.000000  rCut: 1.25
    l: 2  eig: 1.000000  rCut: 1.25
  Partial core density with radius 0.7
  Transforming core density to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.25 bohrs.

Reading pseudopotential file '/home/cbu/jdftx/build/pseudopotentials/GBRV/h_pbe.uspp':
  Title: H.  Created by USPP 7.3.6 on 2-4-15
  Reference state energy: -0.458849.  1 valence electrons in orbitals:
    |100>  occupation: 1  eigenvalue: -0.238595
  lMax: 0  lLocal: 1  QijEcut: 6
  2 projectors sampled on a log grid with 395 points:
    l: 0  eig: -0.238595  rCut: 1.2
    l: 0  eig: 1.000000  rCut: 1.2
  Transforming local potential to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming nonlocal projectors to a uniform radial grid of dG=0.02 with 432 points.
  Transforming density augmentations to a uniform radial grid of dG=0.02 with 1255 points.
  Transforming atomic orbitals to a uniform radial grid of dG=0.02 with 432 points.
  Core radius for overlap checks: 1.20 bohrs.

Initialized 5 species with 129 total atoms.

Folded 1 k-points by 4x4x1 to 16 k-points.

---------- Setting up k-points, bands, fillings ----------
Adding inversion symmetry to k-mesh for non-inversion-symmetric unit cell.
Reduced to 8 k-points under symmetry. 
Computing the number of bands and number of electrons
Calculating initial fillings.
nElectrons: 531.000000   nBands: 522   nStates: 16

----- Setting up reduced wavefunction bases (one per k-point) -----
average nbasis = 213660.750 , ideal nbasis = 213658.714

---------- Setting up ewald sum ----------
Optimum gaussian width for ewald sums = 6.538914 bohr.
Real space sum over 343 unit cells with max indices [  3  3  3  ]
Reciprocal space sum over 9177 terms with max indices [  10  9  11  ]

---------- Allocating electronic variables ----------
Initializing wave functions:  linear combination of atomic orbitals
C pseudo-atom occupations:   s ( 2 )  p ( 2 )
N pseudo-atom occupations:   s ( 2 )  p ( 3 )
Fe pseudo-atom occupations:   s ( 2 0 )  p ( 6 0 )  d ( 8 )
O pseudo-atom occupations:   s ( 2 )  p ( 4 )
H pseudo-atom occupations:   s ( 1 )
    FillingsUpdate:  mu: -0.218488695  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00449  Tot: -0.00000 ]
LCAOMinimize: Iter:   0  F: -853.5643754224834083  |grad|_K:  3.905e-03  alpha:  1.000e+00
    FillingsUpdate:  mu: -0.243270121  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00458  Tot: -0.00000 ]
LCAOMinimize: Iter:   1  F: -857.4906873333546855  |grad|_K:  2.146e-03  alpha:  9.004e-02  linmin: -1.680e-01  cgtest:  1.099e+00  t[s]:     34.69
LCAOMinimize: Encountered beta<0, resetting CG.
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 2.701069e-01.
    FillingsUpdate:  mu: -0.253530460  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00467  Tot: +0.00000 ]
LCAOMinimize: Iter:   2  F: -859.3971037955315069  |grad|_K:  2.275e-03  alpha: -8.947e-01  linmin: -1.927e-01  cgtest:  7.431e-01  t[s]:     38.81
LCAOMinimize:   Step increased F by 6.253937e+00, reducing alpha to 2.058181e-02.
    FillingsUpdate:  mu: -0.255621284  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00470  Tot: -0.00000 ]
LCAOMinimize: Iter:   3  F: -860.2445245481177380  |grad|_K:  3.008e-03  alpha:  2.058e-02  linmin: -2.443e-01  cgtest:  7.031e-01  t[s]:     46.21
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 6.174543e-02.
    FillingsUpdate:  mu: -0.204669487  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00475  Tot: -0.00000 ]
LCAOMinimize: Iter:   4  F: -868.8857767345465390  |grad|_K:  1.872e-03  alpha: -2.741e-03  linmin: -4.297e-01  cgtest:  2.460e-01  t[s]:     50.36
LCAOMinimize:   Step increased F by 8.127568e+00, reducing alpha to 3.056556e-02.
LCAOMinimize:   Step increased F by 2.226748e+00, reducing alpha to 3.056556e-03.
    FillingsUpdate:  mu: -0.201416756  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00622  Tot: -0.00146 ]
LCAOMinimize: Iter:   5  F: -868.9962486947258640  |grad|_K:  1.468e-03  alpha:  3.057e-03  linmin: -2.340e-01  cgtest:  8.414e-01  t[s]:     61.01
LCAOMinimize: Encountered beta<0, resetting CG.
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 9.169668e-03.
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.750900e-02.
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 8.252701e-02.
    FillingsUpdate:  mu: -0.211574767  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00494  Tot: +0.00000 ]
LCAOMinimize: Iter:   6  F: -869.8220795658859288  |grad|_K:  4.632e-04  alpha:  1.730e-01  linmin:  2.339e-02  cgtest: -2.776e-01  t[s]:     67.05
    FillingsUpdate:  mu: -0.234544987  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00499  Tot: +0.00000 ]
LCAOMinimize: Iter:   7  F: -869.8429881639015093  |grad|_K:  3.386e-04  alpha:  4.708e-02  linmin: -1.643e-03  cgtest:  2.222e-01  t[s]:     71.27
    FillingsUpdate:  mu: -0.220914425  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00519  Tot: +0.00000 ]
LCAOMinimize: Iter:   8  F: -869.8767261404518649  |grad|_K:  1.688e-04  alpha:  1.398e-01  linmin:  1.960e-02  cgtest: -1.064e-01  t[s]:     75.49
    FillingsUpdate:  mu: -0.231450862  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00529  Tot: -0.00000 ]
LCAOMinimize: Iter:   9  F: -869.8810422819294672  |grad|_K:  1.319e-04  alpha:  7.332e-02  linmin:  4.211e-04  cgtest:  9.662e-02  t[s]:     79.71
    FillingsUpdate:  mu: -0.223132222  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00544  Tot: +0.00000 ]
LCAOMinimize: Iter:  10  F: -869.8844955355218644  |grad|_K:  8.412e-05  alpha:  9.036e-02  linmin: -7.083e-03  cgtest: -4.891e-02  t[s]:     83.98
    FillingsUpdate:  mu: -0.228353879  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00559  Tot: -0.00005 ]
LCAOMinimize: Iter:  11  F: -869.8866017135800348  |grad|_K:  8.119e-05  alpha:  1.327e-01  linmin: -2.202e-03  cgtest: -3.038e-02  t[s]:     88.20
    FillingsUpdate:  mu: -0.223963613  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00556  Tot: -0.00005 ]
LCAOMinimize: Iter:  12  F: -869.8878549939241793  |grad|_K:  5.720e-05  alpha:  8.740e-02  linmin:  8.941e-05  cgtest: -3.686e-03  t[s]:     92.41
    FillingsUpdate:  mu: -0.227797706  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00536  Tot: +0.00000 ]
LCAOMinimize: Iter:  13  F: -869.8887640793544733  |grad|_K:  4.451e-05  alpha:  1.273e-01  linmin:  1.172e-03  cgtest:  2.431e-02  t[s]:     96.62
    FillingsUpdate:  mu: -0.224849089  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00541  Tot: -0.00000 ]
LCAOMinimize: Iter:  14  F: -869.8891947560522340  |grad|_K:  3.509e-05  alpha:  1.003e-01  linmin: -9.164e-04  cgtest:  2.453e-02  t[s]:    100.82
    FillingsUpdate:  mu: -0.228575295  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00696  Tot: -0.00152 ]
LCAOMinimize: Iter:  15  F: -869.8892568641696244  |grad|_K:  5.975e-05  alpha:  1.965e-01  linmin:  1.115e-01  cgtest: -7.117e-01  t[s]:    105.04
LCAOMinimize:   Bad step direction: g.d > 0.
LCAOMinimize:   Undoing step.
LCAOMinimize:   Step failed: resetting search direction.
    FillingsUpdate:  mu: -0.228554709  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00538  Tot: +0.00000 ]
LCAOMinimize: Iter:  16  F: -869.8890799418312554  |grad|_K:  5.975e-05  alpha:  0.000e+00
    FillingsUpdate:  mu: -0.225091606  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00532  Tot: +0.00000 ]
LCAOMinimize: Iter:  17  F: -869.8894193098747110  |grad|_K:  2.551e-05  alpha:  4.471e-02  linmin:  1.944e-02  cgtest: -5.782e-02  t[s]:    112.63
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.341164e-01.
    FillingsUpdate:  mu: -0.225943278  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00540  Tot: +0.00000 ]
LCAOMinimize: Iter:  18  F: -869.8896070012581276  |grad|_K:  8.616e-06  alpha:  1.354e-01  linmin: -1.526e-03  cgtest:  1.330e-01  t[s]:    117.76
    FillingsUpdate:  mu: -0.226701464  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00561  Tot: +0.00000 ]
LCAOMinimize: Iter:  19  F: -869.8896452870756093  |grad|_K:  1.419e-05  alpha:  2.349e-01  linmin: -9.275e-04  cgtest:  9.838e-04  t[s]:    121.97
    FillingsUpdate:  mu: -0.225771620  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00600  Tot: -0.00005 ]
LCAOMinimize: Iter:  20  F: -869.8896816926164774  |grad|_K:  8.449e-06  alpha:  8.226e-02  linmin:  1.264e-03  cgtest: -1.954e-02  t[s]:    126.18
    FillingsUpdate:  mu: -0.225986138  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00675  Tot: +0.00000 ]
LCAOMinimize: Iter:  21  F: -869.8897157378565907  |grad|_K:  3.837e-06  alpha:  1.991e-01  linmin: -6.708e-02  cgtest:  3.337e-01  t[s]:    130.39
LCAOMinimize:   Wrong curvature in test step, increasing alphaT to 5.972960e-01.
    FillingsUpdate:  mu: -0.226390680  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00808  Tot: -0.00147 ]
LCAOMinimize: Iter:  22  F: -869.8898925088742544  |grad|_K:  6.198e-06  alpha: -8.072e-03  linmin:  3.428e-02  cgtest: -1.380e-01  t[s]:    134.51
    FillingsUpdate:  mu: -0.226192104  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00793  Tot: -0.00147 ]
LCAOMinimize: Iter:  23  F: -869.8898965074354237  |grad|_K:  3.797e-06  alpha:  5.346e-02  linmin: -2.760e-02  cgtest:  7.863e-01  t[s]:    138.74
LCAOMinimize: Encountered beta<0, resetting CG.
    FillingsUpdate:  mu: -0.225935974  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00809  Tot: -0.00152 ]
LCAOMinimize: Iter:  24  F: -869.8899034825478793  |grad|_K:  2.914e-06  alpha:  6.609e-02  linmin: -2.541e-03  cgtest:  7.694e-03  t[s]:    143.08
LCAOMinimize:   Step increased F by 1.757290e-04, reducing alpha to 1.483984e-03.
LCAOMinimize:   Step increased F by 1.761944e-04, reducing alpha to 1.483984e-04.
LCAOMinimize:   Step increased F by 1.759319e-04, reducing alpha to 1.483984e-05.
LCAOMinimize:   Step failed to reduce F after 3 attempts. Quitting step.
LCAOMinimize:   Undoing step.
LCAOMinimize:   Step failed: resetting search direction.
    FillingsUpdate:  mu: -0.225916413  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00662  Tot: -0.00005 ]
LCAOMinimize: Iter:  25  F: -869.8897275501227568  |grad|_K:  2.934e-06  alpha:  1.484e-05
    FillingsUpdate:  mu: -0.226084098  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00653  Tot: +0.00000 ]
LCAOMinimize: Iter:  26  F: -869.8897283351014948  |grad|_K:  1.882e-06  alpha:  4.920e-02  linmin: -9.929e-02  cgtest:  2.951e-01  t[s]:    161.18
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.475895e-01.
    FillingsUpdate:  mu: -0.225956243  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00648  Tot: -0.00005 ]
LCAOMinimize: Iter:  27  F: -869.8897310793714723  |grad|_K:  2.382e-06  alpha:  2.846e-01  linmin:  1.195e-03  cgtest: -1.829e-01  t[s]:    166.31
    FillingsUpdate:  mu: -0.226143244  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00632  Tot: -0.00000 ]
LCAOMinimize: Iter:  28  F: -869.8897321070922999  |grad|_K:  2.615e-06  alpha:  1.085e-01  linmin:  5.069e-05  cgtest:  8.300e-02  t[s]:    170.52
    FillingsUpdate:  mu: -0.225990031  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00653  Tot: -0.00005 ]
LCAOMinimize: Iter:  29  F: -869.8897356697402756  |grad|_K:  1.545e-06  alpha:  2.161e-01  linmin: -4.801e-03  cgtest:  9.720e-03  t[s]:    174.72
    FillingsUpdate:  mu: -0.226094007  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00664  Tot: -0.00005 ]
LCAOMinimize: Iter:  30  F: -869.8897362546910017  |grad|_K:  1.292e-06  alpha:  9.873e-02  linmin: -3.753e-02  cgtest:  1.421e-01  t[s]:    178.93
LCAOMinimize:   Step increased F by 2.214506e-09, reducing alpha to 5.391984e-03.
LCAOMinimize:   Step increased F by 2.712251e-07, reducing alpha to 5.391984e-04.
LCAOMinimize:   Step increased F by 3.092365e-07, reducing alpha to 5.391984e-05.
LCAOMinimize:   Step failed to reduce F after 3 attempts. Quitting step.
LCAOMinimize:   Undoing step.
LCAOMinimize:   Step failed: resetting search direction.
    FillingsUpdate:  mu: -0.226091392  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00652  Tot: +0.00000 ]
LCAOMinimize: Iter:  31  F: -869.8897359449641726  |grad|_K:  1.308e-06  alpha:  5.392e-05
    FillingsUpdate:  mu: -0.226002509  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00650  Tot: -0.00000 ]
LCAOMinimize: Iter:  32  F: -869.8897361687110106  |grad|_K:  9.367e-07  alpha:  5.974e-02  linmin:  3.495e-04  cgtest: -1.150e-03  t[s]:    197.05
LCAOMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 1.792345e-01.
    FillingsUpdate:  mu: -0.226047568  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00652  Tot: -0.00005 ]
LCAOMinimize: Iter:  33  F: -869.8897368351225623  |grad|_K:  4.591e-07  alpha:  1.845e-01  linmin:  1.813e-03  cgtest: -2.910e-03  t[s]:    202.24
LCAOMinimize: Converged (|Delta F|<1.000000e-06 for 2 iters).
----- createFluidSolver() ----- (Fluid-side solver setup)
   Initializing fluid molecule 'H2O'
     Initializing site 'O'
       Electron density: proportional to exp(-r/0.36935)*erfc((r-0.51523)/0.36823) with norm 6.826
       Charge density: gaussian nuclear width 0.478731 with net site charge 0.826
       Polarizability: cuspless exponential with width 0.32 and norm 3.73
       Hard sphere radius: 2.57003 bohrs
       Positions in reference frame:
         [ +0.000000 +0.000000 +0.000000 ]
     Initializing site 'H'
       Electron density: proportional to exp(-r/0.34641)*erfc((r-0)/0.390882) with norm 0.587
       Charge density: gaussian nuclear width 0.377945 with net site charge -0.413
       Polarizability: cuspless exponential with width 0.39 and norm 3.3
       Positions in reference frame:
         [ +0.000000 -1.441945 +1.122523 ]
         [ +0.000000 +1.441945 +1.122523 ]
     Net charge: 0   dipole magnitude: 0.927204
     Initializing spherical shell mfKernel with radius 2.61727 Bohr
     deltaS corrections:
       site 'O': -7.54299
       site 'H': -6.83917
   Initializing fluid molecule 'Na+'
     Initializing site 'Na'
       Electron density: proportional to exp(-r/0.19682)*erfc((r-0.71491)/0.41314) with norm 8.1383
       Charge density: gaussian nuclear width 0.365347 with net site charge -1
       Hard sphere radius: 1.86327 bohrs
       Positions in reference frame:
         [ +0.000000 +0.000000 +0.000000 ]
     Net charge: -1   dipole magnitude: 0
     Initializing gaussian mfKernel with width: 1.55004 Bohr
     deltaS corrections:
       site 'Na': -22.3555
   Initializing fluid molecule 'F-'
     Initializing site 'F'
       Electron density: proportional to exp(-r/0.38886)*erfc((r-0)/0.438782) with norm 8
       Charge density: gaussian nuclear width 0.374796 with net site charge 1
       Hard sphere radius: 2.39995 bohrs
       Positions in reference frame:
         [ +0.000000 +0.000000 +0.000000 ]
     Net charge: 1   dipole magnitude: 0
     Initializing gaussian mfKernel with width: 1.59012 Bohr
     deltaS corrections:
       site 'F': -9.04335

Correction to mu due to finite nuclear width = -0.0105345
   Cavity determined by nc: 0.001 and sigma: 0.6
   Effective cavity tension: 9.5e-06 Eh/bohr^2 to account for cavitation and dispersion.
   Nonlinear dielectric with epsBulk = 78.4 and epsInf = 1.77 with density Nmol = 0.0049383 of dipoles pMol = 0.92466 at T = 298 K.
   Nonlinear ions with screening length = 5.74355 bohrs and Z = 1 at T = 298 K.

---- Citations for features of the code used in this run ----

   Software package:
      R. Sundararaman, K. Letchworth-Weaver, K.A. Schwarz, D. Gunceler, Y. Ozhabes and T.A. Arias, 'JDFTx: software for joint density-functional theory', SoftwareX 6, 278 (2017)

   Simplified rotationally-invariant DFT+U:
      S. L. Dudarev et al., Phys. Rev. B 57, 1505 (1998)

   gga-PBE exchange-correlation functional:
      J.P. Perdew, K. Burke and M. Ernzerhof, Phys. Rev. Lett. 77, 3865 (1996)

   Pseudopotentials:
      KF Garrity, JW Bennett, KM Rabe and D Vanderbilt, Comput. Mater. Sci. 81, 446 (2014)

   Linear/nonlinear dielectric/ionic fluid model with effective cavity tension:
      D. Gunceler, K. Letchworth-Weaver, R. Sundararaman, K.A. Schwarz and T.A. Arias, Modelling Simul. Mater. Sci. Eng. 21, 074005 (2013)

   Total energy minimization with Auxiliary Hamiltonian:
      C. Freysoldt, S. Boeck, and J. Neugebauer, Phys. Rev. B 79, 241103(R) (2009)

This list may not be complete. Please suggest additional citations or
report any other bugs at https://github.com/shankar1729/jdftx/issues

Initialization completed successfully at t[s]:    203.14

--------- Lattice Minimization ---------
Fluid solver invoked on fresh (random / LCAO) wavefunctions
Running a vacuum solve first:

-------- Initial electronic minimization -----------
    FillingsUpdate:  mu: -0.226046421  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00640  Tot: +0.00000 ]
ElecMinimize: Iter:   0  F: -869.889736522655880  |grad|_K:        nan  alpha:  1.000e-01
ElecMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 3.000000e-01.
    FillingsUpdate:  mu: -0.125692955  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00447  Tot: +0.00000 ]
    SubspaceRotationAdjust: set factor to 0.752
ElecMinimize: Iter:   1  F: -876.985559286236139  |grad|_K:        nan  alpha:  6.695e-01  linmin: -1.093e-03  t[s]:    213.70
    FillingsUpdate:  mu: -0.062180447  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00380  Tot: +0.00000 ]
    SubspaceRotationAdjust: set factor to 0.437
ElecMinimize: Iter:   2  F: -877.047990619560210  |grad|_K:        nan  alpha:  1.117e-01  linmin:  2.259e-03  t[s]:    219.06
    FillingsUpdate:  mu: -0.056109622  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00327  Tot: -0.00006 ]
    SubspaceRotationAdjust: set factor to 0.33
ElecMinimize: Iter:   3  F: -877.323132396396431  |grad|_K:        nan  alpha:  2.045e-01  linmin: -2.637e-06  t[s]:    224.42
    FillingsUpdate:  mu: -0.084352370  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00340  Tot: -0.00006 ]
    SubspaceRotationAdjust: set factor to 0.504
ElecMinimize: Iter:   4  F: -877.578744788697009  |grad|_K:        nan  alpha:  8.348e-02  linmin: -4.340e-04  t[s]:    229.76
    FillingsUpdate:  mu: -0.093728973  nElectrons: 531.000000  magneticMoment: [ Abs: 0.00760  Tot: +0.00000 ]
    SubspaceRotationAdjust: set factor to 0.586
ElecMinimize: Iter:   5  F: -877.595881591809416  |grad|_K:        nan  alpha:  6.871e-02  linmin: -1.705e-03  t[s]:    235.12
    FillingsUpdate:  mu: -0.101246459  nElectrons: 531.000000  magneticMoment: [ Abs: 0.01933  Tot: -0.00128 ]
    SubspaceRotationAdjust: set factor to 0.569
ElecMinimize: Iter:   6  F: -877.682070575327202  |grad|_K:        nan  alpha:  1.797e-01  linmin: -3.850e-04  t[s]:    240.49
    FillingsUpdate:  mu: -0.078967540  nElectrons: 531.000000  magneticMoment: [ Abs: 0.02550  Tot: -0.00129 ]
    SubspaceRotationAdjust: set factor to 0.368
ElecMinimize: Iter:   7  F: -877.703782332433320  |grad|_K:        nan  alpha:  1.558e-01  linmin:  7.988e-04  t[s]:    245.84
WARNING: 450 elements failed to converge in cusolverDn eigenvalue routine Zheevd; falling back to CPU LAPACK.
WARNING: 474 elements failed to converge in cusolverDn eigenvalue routine Zheevd; falling back to CPU LAPACK.
ElecMinimize:   Step failed with F = -nan, reducing alpha to 7.377850e-03.
    FillingsUpdate:  mu: -0.079204821  nElectrons: 531.000000  magneticMoment: [ Abs: 0.02435  Tot: +0.00000 ]
    SubspaceRotationAdjust: set factor to 0.267
ElecMinimize: Iter:   8  F: -877.704511547162269  |grad|_K:        nan  alpha:  7.378e-03  linmin: -3.424e-04  t[s]:    254.69
ElecMinimize:   Predicted alpha/alphaT>3.000000, increasing alphaT to 2.213355e-02.
ElecMinimize:   Wrong curvature in test step, increasing alphaT to 6.640065e-02.
    FillingsUpdate:  mu: -0.082101039  nElectrons: 531.000000  magneticMoment: [ Abs: 0.02559  Tot: -0.00129 ]
    SubspaceRotationAdjust: set factor to 0.234
ElecMinimize: Iter:   9  F: -877.730120652254413  |grad|_K:        nan  alpha: -6.402e-01  linmin: -1.119e-03  t[s]:    261.00
ElecMinimize:   Wrong curvature in test step, increasing alphaT to 3.000000e-01.
WARNING: 504 elements failed to converge in cusolverDn eigenvalue routine Zheevd; falling back to CPU LAPACK.
WARNING: 508 elements failed to converge in cusolverDn eigenvalue routine Zheevd; falling back to CPU LAPACK.
    FillingsUpdate:  mu: -0.082101039  nElectrons: 531.000000  magneticMoment: [ Abs: 0.02809  Tot: -0.00129 ]
    SubspaceRotationAdjust: set factor to -nan
ElecMinimize: Iter:  10  F: -nan  |grad|_K:       -nan  alpha: -2.378e-01  linmin:       -nan  t[s]:    265.92
ElecMinimize: |grad|_K=-nan. Stopping ...
Setting wave functions to eigenvectors of Hamiltonian
Vacuum energy after initial minimize, F = -nan

-------- Electronic minimization -----------
    Nonlinear fluid (bulk dielectric constant: 78.4) occupying -nan of unit cell
    Nonlinear screening (bulk screening length: 5.74355 bohrs) occupying -nan of unit cell
    Nonlinear solve completed after 0 iterations at t[s]:    266.91
WARNING: 491 elements failed to converge in cusolverDn eigenvalue routine Zheevd; falling back to CPU LAPACK.
WARNING: 513 elements failed to converge in cusolverDn eigenvalue routine Zheevd; falling back to CPU LAPACK.
    FillingsUpdate:  mu: -0.082101039  nElectrons: 531.000000  magneticMoment: [ Abs:    -nan  Tot:     -nan ]
ElecMinimize: Iter:   0  F: -nan  |grad|_K:        nan  alpha:  1.000e-01
ElecMinimize: |grad|_K=nan. Stopping ...
Setting wave functions to eigenvectors of Hamiltonian
# Lattice vectors:
R = 
[      37.2276            0            0  ]
[            0      32.3143            0  ]
[            0            0       41.574  ]
unit cell volume = 50012.9

# Strain tensor in Cartesian coordinates:
[            0            0            0  ]
[            0            0            0  ]
[            0            0            0  ]

# Stress tensor in Cartesian coordinates [Eh/a0^3]:
[         -nan         -nan         -nan  ]
[         -nan         -nan         -nan  ]
[         -nan         -nan         -nan  ]

# Ionic positions in lattice coordinates:
ion C  -0.000029707957535   0.999734075373656   0.445928226069441 1
ion C  -0.000030660203837   0.499844907911305   0.444322186596028 1
ion C   0.499970372367738   0.000280714830126   0.446154934120106 1
ion C   0.438015305987727   0.127431817301940   0.446133786332907 1
ion C   0.435820836742936   0.618613767676327   0.445833776102915 1
ion C   0.937452161300365   0.124476331537161   0.445388090202350 1
ion C   0.937724355040379   0.625130877346753   0.444740465576478 1
ion C   0.374526832562430   0.252276673076671   0.445828109043488 1
ion C   0.375273370438041   0.747794284053398   0.446149721831786 1
ion C   0.875366000999245   0.249386918048184   0.444811996267458 1
ion C   0.874994427215820   0.750167816094456   0.445332331271550 1
ion C   0.309551550942670   0.374875987351058   0.445489572828667 1
ion C   0.313150509895064   0.874337377267287   0.445905891444084 1
ion C   0.813960227052455   0.374527353147648   0.444658056041645 1
ion C   0.812099469165706   0.874970944221219   0.445642451566290 1
ion C   0.250603457195469   0.999892226291721   0.445693193186235 1
ion C   0.247611804433649   0.499671157297559   0.445225706704760 1
ion C   0.749342065784283   0.999894361541899   0.445627922441240 1
ion C   0.752330127890262   0.499678373962203   0.445054796180746 1
ion C   0.187587971865206   0.124579311674992   0.445341658164641 1
ion C   0.186686814053406   0.625051996155078   0.445139452774535 1
ion C   0.686964046400407   0.125644470510937   0.445698587716742 1
ion C   0.688814237612989   0.624416798255218   0.445556728165458 1
ion C   0.124572628291732   0.249391664033795   0.444818734697330 1
ion C   0.124947681092763   0.750165095121656   0.445511251936340 1
ion C   0.625373840867768   0.252286237468929   0.445808603648308 1
ion C   0.624646980381510   0.747788946014781   0.446041101655852 1
ion C   0.062007133961267   0.374499878653393   0.444385061057204 1
ion C   0.062578350545449   0.875008154040588   0.445923393957856 1
ion C   0.561977715777834   0.873249178502399   0.446132772248917 1
ion C   0.125225357659535   0.999749782341358   0.445755187171603 1
ion C   0.123965402731164   0.499830043804282   0.444543827421787 1
ion C   0.624206175584290   0.000200254746304   0.445946847267422 1
ion C   0.628849169560916   0.500199213027952   0.445178198972218 1
ion C   0.062489127927124   0.124477827397682   0.445390155345229 1
ion C   0.062216478130419   0.625128359913442   0.444865777737386 1
ion C   0.561910178089229   0.127431419525230   0.446133108704057 1
ion C   0.564033023167032   0.618642800770310   0.445773610784156 1
ion C  -0.000030916517966   0.249371385965329   0.444772375090896 1
ion C  -0.000028939857070   0.750187134694269   0.445459262751891 1
ion C   0.499947202057239   0.256681546043961   0.446157175193274 1
ion C   0.499960330485616   0.744856014781870   0.446378659811442 1
ion C   0.437963315791167   0.873246940231938   0.446176236049555 1
ion C   0.937933248977612   0.374500804682541   0.444335224671986 1
ion C   0.937366608000279   0.875011158256017   0.445863223270998 1
ion C   0.375737916349473   0.000198004275686   0.445976748073236 1
ion C   0.371087557659699   0.500217543714721   0.445310332270646 1
ion C   0.874719480395426   0.999749434033385   0.445712277918779 1
ion C   0.875975729556790   0.499833594597233   0.444402785504607 1
ion C   0.312966925438063   0.125643892723006   0.445716129801589 1
ion C   0.311122106676233   0.624397264046538   0.445714329093068 1
ion C   0.812349574365409   0.124576107130399   0.445335962125694 1
ion C   0.813255859005537   0.625058890093383   0.444929114922949 1
ion C   0.249215281853863   0.249673316041915   0.445208288845139 1
ion C   0.249971151390751   0.749814911392065   0.445685077785866 1
ion C   0.750718700906891   0.249664077026149   0.445207015162407 1
ion C   0.749969856130867   0.749824751720888   0.445506173635731 1
ion C   0.185984222507043   0.374536733637164   0.444741389740409 1
ion C   0.187846439478187   0.874968850569466   0.445757398758088 1
ion C   0.690402766833897   0.374855805760445   0.445398413468850 1
ion C   0.686794155101733   0.874338057164529   0.445810795947367 1
ion C   0.062578252559700   0.041187935828431   0.445753489402645 1
ion C   0.062007159373190   0.541697378677913   0.444510094254811 1
ion C   0.561977522154194   0.042945428160908   0.446092496408946 1
ion C  -0.000028720931097   0.166009496261780   0.445198040753436 1
ion C  -0.000031456785661   0.666825131090442   0.445003747493612 1
ion C   0.499959193870288   0.171338662282984   0.446258445641774 1
ion C   0.499947756190214   0.659514185073599   0.446287999234636 1
ion C   0.435824900974686   0.297594827763685   0.445663621800177 1
ion C   0.438014970687155   0.788761749068432   0.446276296560717 1
ion C   0.937724355658082   0.291066364715505   0.444594972468324 1
ion C   0.937451776549768   0.791721051919954   0.445584046857019 1
ion C   0.371095236325109   0.415975601693076   0.445242788509375 1
ion C   0.375737578134043   0.915996994320677   0.446023127067455 1
ion C   0.875976734583145   0.416363659032656   0.444371807837518 1
ion C   0.874719219172485   0.916447270404974   0.445773823922606 1
ion C   0.313150343412540   0.041857728203515   0.445784683992839 1
ion C   0.309552497304054   0.541316950477765   0.445605191982716 1
ion C   0.812099480136245   0.041224162104735   0.445540019126919 1
ion C   0.813959222220814   0.541669159899156   0.444719411187060 1
ion C   0.249971068047741   0.166380587618098   0.445381622484253 1
ion C   0.249215575450700   0.666521707004682   0.445505943479218 1
ion C   0.749970247280581   0.166369966833721   0.445380783794484 1
ion C   0.750716357808668   0.666531380815207   0.445315188971194 1
ion C   0.186687488858668   0.291144316489885   0.444843113487566 1
ion C   0.187587326407784   0.791617150275487   0.445657923188448 1
ion C   0.688816097315192   0.291777924710189   0.445472765785234 1
ion C   0.686962945359704   0.790550592012889   0.445802958260784 1
ion C   0.123965970430202   0.416366785487216   0.444470244775231 1
ion C   0.125224918439458   0.916447325238592   0.445844976282335 1
ion C   0.628845947293803   0.415996038480234   0.445158324182246 1
ion C   0.624206026326245   0.915994328111704   0.445970882753547 1
ion C   0.187846245047716   0.041226229234834   0.445577763821630 1
ion C   0.185984968797380   0.541659656351465   0.444904437613512 1
ion C   0.686794131724760   0.041856228275513   0.445740212079925 1
ion C   0.690399093862252   0.541337392340211   0.445443198898003 1
ion C   0.124947707352462   0.166031148718795   0.445162986608232 1
ion C   0.124572318884695   0.666804287607905   0.445158329410321 1
ion C   0.624646559072846   0.168404605266302   0.445942603516031 1
ion C   0.625372400142354   0.663911457622512   0.445904336010831 1
ion C   0.062216706855987   0.291068373077357   0.444612860557117 1
ion C   0.062488484457463   0.791719465959812   0.445678612007524 1
ion C   0.564030021969983   0.297561897132438   0.445688774394897 1
ion C   0.561909832600928   0.788761728913306   0.446217398341270 1
ion C  -0.000030838243679   0.416352361580263   0.444277775164100 1
ion C  -0.000030130492913   0.916463508061080   0.446005686649703 1
ion C   0.499970091625480   0.915913447038932   0.446179752741310 1
ion C   0.437962547101906   0.042947684813139   0.446098956776782 1
ion C   0.937366484108135   0.041185020384516   0.445739088473453 1
ion C   0.937932656728495   0.541696490027580   0.444410054418785 1
ion C   0.375273216462532   0.168399965303887   0.445953582197257 1
ion C   0.374525826761453   0.663922804369501   0.446025889218876 1
ion C   0.874994540275412   0.166028512978554   0.445166132015055 1
ion C   0.875365238026728   0.666809909495606   0.444959118556613 1
ion C   0.311121345734956   0.291797780477080   0.445499570110804 1
ion C   0.312966624765426   0.790551938488303   0.445933039306181 1
ion C   0.813256466578631   0.291137701674143   0.444821697482967 1
ion C   0.812349159019483   0.791620366282151   0.445482903196133 1
ion C   0.247612153297023   0.416524751675460   0.445153640713813 1
ion C   0.250603348791376   0.916303676623925   0.445773578241327 1
ion C   0.752331534809626   0.416517853532229   0.445029057716477 1
ion C   0.749341833246069   0.916300678674924   0.445678645949204 1
ion N   0.567824723640165   0.378411454019348   0.444462488104865 1
ion N   0.431928410032153   0.378301497433238   0.444325030101871 1
ion N   0.567836387517307   0.537805623992100   0.444476742065708 1
ion N   0.431924098466622   0.537911834829578   0.444499847359895 1
ion Fe   0.499803939860738   0.458211757446663   0.450834969008521 1
ion O   0.498458244590283   0.457838499544820   0.533949429396895 1
ion H   0.546214242780432   0.456126955822647   0.546043316697304 1

# Forces in Lattice coordinates:
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force C                -nan                -nan                -nan 1
force N                -nan                -nan                -nan 1
force N                -nan                -nan                -nan 1
force N                -nan                -nan                -nan 1
force N                -nan                -nan                -nan 1
force Fe                -nan                -nan                -nan 1
force O                -nan                -nan                -nan 1
force H                -nan                -nan                -nan 1

# Energy components:
   A_diel =                      -nan
   Eewald =     3705.0013618655716527
       EH =                      -nan
     Eloc =                      -nan
      Enl =                      -nan
      Exc =                      -nan
 Exc_core =        5.3135730363448159
       KE =                      -nan
        U =                      -nan
-------------------------------------
     Etot =                      -nan
       TS =        0.4577879045887624
-------------------------------------
        F =                      -nan

Dumping '441_nlpcm.ionpos' ... done
Dumping '441_nlpcm.stress' ... done
LatticeMinimize: Iter:   0  F: -nan  |grad|_K:       -nan  t[s]:    273.28
LatticeMinimize: |grad|_K=-nan. Stopping ...

#--- Lowdin population analysis ---
# oxidation-state C -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan -nan
# magnetic-moments C +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan +nan
# oxidation-state N -nan -nan -nan -nan
# magnetic-moments N +nan +nan +nan +nan
# oxidation-state Fe -nan
# magnetic-moments Fe +nan
# oxidation-state O -nan
# magnetic-moments O +nan
# oxidation-state H -nan
# magnetic-moments H +nan

Matrix not positive-definite at leading minor# 1 in CuSolver Cholesky routine Zpotrf.

Stack trace:
     0: /home/cbu/jdftx/build/libjdftx_gpu.so(_Z10printStackb+0x27) [0x15220eae83a7]
     1: /home/cbu/jdftx/build/libjdftx_gpu.so(_Z14stackTraceExiti+0xd) [0x15220eae8acd]
     2: /home/cbu/jdftx/build/libjdftx_gpu.so(_Z8choleskyRK6matrixb+0x372) [0x15220eaf6d72]
     3: /home/cbu/jdftx/build/libjdftx_gpu.so(_Z11orthoMatrixRK6matrix+0xdc) [0x15220eaf749c]
     4: /home/cbu/jdftx/build/libjdftx_gpu.so(_ZN8ElecVars14orthonormalizeEiP6matrix+0x141) [0x15220ec943c1]
     5: /home/cbu/jdftx/build/libjdftx_gpu.so(_ZN16LatticeMinimizer4stepERK15LatticeGradientd+0xcb6) [0x15220ed8f1f6]
     6: /home/cbu/jdftx/build/libjdftx_gpu.so(_ZN16LatticeMinimizer8minimizeERK14MinimizeParams+0x64) [0x15220ed90464]
     7: /home/cbu/jdftx/build/jdftx_gpu() [0x40868e]
     8: /lib64/libc.so.6(__libc_start_main+0xef) [0x1521fe43e24d]
     9: /home/cbu/jdftx/build/jdftx_gpu() [0x407ffa]
Writing 'jdftx-stacktrace' (for use with script printStackTrace): done.
shankar1729 commented 5 months ago

It does appear that the NAN only appears after the first fluid step, so yes, must be some MPI routine that the fluid is calling. Please confirm this by:

  1. Checking a vacuum GPU job split across nodes
  2. Check LinearPCM and NonlinearPCM to see if it is fluid specific
  3. See if recompiling with CudaAwareMPI=no makes the issue go away

That will narrow down the source of the issue hopefully. I have verified that jdftx works correctly for multi-GPU-node NonlinearPCM jobs, including with nvhpc and cuda-aware cray-mpi on Perlmutter.

ColinBundschu commented 1 month ago

I am still actively looking into this, and I will close the issue when I solve it. It's on my TODO list.

ColinBundschu commented 4 weeks ago

For whatever reason this issue no longer reproduces for me. Closing since I guess it is fixed?