search

geoschem / geos-chem

GEOS-Chem "Science Codebase" repository. Contains GEOS-Chem science routines, run directory generation scripts, and interface code. This repository is used as a submodule within the GCClassic and GCHP wrappers, as well as in other modeling contexts (external ESMs).
http://geos-chem.org
Other
167 stars 163 forks source link

Section Type Conflict When Defining Lots of ProdLoss Groups #2400

Open AlfredMayhew opened 3 months ago

AlfredMayhew commented 3 months ago

Your name

Alfred Mayhew

Your affiliation

University of Utah

What happened? What did you expect to happen?

My current project involves adding lots (thousands) of additional reactions to the fullchem mechanism. I have done this using the 'custom' mechanism directory in kpp and everything works fine. However, I also would like to track the production of the new species I have added. Given the issues discussed in #1991, I think the best option is to add dummy species and use the ProdLoss diagnostic to track the production of the dummy species.

I did this, creating dummy species called GASDUM1, GASDUM2, ... etc. as well as AERDUM1, AERDUM2, ... etc. for each of my added reactions. Then, in custom.kpp I defined production diagnostics for each of these dummy species, like: PGASDUM1 : GASDUM1;, PGASDUM2 : GASDUM2;, ... etc. Given the large number of added reactions, this equated to 1383 families (including the 7 default families).

I have changed MAX_FAMILIES in the kpp src/gdata.h file and recompiled KPP, so I don't get the too many families message when building my GEOS-Chem mechanism. However, when it comes to building the model in my run directory, at the make -j command, I get the following error:

[ 48%] Built target ObsPack
[ 48%] Building Fortran object src/GEOS-Chem/KPP/custom/CMakeFiles/KPP.dir/gckpp_Global.F90.o
[ 51%] Built target History
/uufs/chpc.utah.edu/common/home/u6052471/haskins-group1/users/amayhew/GEOS-Chem_Models/runs/Dimer_runs/temp/CodeDir/src/GEOS-Chem/KPP/custom/gckpp_Global.F90:100:32:

  100 |   INTEGER :: JVS_MAP(LU_NONZERO)
      |                                ^
Error: 'jvs_map' causes a section type conflict with 'var'
/uufs/chpc.utah.edu/common/home/u6052471/haskins-group1/users/amayhew/GEOS-Chem_Models/runs/Dimer_runs/temp/CodeDir/src/GEOS-Chem/KPP/custom/gckpp_Global.F90:50:34:

   50 |   REAL(kind=dp), POINTER :: VAR(:)
      |                                  ^
note: 'var' was declared here
make[2]: *** [src/GEOS-Chem/KPP/custom/CMakeFiles/KPP.dir/build.make:140: src/GEOS-Chem/KPP/custom/CMakeFiles/KPP.dir/gckpp_Global.F90.o] Error 1
make[1]: *** [CMakeFiles/Makefile2:991: src/GEOS-Chem/KPP/custom/CMakeFiles/KPP.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
[ 59%] Built target HCOX
make: *** [Makefile:136: all] Error 2

I've tried looking into this but I have no idea what is going on! Below is a list of things I've tried:

So it seems like it is something to do with the very large number of ProdLoss families defined. I'd really appreciate it if anyone had any ideas on how I can resolve this issue or investigate it further.

What are the steps to reproduce the bug?

Please attach any relevant configuration and log files.

No response

What GEOS-Chem version were you using?

14.3.1

What environment were you running GEOS-Chem on?

Local cluster

What compiler and version were you using?

gcc11.2.0

Will you be addressing this bug yourself?

Yes, but I will need some help

In what configuration were you running GEOS-Chem?

GCClassic

What simulation were you running?

Full chemistry

As what resolution were you running GEOS-Chem?

2x2.5

What meterology fields did you use?

MERRA-2

Additional information

No response

yantosca commented 3 months ago

Thanks for writing @AlfredMayhew. I've never seen this error before. I found some posts on Stack Overflow that describe this:

This could be triggered by something that is getting inlined (maybe the commonIncludeVars.H file). But I don't have a great idea of how to work around it. You could try reducing then number of dummy species (maybe by half) and then see if that compiles. Then repeat until you find the number of dummy species that doesn't cause this error.

Tagging @RolfSander @jimmielin

RolfSander commented 3 months ago

IIUC, this is a fortran error, not a C error. The gcc compiler is only involved because gfortran uses gcc as a backend (AFAIK). Could you try a different fortran compiler and check what happens then?

AlfredMayhew commented 3 months ago

Thanks both. I'll try these suggestions and get back with the results.

AlfredMayhew commented 2 months ago

Hi, @RolfSander, I have tried switching to intel compilers and I got the same error, so that didn't seem to help.

@yantosca, I tried looking for the boundary of number of families at which the error occurred, and strangely it seems to vary on different attempts. The first time I tried I found that 638 families was the maximum number of families (including the 7 default families) before an error occurred, however I tried again the next day and found the threshold to be a total of 642 families. I found the limit to be 642 families again this morning, but then this afternoon it had increased to 644.

I have also tried reducing the length of the names of the dummy species since I thought maybe it was something to do with an exceedance of character limits in a Fortran file. Originally they were named GASDUM1, GASDUM2, ..., GASDUM1345, I renamed them to G1, G2, ..., G1345. This also had a threshold of 644 families, so the length of the species/family names seemed to have no impact.

Given the variability in this threshold I'm thinking that the issue could be related to the local cluster configuration and resources available to me. However, the only variables I can think to change are the memory allocated and the number of cpus. I have tested both of these by creating a model with one family more than is required to raise the error and then trying to build the model with 100GB of memory and 10 cpus allocated to the job. I tried this with additional memory/ cpus during both the kpp build_mechanism.sh stage and the model compilation stage. This didn't make a difference and I got the same error.

If you have any suggestions for other variables I could try adjusting, or any alternative explanation for this maximum number of families varying around 640 then that would be very helpful.

yantosca commented 2 months ago

Thanks for the update @AlfredMayhew. Can you post your code directory as a tarball here? I can try to see if I can replicate the build error on my system.

AlfredMayhew commented 2 months ago

Of course, I've attached the src directory from my GEOS-Chem version. I've made some changes to some of the other GEOS-Chem code to incorperate my dimer species into the aerosol module. I think you should be OK to just copy the mechanism and kpp file from src/GEOS-Chem/src/custom/ since the changes to the other code files should only matter when running the model, but I figured I'd include the whole set of files just in case. This also includes the kpp file with all 1376 additional families defined, my last successful run had all families after PGASDUM637 removed (hence 644 total families, 637 + the 7 default families).

I've also attached some files from the run directory in case they're needed: geos_chem.yml with the dimer species added as transported species, species_database.yml with the dimer species and dummy species included, a modified FAST_JX directory with additional photolysis definitions used in my modified mechanism. Again, I think these should only be needed at runtime, but I've included them just in case. I've added .txt to the .yml file names so that GitHub will let me attach the files.

Hopefully that's everything you need, let me know if I've missed anything or made any mistakes. Thanks for your help.

src.zip FAST_JX_COPY.zip geoschem_config.yml.txt species_database.yml.txt

yantosca commented 2 months ago

Hi @AlfredMayhew. Here's what I did:

  1. Started an Ubuntu instance under Windows Subsystem for Linux.
  2. Downloaded and uncompressed your src.zip file in the Ubuntu instance.
  3. In KPP, set MAX_FAMILIES=5000in src/gdata.h and recompiled. This was done in a branch off of main (KPP 3.1.1).
  4. $ cd src/GEOS-Chem/KPP
  5. Started a Conda environment with Python (for the OH reactivity diagnostic)
  6. $ ./build_mechanism.sh custom

This resulted in:

This is KPP-3.1.1.

KPP is parsing the equation file.
KPP is computing Jacobian sparsity structure.
KPP is starting the code generation.
KPP is initializing the code generation.
KPP is generating the monitor data:
    - gckpp_Monitor
KPP is generating the utility data:
    - gckpp_Util
KPP is generating the global declarations:
    - gckpp_Main
KPP is generating the ODE function:
    - gckpp_Function
KPP is generating the ODE Jacobian:
    - gckpp_Jacobian
    - gckpp_JacobianSP
KPP is generating the linear algebra routines:
    - gckpp_LinearAlgebra
KPP is generating the utility functions:
    - gckpp_Util
KPP is generating the rate laws:
    - gckpp_Rates
KPP is generating the parameters:
    - gckpp_Parameters
KPP is generating the global data:
    - gckpp_Global
KPP is generating the driver from none.F90:
    - gckpp_Main
KPP is starting the code post-processing.

KPP has successfully created the model "gckpp".

Reactivity consists of 270 reactions
Written to gckpp_Util.F90

I didn't compile the model with it, I just wanted to see if I could build the mechanism files.

The limits in my instance were:

$ ulimit -a
real-time non-blocking time  (microseconds, -R) unlimited
core file size              (blocks, -c) unlimited
data seg size               (kbytes, -d) unlimited
scheduling priority                 (-e) 0
file size                   (blocks, -f) unlimited
pending signals                     (-i) 31127
max locked memory           (kbytes, -l) 65536
max memory size             (kbytes, -m) unlimited
open files                          (-n) 1024
pipe size                (512 bytes, -p) 8
POSIX message queues         (bytes, -q) 819200
real-time priority                  (-r) 0
stack size                  (kbytes, -s) unlimited
cpu time                   (seconds, -t) unlimited
max user processes                  (-u) 31127
virtual memory              (kbytes, -v) unlimited
file locks                          (-x) unlimited

My environment has these settings:

SHELL=/bin/bash
LIBGL_ALWAYS_INDIRECT=1
GREP_COLOR=32
WSL2_GUI_APPS_ENABLED=1
CONDA_EXE=/home/bob/mambaforge/bin/conda
_CE_M=
WSL_DISTRO_NAME=Ubuntu
SSH_AUTH_SOCK=/tmp/ssh-XXXXXXMsWBDE/agent.404
SSH_AGENT_PID=408
XML_CATALOG_FILES=file:///home/bob/mambaforge/envs/gcpy_env/etc/xml/catalog file:///etc/xml/catalog
NAME=EAS-RYANTOSCA2A
PWD=/home/bob/test/src/GEOS-Chem/KPP
GSETTINGS_SCHEMA_DIR=/home/bob/mambaforge/envs/gcpy_env/share/glib-2.0/schemas
LOGNAME=bob
CONDA_PREFIX=/home/bob/mambaforge/envs/gcpy_env
GSETTINGS_SCHEMA_DIR_CONDA_BACKUP=
CXX=g++
HOME=/home/bob
LANG=en_US.UTF-8
WSL_INTEROP=/run/WSL/2002_interop
LS_COLORS=no=00:fi=00:di=01;33:ln=01;36:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:su=37;41:sg=30;43:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;37:*.tgz=01;37:*.arj=01;37:*.taz=01;37:*.lzh=01;37:*.zip=01;37:*.z=01;37:*.Z=01;37:*.gz=01;37:*.bz2=01;37:*.deb=01;37:*.rpm=01;37:*.jar=01;37:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.avi=01;35:*.fli=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.flac=01;35:*.mp3=01;35:*.mpc=01;35:*.ogg=01;35:*.wav=01;35:
WAYLAND_DISPLAY=wayland-0
CONDA_PROMPT_MODIFIER=(gcpy_env) 
PYLINTRC=~/.pylint.rc
KPP_HOME=/home/bob/repos/KPP3
PYTHONPATH=:/home/bob/repos/gcpy
TERM=xterm-256color
_CE_CONDA=
F77=gfortran
USER=bob
CONDA_SHLVL=1
DISPLAY=:0
SHLVL=1
PROJ_DATA=/home/bob/mambaforge/envs/gcpy_env/share/proj
MPLBACKEND=tkagg
CONDA_PYTHON_EXE=/home/bob/mambaforge/bin/python
XDG_RUNTIME_DIR=/tmp/runtime-bob
CONDA_DEFAULT_ENV=gcpy_env
WSLENV=
FC=gfortran
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
PATH=/home/bob/mambaforge/envs/gcpy_env/bin:/home/bob/bin:/home/bob/.local/bin:/bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:.:/home/bob/mambaforge/condabin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files (x86)/Common Files/Oracle/Java/java8path:/mnt/c/Program Files (x86)/Common Files/Oracle/Java/javapath:/mnt/c/windows/system32:/mnt/c/windows:/mnt/c/windows/System32/Wbem:/mnt/c/windows/System32/WindowsPowerShell/v1.0:/mnt/c/windows/System32/OpenSSH:/mnt/c/Users/rmy586/AppData/Local/Microsoft/WindowsApps:/snap/bin:/home/bob/repos/KPP3/bin:/home/bob/repos/bashdatacatalog/bin
CC=gcc
PYTHONWARNINGS=ignore::DeprecationWarning
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1000/bus
HOSTTYPE=x86_64
PULSE_SERVER=unix:/mnt/wslg/PulseServer
PROJ_NETWORK=ON
OLDPWD=/home/bob/test/src/GEOS-Chem
_=/bin/printenv
AlfredMayhew commented 2 months ago

That matches the output I get. The error then comes when I create a run directory and try to build the model with make -j.

When I type ulimit -a I get the following output. It looks like there's no huge differences apart from the core file size? I'm not quite sure what it means that this is set to 0. I changed it to unlimited with ulimit -c unlimited and I still get the same error.

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 1541179
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) 52428800
open files                      (-n) 16384
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) unlimited
cpu time               (seconds, -t) unlimited
max user processes              (-u) 65535
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I've also included my environment variables (from the env command) below:

HYDRA_LAUNCHER_EXTRA_ARGS=--external-launcher
SLURM_MPI_TYPE=none
CONDA_SHLVL=0
LS_COLORS=rs=0:di=38;5;33:ln=38;5;51:mh=00:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=01;05;37;41:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;40:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.zst=38;5;9:*.tzst=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.wim=38;5;9:*.swm=38;5;9:*.dwm=38;5;9:*.esd=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.mjpg=38;5;13:*.mjpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.m4a=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.oga=38;5;45:*.opus=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
LD_LIBRARY_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/lib64:/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/lib:/uufs/notchpeak.peaks/sys/installdir/slurm/std/lib
CONDA_EXE=/uufs/chpc.utah.edu/common/home/u6052471/software/miniconda3/bin/conda
SRUN_DEBUG=3
SLURM_STEP_ID=4294967290
SLURM_NODEID=0
SLURM_TASK_PID=398609
__LMOD_REF_COUNT_PATH=/uufs/chpc.utah.edu/common/home/u6052471/software/KPP/bin:1;/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/bin:1;/uufs/chpc.utah.edu/common/home/u6052471/software/miniconda3/bin:1;/uufs/chpc.utah.edu/sys/bin:1;/usr/local/bin:1;/usr/bin:1;/usr/local/sbin:1;/usr/sbin:1;/uufs/chpc.utah.edu/common/home/u6052471/.dotnet/tools:1;/uufs/notchpeak.peaks/sys/installdir/slurm/std/bin:3;/usr/lib/fastx/3/apps:1;/uufs/chpc.utah.edu/common/home/u6052471/bin:1;/uufs/chpc.utah.edu/common/home/u6052471/software/atchem-lib/numdiff/bin:1;/uufs/chpc.utah.edu/common/home/u6052471/.gem/bin:1;/opt/rocm/bin:1;/opt/rocm/profiler/bin:1;/opt/rocm/opencl/bin:1
_ModuleTable002_=LAp9LApnY2MgPSB7CmZuID0gIi91dWZzL2NocGMudXRhaC5lZHUvc3lzL21vZHVsZWZpbGVzL3NwYWNrL2xpbnV4LXJvY2t5OC14ODZfNjQvQ29yZS9saW51eC1yb2NreTgtbmVoYWxlbS9nY2MvMTEuMi4wLmx1YSIsCmZ1bGxOYW1lID0gImdjYy8xMS4yLjAiLApsb2FkT3JkZXIgPSAzLApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImdjYy8xMS4yLjAiLAp3ViA9ICIwMDAwMDAwMTEuMDAwMDAwMDAyLip6ZmluYWwiLAp9LAprcHAgPSB7CmZuID0gIi91dWZzL2NocGMudXRhaC5lZHUvY29tbW9uL2hvbWUvdTYwNTI0NzEvTXlNb2R1bGVzL2twcC8zLjAuMi5sdWEiLApmdWxsTmFtZSA9ICJrcHAvMy4wLjIiLApsb2Fk
INCLUDE=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/include
SLURM_PRIO_PROCESS=0
__LMOD_REF_COUNT_C_INCLUDE_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/include:1
I_MPI_HYDRA_BOOTSTRAP_EXEC_EXTRA_ARGS=--external-launcher
LANG=en_US.UTF-8
SLURM_SUBMIT_DIR=/uufs/chpc.utah.edu/common/home/u6052471
HISTCONTROL=ignoredups
MODULERCFILE=/uufs/chpc.utah.edu/sys/modulefiles/etc/rc_r8
OMPI_MCA_plm_slurm_args=--external-launcher
DISPLAY=notchpeak1:278
GUESTFISH_RESTORE=\e[0m
LMOD_FAMILY_COMPILER_VERSION=11.2.0
HOSTNAME=notchpeak1
LMOD_SYSTEM_DEFAULT_MODULES=chpc
__LMOD_REF_COUNT__LMFILES_=/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8/Core/chpc/1.0.lua:1;/uufs/chpc.utah.edu/common/home/u6052471/MyModules/miniconda3/latest.lua:1;/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Core/linux-rocky8-nehalem/gcc/11.2.0.lua:1;/uufs/chpc.utah.edu/common/home/u6052471/MyModules/kpp/3.0.2.lua:1
C_INCLUDE_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/include
SLURM_STEPID=4294967290
SLURM_SRUN_COMM_HOST=10.242.68.1
__LMOD_SET_FPATH=1
COLORTERM=truecolor
GUESTFISH_INIT=\e[1;34m
GEM_HOME=/uufs/chpc.utah.edu/common/home/u6052471/.gem
FPATH=/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8/init/ksh_funcs
_ModuleTable007_=dWxlZmlsZXMvQ29yZSIsCn0K
__LMOD_REF_COUNT_CPLUS_INCLUDE_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/include:1
KDEDIRS=/usr
SLURM_PROCID=0
SLURM_JOB_GID=4001067
__LMOD_REF_COUNT_INCLUDE=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/include:1
BASHRC_LOADED=0
__LMOD_REF_COUNT_LD_LIBRARY_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/lib64:1;/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/lib:1;/uufs/notchpeak.peaks/sys/installdir/slurm/std/lib:3
SLURMD_NODENAME=notch308
SLURM_JOB_END_TIME=1723078678
_ModuleTable004_=PSB7CiIvdXVmcy9jaHBjLnV0YWguZWR1L3N5cy9tb2R1bGVmaWxlcy9zcGFjay9saW51eC1yb2NreTgteDg2XzY0L0NvbXBpbGVyL2xpbnV4LXJvY2t5OC1za3lsYWtlX2F2eDUxMi9nY2MvMTEuMi4wIiwgIi91dWZzL2NocGMudXRhaC5lZHUvc3lzL21vZHVsZWZpbGVzL0NIUEMtcjgvQ29tcGlsZXIvZ2NjLzExLjIuMCIKLCAiL3V1ZnMvY2hwYy51dGFoLmVkdS9zeXMvbW9kdWxlZmlsZXMvc3BhY2svbGludXgtcm9ja3k4LXg4Nl82NC9Db21waWxlci9saW51eC1yb2NreTgteDg2XzY0L2djYy8xMS4yLjAiCiwgIi91dWZzL2NocGMudXRhaC5lZHUvc3lzL21vZHVsZWZpbGVzL3NwYWNrL2xpbnV4LXJvY2t5OC14ODZfNjQvQ29tcGlsZXIvbGludXgtcm9ja3k4LW5laGFsZW0v
SSH_AUTH_SOCK=/tmp/FastX-fcf5e7f4dc6f4156984c8daa28057c83/agent.1142688
SLURM_TASKS_PER_NODE=1
DOTNET_ROOT=/usr/lib64/dotnet
S_COLORS=auto
PRTE_MCA_plm_slurm_args=--external-launcher
_CE_M=
ESMFMKFILE=/uufs/chpc.utah.edu/common/home/u6052471/software/miniconda3/lib/esmf.mk
which_declare=declare -f
CC=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/bin/gcc
XDG_SESSION_ID=144775
DOTNET_BUNDLE_EXTRACT_BASE_DIR=/uufs/chpc.utah.edu/common/home/u6052471/.cache/dotnet_bundle_extract
USER=u6052471
SLURM_NNODES=1
SLURM_LAUNCH_NODE_IPADDR=10.242.68.1
GUESTFISH_PS1=\[\e[1;32m\]><fs>\[\e[0;31m\] 
LMOD_FAMILY_PYTHON_VERSION=latest
SLURM_STEP_TASKS_PER_NODE=1
SLURM_JOB_START_TIME=1723049878
__LMOD_REF_COUNT_MODULEPATH=/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Compiler/linux-rocky8-skylake_avx512/gcc/11.2.0:1;/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8/Compiler/gcc/11.2.0:1;/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Compiler/linux-rocky8-x86_64/gcc/11.2.0:1;/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Compiler/linux-rocky8-nehalem/gcc/11.2.0:1;/uufs/chpc.utah.edu/common/home/u6052471/MyModules:1;/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Core/linux-rocky8-nehalem:3;/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Core/linux-rocky8-x86_64:3;/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8/Linux:1;/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8/Core:1;/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8/modulefiles/Core:1
__LMOD_REF_COUNT_LOADEDMODULES=chpc/1.0:1;miniconda3/latest:1;gcc/11.2.0:1;kpp/3.0.2:1
PWD=/uufs/chpc.utah.edu/common/home/u6052471
SSH_ASKPASS=/usr/libexec/openssh/gnome-ssh-askpass
GCC_ROOT=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf
SLURM_JOB_NODELIST=notch308
HOME=/uufs/chpc.utah.edu/common/home/u6052471
SLURM_CLUSTER_NAME=notchpeak
CONDA_PYTHON_EXE=/uufs/chpc.utah.edu/common/home/u6052471/software/miniconda3/bin/python
CMAKE_PREFIX_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf
SLURM_NODELIST=notch308
SSH_CLIENT=10.128.61.78 51371 22
LMOD_VERSION=8.6
SLURM_NTASKS=1
F77=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/bin/gfortran
KRB5CCNAME=KEYRING:persistent:1666494
SLURM_JOB_CPUS_PER_NODE=1
BASH_ENV=/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8/init/bash
XDG_DATA_DIRS=/uufs/chpc.utah.edu/common/home/u6052471/.local/share/flatpak/exports/share:/var/lib/flatpak/exports/share:/usr/local/share:/usr/share
SLURM_TOPOLOGY_ADDR=notch308
FASTX_SESSION_DIR=/uufs/chpc.utah.edu/common/home/u6052471/.fastx_server/notchpeak1/sessions/FX3-fcf5e7f4dc6f4156984c8daa28057c83
OSVER=8.8
_CE_CONDA=
SLURMD_DEBUG=2
FASTX_SESSION_ID=fcf5e7f4dc6f4156984c8daa28057c83
PYTHONPREFIX=/uufs/chpc.utah.edu/common/home/u6052471/software/miniconda3
__LMOD_REF_COUNT_LIBRARY_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/lib64:1;/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/lib:1
SLURM_STEP_NODELIST=notch308
SLURM_JOB_NAME=8hr_Interactive_Job
SLURM_SRUN_COMM_PORT=36239
TMPDIR=/scratch/local/u6052471/1454160
LIBRARY_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/lib64:/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/lib
__LMOD_REF_COUNT_CMAKE_PREFIX_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf:1
LMOD_sys=Linux
SLURM_JOBID=1454160
_ModuleTable001_=X01vZHVsZVRhYmxlXyA9IHsKTVR2ZXJzaW9uID0gMywKY19yZWJ1aWxkVGltZSA9IGZhbHNlLApjX3Nob3J0VGltZSA9IGZhbHNlLApkZXB0aFQgPSB7fSwKZmFtaWx5ID0gewpjb21waWxlciA9ICJnY2MiLApweXRob24gPSAibWluaWNvbmRhMyIsCn0sCm1UID0gewpjaHBjID0gewpmbiA9ICIvdXVmcy9jaHBjLnV0YWguZWR1L3N5cy9tb2R1bGVmaWxlcy9DSFBDLXI4L0NvcmUvY2hwYy8xLjAubHVhIiwKZnVsbE5hbWUgPSAiY2hwYy8xLjAiLApsb2FkT3JkZXIgPSAxLApwcm9wVCA9IHsKbG1vZCA9IHsKc3RpY2t5ID0gMSwKfSwKfSwKc3RhY2tEZXB0aCA9IDAsCnN0YXR1cyA9ICJhY3RpdmUiLAp1c2VyTmFtZSA9ICJjaHBjIiwKd1YgPSAiMDAwMDAwMDAxLip6ZmluYWwi
SLURM_CONF=/uufs/notchpeak.peaks/sys/var/slurm/etc/slurm.conf
LOADEDMODULES=chpc/1.0:miniconda3/latest:gcc/11.2.0:kpp/3.0.2
FC=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/bin/gfortran
CPUARCH=cascadelake
_ModuleTable006_=YWxsZGlyL2xtb2QvOC42LXI4L21vZHVsZWZpbGVzL0NvcmUiLAp9LApzeXN0ZW1CYXNlTVBBVEggPSAiL3V1ZnMvY2hwYy51dGFoLmVkdS9zeXMvbW9kdWxlZmlsZXMvc3BhY2svbGludXgtcm9ja3k4LXg4Nl82NC9Db3JlL2xpbnV4LXJvY2t5OC1uZWhhbGVtOi91dWZzL2NocGMudXRhaC5lZHUvc3lzL21vZHVsZWZpbGVzL3NwYWNrL2xpbnV4LXJvY2t5OC14ODZfNjQvQ29yZS9saW51eC1yb2NreTgteDg2XzY0Oi91dWZzL2NocGMudXRhaC5lZHUvc3lzL21vZHVsZWZpbGVzL0NIUEMtcjgvTGludXg6L3V1ZnMvY2hwYy51dGFoLmVkdS9zeXMvbW9kdWxlZmlsZXMvQ0hQQy1yOC9Db3JlOi91dWZzL2NocGMudXRhaC5lZHUvc3lzL2luc3RhbGxkaXIvbG1vZC84LjYtcjgvbW9k
__LMOD_REF_COUNT_MANPATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/share/man:1;/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8/share/man:1;/uufs/chpc.utah.edu/sys/man:1;/usr/local/share/man:1;/usr/share/man:1;/uufs/notchpeak.peaks/sys/installdir/slurm/std/share/man:4
_ModuleTable003_=T3JkZXIgPSA0LApwcm9wVCA9IHt9LApzdGFja0RlcHRoID0gMCwKc3RhdHVzID0gImFjdGl2ZSIsCnVzZXJOYW1lID0gImtwcCIsCndWID0gIjAwMDAwMDAwMy4wMDAwMDAwMDAuMDAwMDAwMDAyLip6ZmluYWwiLAp9LAptaW5pY29uZGEzID0gewpmbiA9ICIvdXVmcy9jaHBjLnV0YWguZWR1L2NvbW1vbi9ob21lL3U2MDUyNDcxL015TW9kdWxlcy9taW5pY29uZGEzL2xhdGVzdC5sdWEiLApmdWxsTmFtZSA9ICJtaW5pY29uZGEzL2xhdGVzdCIsCmxvYWRPcmRlciA9IDIsCnByb3BUID0ge30sCnN0YWNrRGVwdGggPSAwLApzdGF0dXMgPSAiYWN0aXZlIiwKdXNlck5hbWUgPSAibWluaWNvbmRhMy9sYXRlc3QiLAp3ViA9ICIqbGF0ZXN0Lip6ZmluYWwiLAp9LAp9LAptcGF0aEEg
LMOD_ROOT=/uufs/chpc.utah.edu/sys/installdir/lmod
SLURM_JOB_QOS=notchpeak-shared-short
SLURM_TOPOLOGY_ADDR_PATTERN=node
UUFSCELL=notchpeak.peaks
MAIL=/var/spool/mail/u6052471
SLURM_CPUS_ON_NODE=1
CXX=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/bin/g++
SLURM_JOB_NUM_NODES=1
__Init_Default_Modules=1
SLURM_MEM_PER_NODE=51200
TERM=xterm-256color
VTE_VERSION=5204
SHELL=/bin/bash
SLURM_JOB_UID=1666494
_ModuleTable_Sz_=7
SLURM_JOB_PARTITION=notchpeak-shared-short
LMOD_FAMILY_COMPILER=gcc
SLURM_SCRIPT_CONTEXT=prolog_task
OSREL=Rocky
CUDA_DEVICE_ORDER=PCI_BUS_ID
SLURM_PTY_WIN_ROW=50
TMOUT=0
KPP_HOME=/uufs/chpc.utah.edu/common/home/u6052471/software/KPP/
SLURM_JOB_USER=u6052471
SLURM_PTY_WIN_COL=190
SLURM_NPROCS=1
SHLVL=4
SLURM_SUBMIT_HOST=notchpeak1
PYTHONPATH=:/uufs/chpc.utah.edu/common/home/u6052471/haskins-group1/GroupSoftware/GCPy/:/uufs/chpc.utah.edu/common/home/u6052471/haskins-group1/users/amayhew/autoAtChem2/:/uufs/chpc.utah.edu/common/home/u6052471/haskins-group1/users/amayhew/GECKO_Tools/:/uufs/chpc.utah.edu/common/home/u6052471/haskins-group1/users/amayhew/pyTUV/:/uufs/chpc.utah.edu/common/home/u6052471/haskins-group1/users/amayhew/AtChemTools/
SLURM_JOB_ACCOUNT=notchpeak-shared-short
LMOD_SHORTTIME=86400
MANPATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/share/man:/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8/share/man:/uufs/chpc.utah.edu/sys/man:/usr/local/share/man:/usr/share/man:/uufs/notchpeak.peaks/sys/installdir/slurm/std/share/man
WINDOWID=4194307
SLURM_STEP_LAUNCHER_PORT=36239
MODULEPATH=/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Compiler/linux-rocky8-skylake_avx512/gcc/11.2.0:/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8/Compiler/gcc/11.2.0:/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Compiler/linux-rocky8-x86_64/gcc/11.2.0:/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Compiler/linux-rocky8-nehalem/gcc/11.2.0:/uufs/chpc.utah.edu/common/home/u6052471/MyModules:/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Core/linux-rocky8-nehalem:/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Core/linux-rocky8-x86_64:/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8/Linux:/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8/Core:/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8/modulefiles/Core
SLURM_PTY_PORT=33977
SLURM_GTIDS=0
LOGNAME=u6052471
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1666494/bus
XDG_RUNTIME_DIR=/run/user/1666494
CPLUS_INCLUDE_PATH=/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/include
MODULEPATH_ROOT=/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8
PATH=/uufs/chpc.utah.edu/common/home/u6052471/software/KPP/bin:/uufs/chpc.utah.edu/sys/spack/linux-rocky8-nehalem/gcc-8.5.0/gcc-11.2.0-wc2i7djjsrbmf7or7prpvcjf4xlcijdf/bin:/uufs/chpc.utah.edu/common/home/u6052471/software/miniconda3/bin:/uufs/chpc.utah.edu/sys/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/uufs/chpc.utah.edu/common/home/u6052471/.dotnet/tools:/uufs/notchpeak.peaks/sys/installdir/slurm/std/bin:/usr/lib/fastx/3/apps:/uufs/chpc.utah.edu/common/home/u6052471/bin:/uufs/chpc.utah.edu/common/home/u6052471/software/atchem-lib/numdiff/bin:/uufs/chpc.utah.edu/common/home/u6052471/.gem/bin:/opt/rocm/bin:/opt/rocm/profiler/bin:/opt/rocm/opencl/bin
SLURM_JOB_ID=1454160
_LMFILES_=/uufs/chpc.utah.edu/sys/modulefiles/CHPC-r8/Core/chpc/1.0.lua:/uufs/chpc.utah.edu/common/home/u6052471/MyModules/miniconda3/latest.lua:/uufs/chpc.utah.edu/sys/modulefiles/spack/linux-rocky8-x86_64/Core/linux-rocky8-nehalem/gcc/11.2.0.lua:/uufs/chpc.utah.edu/common/home/u6052471/MyModules/kpp/3.0.2.lua
GUESTFISH_OUTPUT=\e[0m
DEBUGINFOD_URLS=https://debuginfod.centos.org/ 
SLURM_STEP_NUM_TASKS=1
MODULESHOME=/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8
LMOD_SETTARG_FULL_SUPPORT=no
HISTSIZE=1000
LMOD_PKG=/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8
KPP_FLEX_LIB_DIR=/usr/lib64
SLURM_STEP_NUM_NODES=1
VGL_DISPLAY=egl0
_ModuleTable005_=Z2NjLzExLjIuMCIsICIvdXVmcy9jaHBjLnV0YWguZWR1L2NvbW1vbi9ob21lL3U2MDUyNDcxL015TW9kdWxlcyIKLCAiL3V1ZnMvY2hwYy51dGFoLmVkdS9zeXMvbW9kdWxlZmlsZXMvc3BhY2svbGludXgtcm9ja3k4LXg4Nl82NC9Db3JlL2xpbnV4LXJvY2t5OC1uZWhhbGVtIiwgIi91dWZzL2NocGMudXRhaC5lZHUvc3lzL21vZHVsZWZpbGVzL3NwYWNrL2xpbnV4LXJvY2t5OC14ODZfNjQvQ29yZS9saW51eC1yb2NreTgteDg2XzY0IgosICIvdXVmcy9jaHBjLnV0YWguZWR1L3N5cy9tb2R1bGVmaWxlcy9DSFBDLXI4L0xpbnV4IiwgIi91dWZzL2NocGMudXRhaC5lZHUvc3lzL21vZHVsZWZpbGVzL0NIUEMtcjgvQ29yZSIsICIvdXVmcy9jaHBjLnV0YWguZWR1L3N5cy9pbnN0
LMOD_CMD=/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8/libexec/lmod
SLURM_LOCALID=0
CVS_RSH=ssh
LESSOPEN=||/usr/bin/lesspipe.sh %s
LMOD_DIR=/uufs/chpc.utah.edu/sys/installdir/lmod/8.6-r8/libexec
BASH_FUNC_which%%=() {  ( alias;
 eval ${which_declare} ) | /usr/bin/which --tty-only --read-alias --read-functions --show-tilde --show-dot $@
}
BASH_FUNC_module%%=() {  eval $($LMOD_CMD bash "$@") && eval $(${LMOD_SETTARG_CMD:-:} -s sh)
}
BASH_FUNC_ml%%=() {  eval $($LMOD_DIR/ml_cmd "$@")
}
_=/usr/bin/env