[WIP] compute and save the ISRF in every cell

[dnarayanan]

the upshot is that by setting in the python code:

m.compute_isrf(True)

hyperion will save the spectrum of energy deposited in every cell. this can be accessed in the hdf5 file via:

import numpy as np
import h5py

f = h5py.File('pd_skirt_comparison.134.rtout.sed')
dset = f['iteration_00007']
print(np.max(dset['specific_energy_sum_nu']))
print(dset['ISRF_frequency_bins'][:])

where the units are erg/s/g (and the "g" is the dust mass in the cell).

this, however, doesn't work for AMR grids because I'm having trouble understanding the file structure, and am unable to get the code to compile. As a result, I'm labeling this PR as a work in progress, and happy to iterate on this. Two main areas I could use eyes from @astrofrog are is:

1. accuracy of code!
2. if #1 is okay, then to see by analogy how to update the AMR code in the fortran modules

[astrofrog]

Thanks for all your work on this! Will try and review soon 😊

[astrofrog]

@dnarayanan - could you rebase this on the latest main branch from this repository? I have finally managed to fix the CI so that it can give meaningful results again.

[dnarayanan]

Happy to ! (Will need to be Monday likely).

Just to confirm since I’m not super with git, I want to follow a procedure like this one (that I googled)?

https://verdantfox.com/blog/view/how-to-git-rebase-mainmaster-onto-your-feature-branch-even-with-merge-conflicts

[astrofrog]

Yes similar to that - but make sure you use this repository as the one you are rebasing on. So assuming you don't already have this repository set up as 'upstream':

git remote add upstream https://github.com/hyperion-rt/hyperion.git
git fetch upstream
git rebase upstream/main

You'll then need to do:

git push -f origin master

to force push the changes since the git history has changed.

I'll keep a backup of your work here in case anything goes wrong :smile:

[dnarayanan]

thanks for the step by step :) i would have never figured out that git order of operations. i've just rebased (i believe)!

[astrofrog]

Great! There are some real test failures, you can run the tests locally by running:

tox -e test

[dnarayanan]

thanks tom! so i have to admit i'm not quite sure how to interpret the tox results. when i run the test locally, i get:

(py38) [desika.narayanan@login6 hyperion_dn]$ tox -e test
warning: no previously-included files found matching 'scripts/mctherm2hyperion'
warning: no previously-included files matching '*.pyx' found under directory 'hyperion'
warning: no previously-included files matching '*.pyc' found under directory 'hyperion'
warning: no previously-included files matching '*.o' found under directory 'hyperion'
warning: no previously-included files found matching 'scripts/mctherm2hyperion'
warning: no previously-included files matching '*.pyx' found under directory 'hyperion'
warning: no previously-included files matching '*.pyc' found under directory 'hyperion'
warning: no previously-included files matching '*.o' found under directory 'hyperion'
test inst-nodeps: /home/desika.narayanan/hyperion_dn/.tox/.tmp/package/1/Hyperion-0.9.11.dev67+gac54b35.tar.gz
test installed: astropy==5.2.2,asttokens==2.4.0,backcall==0.2.0,cmyt==1.3.1,comm==0.1.4,contourpy==1.1.0,cycler==0.11.0,decorator==5.1.1,ewah-bool-utils==1.0.2,exceptiongroup==1.1.3,executing==1.2.0,fonttools==4.42.1,h5py==3.9.0,Hyperion @ file:///home/desika.narayanan/hyperion_dn/.tox/.tmp/package/1/Hyperion-0.9.11.dev67%2Bgac54b35.tar.gz,importlib-resources==6.0.1,iniconfig==2.0.0,ipython==8.12.2,ipywidgets==8.1.0,jedi==0.19.0,jupyterlab-widgets==3.0.8,kiwisolver==1.4.5,matplotlib==3.7.2,matplotlib-inline==0.1.6,more-itertools==10.1.0,mpmath==1.3.0,numpy==1.21.6,packaging==23.1,parso==0.8.3,pexpect==4.8.0,pickleshare==0.7.5,Pillow==10.0.0,pluggy==1.3.0,prompt-toolkit==3.0.39,ptyprocess==0.7.0,pure-eval==0.2.2,pyerfa==2.0.0.3,Pygments==2.16.1,pyparsing==3.0.9,pytest==7.4.1,python-dateutil==2.8.2,PyYAML==6.0.1,six==1.16.0,stack-data==0.6.2,sympy==1.12,tomli==2.0.1,tomli_w==1.0.0,tqdm==4.66.1,traitlets==5.9.0,typing_extensions==4.7.1,unyt==2.9.5,wcwidth==0.2.6,widgetsnbextension==4.0.8,yt==4.2.2,zipp==3.16.2
test run-test-pre: PYTHONHASHSEED='1699501006'
test run-test: commands[0] | pip freeze
astropy==5.2.2
asttokens==2.4.0
backcall==0.2.0
cmyt==1.3.1
comm==0.1.4
contourpy==1.1.0
cycler==0.11.0
decorator==5.1.1
ewah-bool-utils==1.0.2
exceptiongroup==1.1.3
executing==1.2.0
fonttools==4.42.1
h5py==3.9.0
Hyperion @ file:///home/desika.narayanan/hyperion_dn/.tox/.tmp/package/1/Hyperion-0.9.11.dev67%2Bgac54b35.tar.gz
importlib-resources==6.0.1
iniconfig==2.0.0
ipython==8.12.2
ipywidgets==8.1.0
jedi==0.19.0
jupyterlab-widgets==3.0.8
kiwisolver==1.4.5
matplotlib==3.7.2
matplotlib-inline==0.1.6
more-itertools==10.1.0
mpmath==1.3.0
numpy==1.21.6
packaging==23.1
parso==0.8.3
pexpect==4.8.0
pickleshare==0.7.5
Pillow==10.0.0
pluggy==1.3.0
prompt-toolkit==3.0.39
ptyprocess==0.7.0
pure-eval==0.2.2
pyerfa==2.0.0.3
Pygments==2.16.1
pyparsing==3.0.9
pytest==7.4.1
python-dateutil==2.8.2
PyYAML==6.0.1
six==1.16.0
stack-data==0.6.2
sympy==1.12
tomli==2.0.1
tomli_w==1.0.0
tqdm==4.66.1
traitlets==5.9.0
typing_extensions==4.7.1
unyt==2.9.5
wcwidth==0.2.6
widgetsnbextension==4.0.8
yt==4.2.2
zipp==3.16.2
test run-test: commands[1] | /home/desika.narayanan/hyperion_dn/configure --prefix=/home/desika.narayanan/hyperion_dn/.tox/test
checking for h5fc... h5fc
checking for h5pfc... no
checking for mpif90... mpif90
configure: creating ./config.status
config.status: creating Makefile
test run-test: commands[2] | mv Makefile /home/desika.narayanan/hyperion_dn/
test run-test: commands[3] | make -C /home/desika.narayanan/hyperion_dn serial
make: Entering directory `/home/desika.narayanan/hyperion_dn'
h5fc -module src/modules -heap-arrays -error-limit 1 -fp-model strict -diag-disable 8291    src/mpi/mpi_core_nompi.f90 fortranlib/src/posix_default.f90 fortranlib/src/base_types.f90 fortranlib/src/lib_messages.f90 fortranlib/src/lib_io.f90 fortranlib/src/lib_version.f90 fortranlib/src/lib_algebra.f90 fortranlib/src/lib_conf.f90 fortranlib/src/lib_array.f90 fortranlib/src/lib_statistics.f90 fortranlib/src/lib_random.f90 fortranlib/src/lib_hdf5_110.f90 fortranlib/src/lib_constants.f90 fortranlib/src/type_angle3d.f90 fortranlib/src/type_vector3d.f90 fortranlib/src/type_stokes.f90 fortranlib/src/type_pdf.f90 src/core/core_lib.f90 src/main/counters.f90 src/mpi/mpi_io_nompi.f90 src/grid/type_grid_cartesian_3d.f90 src/core/type_cell_id_3d.f90 src/core/type_photon.f90 src/main/settings.f90 src/main/performance.f90 src/images/image_type.f90 src/dust/dust_type_4elem.f90 src/dust/dust.f90  src/grid/grid_io.f90 src/grid/grid_geometry_cartesian_3d.f90 src/grid/grid_geometry_common_3d.f90 src/grid/grid_pda_cartesian_3d.f90 src/grid/grid_physics_3d.f90 src/grid/grid_monochromatic.f90 src/dust/dust_interact.f90 src/sources/source_type.f90 src/sources/source.f90 src/grid/grid_propagate_3d.f90 src/grid/grid_mrw_3d.f90 src/grid/grid_pda_3d.f90 src/grid/grid_generic.f90 src/images/images_binned.f90 src/images/images_peeled.f90 src/main/forced_interaction.f90 src/main/setup_rt.f90 src/mpi/mpi_routines_nompi.f90 src/main/iter_lucy.f90 src/main/iter_final.f90 src/main/iter_final_mono.f90 src/main/iter_raytracing.f90 src/main/main.f90 -o bin/hyperion_car

Error: A license for Comp-FL could not be obtained (-1,359,2).

Is your license file in the right location and readable?
The location of your license file should be specified via
the $INTEL_LICENSE_FILE environment variable.

License file(s) used were (in this order):
**  1.  /apps/compilers/intel/2018/1.163/compilers_and_libraries_2018.1.163/linux/bin/intel64/../../Licenses
**  2.  /opt/intel/licenses
**  3.  /Users/Shared/Library/Application Support/Intel/Licenses
**  4.  /apps/compilers/intel/2018/1.163/compilers_and_libraries_2018.1.163/linux/bin/intel64/*.lic

Please refer http://software.intel.com/sites/support/ for more information..

ifort: error #10052: could not checkout FLEXlm license
make: *** [car_serial] Error 1
make: Leaving directory `/home/desika.narayanan/hyperion_dn'
ERROR: InvocationError for command /usr/bin/make -C /home/desika.narayanan/hyperion_dn serial (exited with code 2)

but i'm guessing that that's not the issue for why the tests are failing. when i click on the details of the tox tests reported here in github, i'm not sure what they're reporting. very happy to hunt these down, but i'm afraid i'm not even sure what i'm looking for here.

[dnarayanan]

indeed, a license does seem to exist:

(py38) [desika.narayanan@login6 hyperion_dn]$ echo $INTEL_LICENSE_FILE
/apps/compilers/intel/licenses

[dnarayanan]

hi tom -

it seems in some tests of mine, that i find that the ISRF saved is much higher in runs where i employ more than 1 MPI core. do you have any sense as to where i might begin looking in the code to debug this? i can't tell what might be going on...if there's a normalization by the number of cores that needs to happen somewhere perhaps.

it's not something as simple as the ISRF with 32 cores (for example) is 32x the ISRF energy deposited for a single core...it's like many orders of mag higher.

[astrofrog]

@dnarayanan - sorry for not getting back to you sooner. I am currently in the process of trying to fix compatibility of Hyperion with recent Python/Numpy/Astropy versions and will take a look at this soon.

[dnarayanan]

sounds good thanks!

[astrofrog]

@dnarayanan - now that I've unbroken the CI, could you rebase on top of the latest main branch? I'll then try and take a look at this.

[dnarayanan]

sorry about the delay - i just rebased and pushed!

[dnarayanan]

thanks for finding the typo! i'm trying to figure out how to push the fix but i think i need to set a rule for reconciling divergent branches after the rebase before i can push to github. any advice is helpful!

 ! [rejected]        master -> master (non-fast-forward)
error: failed to push some refs to 'https://github.com/dnarayanan/hyperion.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Integrate the remote changes (e.g.
hint: 'git pull ...') before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.
[desika.narayanan@login7 conf]$ git pull
hint: You have divergent branches and need to specify how to reconcile them.
hint: You can do so by running one of the following commands sometime before
hint: your next pull:
hint:
hint:   git config pull.rebase false  # merge
hint:   git config pull.rebase true   # rebase
hint:   git config pull.ff only       # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.
fatal: Need to specify how to reconcile divergent branches.

not sure which of the 3 options above are best

[astrofrog]

Can you let me know the output of 'git remote'?

[dnarayanan]

[desika.narayanan@login7 hyperion_dn_rebase]$ git remote
origin
upstream

[astrofrog]

Thabks! And the output of 'git log' and 'git status'?

[dnarayanan]

(py38) [desika.narayanan@login7 hyperion_dn_rebase]$ git log
commit 9d028bf01cdd651ef63ca765e6c460909ce48181 (HEAD -> master)
Author: dnarayanan <desika.narayanan@gmail.com>
Date:   Wed May 15 09:30:47 2024 -0400

    fixing typo noticed by tom

commit ac54b3506c054fabf7c1e5ad689bf8477693fa16
Author: dnarayanan <desika.narayanan@gmail.com>
Date:   Thu Aug 11 11:49:45 2022 -0400

    critical bug fix changing indexes starting from 0 to 1 [which caused, naturally, random segfaults]

commit a31bd0cdafe1ea7b6f1ded20ac528960bca4338c
Author: dnarayanan <desika.narayanan@gmail.com>
Date:   Tue May 24 12:42:56 2022 -0400

    one more 5d file turned off

commit 9d0f4d5d2c2b27f546a2edef7e2858694524b67b
Author: dnarayanan <desika.narayanan@gmail.com>
Date:   Tue May 10 12:14:36 2022 -0400

    getting rid of read_grid_5d since its not actually used

commit 3ab74b0c43584fab966093946e6b593250580c34
Author: dnarayanan <desika.narayanan@gmail.com>
Date:   Mon May 9 12:22:59 2022 -0400

    bugs fixed that kept code from running when ISRF is turned off

commit f1696d36df79478a588e20e47738a19d2448aab3
Author: dnarayanan <desika.narayanan@gmail.com>
Date:   Mon May 9 08:51:16 2022 -0400

    some bugs corrected fixing the ISRF calculation.   but some issues remain now when running without ISRF calculation turned on

```
(py38) [desika.narayanan@login7 hyperion_dn_rebase]$ git status

On branch master Your branch and 'origin/master' have diverged, and have 15 and 68 different commits each, respectively. (use "git pull" to merge the remote branch into yours)

Untracked files: (use "git add ..." to include in what will be committed) .tox/ hyperion_dn_rebase/ src/grid/#grid_generic.f90# src/mpi/#mpi_io.f90# tatus

nothing added to commit but untracked files present (use "git add" to track)

[astrofrog]

Thanks! Will take a close look this evening and suggest a fix