search

cholla-hydro / cholla

A GPU-based hydro code
https://github.com/cholla-hydro/cholla/wiki
MIT License
60 stars 32 forks source link

Different Initial Conditions in Sod test for 1 rank vs. 4 #221

Closed bcaddy closed 1 year ago

bcaddy commented 1 year ago

While debugging the MHD code I found a weird error. The initial conditions of the single rank vs. four rank Sod test differ slightly. I'm not sure why or how to address it. The issue is present on both the dev and my dev-mhd-integrator branches.

The difference in values on the dev branch: (ignore the panel labeled null, nothing is intended to be plotted there) sod on dev

This is the 1 rank minus the 4 rank initial conditions

evaneschneider commented 1 year ago

Is this true in main as well?

bcaddy commented 1 year ago

I just checked main and it looks like the issue is there as well

evaneschneider commented 1 year ago

Is this for hydro-only builds as well, or does it only show up with MHD on?

evaneschneider commented 1 year ago

Related, is there any way to check whether this was true before the changes made to the Riemann ICs in the first MHD PR?

alwinm commented 1 year ago

When concatenating datasets with the python script, h5py default behavior is to convert to 32-bit, so that when you look at 1-core vs 4-core runs, one can be off by that conversion error. Not sure if that's exactly what's happening here.

If you want you can try https://raw.githubusercontent.com/alwinm/chute/main/cat.py

cat.hydro(i,'','') will concatenate output #i in the current directory. The default behavior is to inherit the datatype of the original hdf5 rather than convert to float32, but the option to convert is there too.

Alternatively, you can edit lines like this in the cholla concatenation script:

fileout.create_dataset(key, (nx, ny, nz), chunks=(nxl,nyl,nzl), dtype=dtype)
bcaddy commented 1 year ago

This fixes the issue. I'll have a PR up shortly to make sure this doesn't happen again