Port z4c solver into latest available version of master. - [merged]

[jmstone]

In GitLab by @zappa on Apr 13, 2023, 06:21

_Merges z4crebased -> master

Merge request for the z4c solver into master. The changes are as minimal as possible and the kernels of the original code are used for the fundamental parts. The major additions are the finite difference stencils used for discrete derivatives, defined in

src/utils/finite_diff.hpp

and the usage of athena tensor structures, defined from the athena array structures in

src/athena_tensor.hpp

These additions are anyway restricted to the parts of the code concerning the z4c solver.

A detailed (but probably not completely exhaustive) report of the porting of z4c solver in athenak is in porting_report.md

It seems that there are few conflicts that are due to the extension of z4c-specific lines in the following files. It should be fairly easy to merge manually. Conflicts: src/outputs/basetype_output.cpp src/outputs/outputs.hpp src/outputs/restart.cpp src/pgen/pgen.cpp

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 16:49

Commented on src/mesh/meshblock.cpp line 118

idxn's are removed, now calculated on the fly in par_for's

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/mesh/meshblock_pack.cpp line 175

I think adm might be used for their 3+1 hydro solver. Not sure though

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/outputs/basetype_output.cpp line 122

Agree :>

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/outputs/basetype_output.cpp line 491

done

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/pgen/pgen.cpp line 240

done!

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/pgen/z4c_two_puncture.cpp line 20

Personally, I would like to leave that in, as moving it to another branch would add a lot of complication to all z4c users/developers. Jim said that if the dependencies is contained in the two_puncture pgen, that would be fine with him. Also, the two puncture initial data is too hard to implement in athenak to eliminate this dependence.

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/utils/finite_diff.hpp line 2

Yeah, so far I just commented this file out from the linting test. It probably would be worth it to just modify this file and remove write_fd.py

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/z4c/z4c.hpp line 18

Noted

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/utils/finite_diff.hpp line 6

I think this was a design choice made by the penn state/Jena group. In general, one would set the order based on nghost (no need to increase the communication load for nothing...) Might be too much a pain to change these as they are everywhere in the z4c branch rn. Thanks for the comment, and I'll discuss with Jim further tomorrow!

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/z4c/z4c_adm.cpp line 424

Yes, the ZeroClear wraps in a local_deep_copy, which is by default asynchronous. Otherwise it might launch the inner loop while the R and R_dd arrays are still being cleared.

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on src/z4c/z4c_tasks.cpp line 142

I think rk4 is already in hydro, would be nice to add to mhd and other systems as well! Maybe later when we have 4th order spatial scheme for hydro and mhd implemented.

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Commented on CMakeLists.txt line 124

Again, I think that would be too much a pain developing/running binary black holes. (Though I dislike these dependencies as well! I'll check with Jim further.)

[jmstone]

In GitLab by @Hengrui_Zhu on May 22, 2023, 17:12

Thanks for the ultra-detailed comments! Very helpful for cleaning up this branch!

[jmstone]

In GitLab by @mhguo on May 22, 2023, 17:32

Commented on src/outputs/outputs.hpp line 47

I was wondering whether it would be better if we could list these variable names more compactly.

[jmstone]

In GitLab by @mhguo on May 22, 2023, 18:07

Commented on src/z4c/z4c.hpp line 21

Do we really need some of them? For example, pi is already defined by M_PI and used in most places.

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 09:56

Commented on src/z4c/z4c.hpp line 21

Good point! I searched through the code, and much of these macros are not used at all. I will delete them.

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 09:57

Commented on src/outputs/outputs.hpp line 47

I agree. Maybe the numbering should be within each physics system, instead of listing >100 variables in a giant list?

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 09:59

Commented on src/adm/adm.cpp line 35

changed this line in version 25 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 09:59

Commented on src/mesh/meshblock.cpp line 118

changed this line in version 25 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 09:59

Commented on src/outputs/basetype_output.cpp line 491

changed this line in version 25 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 09:59

Commented on src/pgen/pgen.cpp line 240

changed this line in version 25 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 09:59

Commented on src/z4c/z4c.hpp line 21

changed this line in version 25 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 09:59

added 1 commit

• 5a12c725 - incoorporating comments from Patrick

Compare with previous version

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:09

Commented on py/write_FD.py line 1

changed this line in version 26 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:09

Commented on src/utils/finite_diff.hpp line 6

changed this line in version 26 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:09

added 1 commit

• 380fe773 - finite_diff.hpp linting

Compare with previous version

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:17

Commented on src/utils/finite_diff.hpp line 2

changed this line in version 27 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:17

added 1 commit

• f58f8398 - linting for finite_diff.hpp

Compare with previous version

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:24

added 1 commit

• 7dda6456 - white space ...

Compare with previous version

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:29

Commented on py/write_FD.py line 1

deleted write_fd.py

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:29

Commented on src/utils/finite_diff.hpp line 2

now finite_diff.hpp passes linting test

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:29

Commented on src/utils/finite_diff.hpp line 6

These should be equivalent practically. The spatial differencing operator will use up all the ghost in practice, otherwise one would be wasting mpi communication I suppose

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:29

Commented on src/z4c/z4c_tasks.cpp line 46

Jim mentioned that he wants to get rid of nghost in the input file, and once he decided to implement that for hydro/mhd, I will introduce z4c_order based on which NGHOST is calculated

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:29

Commented on src/z4c/z4c_tasks.cpp line 68

good point, done

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:29

Commented on src/athena_tensor.hpp line 337

I'm not sure, I will consult with Jim. Nice catch!

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:38

Commented on src/z4c/z4c_tasks.cpp line 68

changed this line in version 29 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:38

added 1 commit

• 5acdb846 - change variable naming convention in z4c branch

Compare with previous version

[jmstone]

In GitLab by @Hengrui_Zhu on May 23, 2023, 15:43

added 1 commit

• a0f300c3 - lint

Compare with previous version

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:14

Commented on src/athena_tensor.hpp line 337

changed this line in version 31 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:14

Commented on src/athena_tensor.hpp line 137

changed this line in version 31 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:14

Commented on src/athena_tensor.hpp line 258

changed this line in version 31 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:14

added 1 commit

• d4f34a7c - eliminate cuda dependence

Compare with previous version

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:17

added 1 commit

• cc952421 - set padm to nullptr if both z4c and adm missing from input script

Compare with previous version

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:20

Commented on src/mesh/meshblock_pack.cpp line 175

Now padm is set to null, if both z4c and adm block are not found in the input script.

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:20

Commented on src/athena_tensor.hpp line 337

Making the constructor inline functions fixes the issue and eliminates the cuda specific dependence

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:20

Commented on src/athena_tensor.hpp line 137

as above

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:20

Commented on src/athena_tensor.hpp line 258

same as above

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:22

Commented on src/mesh/meshblock_pack.cpp line 175

changed this line in version 33 of the diff

[jmstone]

In GitLab by @Hengrui_Zhu on May 25, 2023, 11:22

added 1 commit

• 4f97251b - linting

Compare with previous version

[jmstone]

In GitLab by @pdmullen on May 25, 2023, 11:30

resolved all threads