convert IndexType to IndexTypeND

[AlexanderSinn]

Summary

Similar to #3969 but for IndexType.

Additional background

A maximum of 31 dimensions are supported so that (1u << dim) can fit into an unsigned int.

Checklist

The proposed changes:

• [ ] fix a bug or incorrect behavior in AMReX
• [x] add new capabilities to AMReX
• [ ] changes answers in the test suite to more than roundoff level
• [ ] are likely to significantly affect the results of downstream AMReX users
• [ ] include documentation in the code and/or rst files, if appropriate

[AlexanderSinn]

Not sure what's wrong with the CUDA EB 2D Hypre CI test.

[WeiqunZhang]

Even without any cache hits, that hypre cuda eb 2d test usually takes about 20 minutes.

[AlexanderSinn]

Im actually able to reproduce the compilation hanging on a HPC cluster (without hypre).

[AlexanderSinn]

It seems that AMReX_EB2_MultiGFab.cpp and AMReX_MLCurlCurl.cpp are affected.

[WeiqunZhang]

For me MLCurlCurl is fine. The issue is this in GFab::buildTypes in AMReX_EB2_MultiGFab.cpp. If I comment that out, it builds.

    AMREX_LAUNCH_HOST_DEVICE_LAMBDA ( nodal_box, tbx,
    {
        amrex_eb2_build_types(tbx, bxg2, s, cell, fx, fy);
    });

[WeiqunZhang]

it's this line cell(i,j,k).setCovered(); on my local computer.

[AlexanderSinn]

For me too, I also commented out cell(i,j,k).setSingleValued();, maybe I was impatient.

[AlexanderSinn]

Not sure what's going on here. If I change it to this, it compiles. Maybe It's something in the if condition that causes it?

        if (    s(i,j  ,k) < 0.0_rt && s(i+1,j  ,k) < 0.0_rt
            && s(i,j+1,k) < 0.0_rt && s(i+1,j+1,k) < 0.0_rt)
        {
            cell(i,j,k).setRegular();
            cell(i,j,k).setCovered();
            cell(i,j,k).setSingleValued();
        }
        else if (s(i,j  ,k) >= 0.0_rt && s(i+1,j  ,k) >= 0.0_rt
            &&  s(i,j+1,k) >= 0.0_rt && s(i+1,j+1,k) >= 0.0_rt)
        {
            //cell(i,j,k).setCovered();
        }
        else
        {
            //cell(i,j,k).setSingleValued();
        }

[WeiqunZhang]

Could you try this?

diff --git a/Src/Base/AMReX_GpuRange.H b/Src/Base/AMReX_GpuRange.H
index ecf9a32fd2..08c77595e7 100644
--- a/Src/Base/AMReX_GpuRange.H
+++ b/Src/Base/AMReX_GpuRange.H
@@ -51,7 +51,7 @@ AMREX_FORCE_INLINE Box at (Box const& b, Long offset) noexcept
                                 static_cast<int>(j),
                                 static_cast<int>(k))};
         iv += b.smallEnd();
-        return Box(iv,iv,b.type());
+        return Box(iv,iv,b.ixType());
     ))
     AMREX_IF_ON_HOST((
         amrex::ignore_unused(offset);

It's more efficient this way anyway.

[WeiqunZhang]

3d actually compiles, but not 2d. This is some weird nvcc bug.

[AlexanderSinn]

It works with the changes to the at function, great!

[WeiqunZhang]

That Windows CI is really slow. https://github.com/AMReX-Codes/amrex/pull/4012