illinois-ceesd / mirgecom

MIRGE-Com is the workhorse simulation application for the Center for Exascale-Enabled Scramjet Design at the University of Illinois.
Other
11 stars 19 forks source link

Limit pocl to <6 (test revert numpy type promotion) #1055

Closed matthiasdiener closed 5 days ago

matthiasdiener commented 3 weeks ago

Related: https://github.com/inducer/pytato/pull/538, https://github.com/inducer/pytato/pull/510

Questions for the review:

inducer commented 3 weeks ago

What's wrong with pocl 6? This is something I've seen, but it should not affect mirgecom.

matthiasdiener commented 3 weeks ago

What's wrong with pocl 6? This is something I've seen, but it should not affect mirgecom.

We have seen substantial slowdowns in our prediction cases with both the new type promotion code as well as pocl-5 vs pocl-6: (1 rank, wall time per step [s]):

smoke_test_ks_3d Porter/GPU Lassen/GPU M1/CPU CI-Linux/CPU
pocl5, old numpy type promo code 0.17 0.18 0.6 1.05
pocl6, old numpy type promo code 0.58 0.47 0.65 1.41
pocl5, new numpy type promo code 0.21 0.19 6.5
pocl6, new numpy type promo code 0.58 0.5 6.5 16.1

For some reason, pocl-6 seems to affect mostly the CUDA devices, while the type promotion code mostly affects the CPU runs.

inducer commented 3 weeks ago

Yikes. Could you hunt for the simplest example that exhibits the issue and compare the PTX for both? This tool may be able to help. Also, could you file an issue for this? (Since I'm not sure this PR is the best place for the discussion.)