Do not make EKAT enable FPEs during initialize_ekat_session

[bartgol]

I marked it as a bug, but it's more like a conceptual change. EKAT should not take care of enabling any FPEs when the ekat session is started. There are issues for downstream apps, in some cases:

• Case 1: the downstream app wants different FPEs: it has to disable all fpes, and enable the desired ones.
• Case 2: the downstream app does not want fpes, but needs to disable FPEs set by an even more upstream app.

Proposal: remove "default" fpes from ekat. We can still have them in the ekat_test_session minilib, but not in the main ekat library. The cmake option EKAT_ENABLE_FPE is then simply enabling the "FPE machinery", which allows one to call ekat::enable_fpes(mask) and the likes.

[jeff-cohere]

Is this urgent? Let me know if you want to chat about it.

[bartgol]

No, as I said in Case 1, host apps can still circumvent this by disabling fpes after ekat's session is inited. I could do it in 5 min, but then have to test downstream apps, so I am just logging this here. I tagged you cause you did some work on FPEs, and thought this might affected you.

[jeff-cohere]

One or the other of us will do it when we get a chance. 😄

[bartgol]

Turns out this is more urgent than I thought. Unfortunately, when running with OpenMP, the mask set by ekat at init time is inherited by all threads (I assume during kokkos init), and when we later try to call ekat::disable_all_fpes(), we only disable them on thread 0.

I'll ge this taken care of right away.