ecmwf-ifs / loki

Freely programmable source-to-source translation for Fortran
https://sites.ecmwf.int/docs/loki/
Apache License 2.0
29 stars 12 forks source link

Consistent, environment-configurable use of Compiler class in JIT compilation #294

Closed reuterbal closed 5 months ago

reuterbal commented 5 months ago

JIT compilation was hard coded to gfortran and not using the Compiler class in the compile_and_load method used throughout the test base. This prevented testing with other gfortran versions or different compilers.

The first change in this PR is to make Loki use consistently the Compiler class for JIT compilation. This unlocks the use of other compilers when executing tests.

The second change is to inspect the environment for common variables setting the compiler (CC, FC, F90) and choose an appropriate compiler. This consists of two steps:

  1. First, we try to identify the compiler "family", for which now "GNU" and "Nvidia" exist (I used the opportuinity to delete the ancient "ESCAPE" compiler definition). This is done by matching the value in the respective env variable against a set of common names, allowing also for absolute/relative paths (e.g. to test user-supplied compilers). A Fortran compiler specification takes precedence over a C-compiler specification in this case.
  2. Second, the executable name and compiler flags are overwritten with any values supplied in the environment.

As a side-effect, this enables tests that require compilation also on MacOS. With a user-installed GNU compiler suite (e.g. via homebrew) and setting CC=gcc-13 FC=gfortran-13 F90=gfortran-13, the tests use now correctly the user-supplied compilers instead of the system-default "gcc-relabeled" clang compiler.

github-actions[bot] commented 5 months ago

Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/294/index.html

codecov[bot] commented 5 months ago

Codecov Report

Attention: Patch coverage is 83.72093% with 14 lines in your changes are missing coverage. Please review.

Project coverage is 94.93%. Comparing base (59303e4) to head (d3aca00).

Files Patch % Lines
loki/build/compiler.py 83.52% 14 Missing :warning:
Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #294 +/- ## ========================================== - Coverage 94.95% 94.93% -0.03% ========================================== Files 153 153 Lines 32052 32110 +58 ========================================== + Hits 30436 30484 +48 - Misses 1616 1626 +10 ``` | [Flag](https://app.codecov.io/gh/ecmwf-ifs/loki/pull/294/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs) | Coverage Δ | | |---|---|---| | [lint_rules](https://app.codecov.io/gh/ecmwf-ifs/loki/pull/294/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs) | `96.39% <ø> (ø)` | | | [loki](https://app.codecov.io/gh/ecmwf-ifs/loki/pull/294/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs) | `95.10% <83.72%> (-0.03%)` | :arrow_down: | | [transformations](https://app.codecov.io/gh/ecmwf-ifs/loki/pull/294/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs) | `92.14% <ø> (ø)` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

reuterbal commented 5 months ago

Thanks, and apologies for not actually checking this properly...

I had a go at this and there is indeed a simple hook in f2py to specify the Fortran compiler. I've pushed two commits to fix this. We're now consistently using the corresponding methods from the Compiler class to invoke f2py and f90wrap, and provide the compiler selector argument. The only caveat seems to be, that f2py does not seem to work with any other C compiler than gcc...

With this, loading the nvidia module on Atos HPCF (which sets FC, F90 etc. accordingly) and then running the tests with

CC=gcc pytest loki

works correctly in principle (i.e., tests that use JIT compilation, and even some transpilation tests, validate). There's a fair number of tests failing with compiler errors, but that seems to be the Nvidia compiler being genuinely unhappy with some of the files in the tests.

So, I'd be comfortable leaving the NvidiaCompiler class in and claim that the build module supports its use, but not claim full compatibility of the test base with nvfortran. Instead, I'd suggest we defer resolving that to a separate PR.

reuterbal commented 5 months ago

After a slightly longer glimpse, it seems most failing tests are due to the use of Fortran 2008's iso_fortran_env and particularly real64 from that. I'm finding only conflicting information but it looks like nvfortran's support of that is limited.