Repo reorganisation: Moving transformations

[mlange05]

The next step in our spring clean (issue #253) is to move all the internally support transformations into the core package, alongside the associated tests. This then also merges the loki.transform general utilities into the more custom loki.transformations packages.

I've tried to restrain myself to only move files and fix-up import headers, with a few small exceptions listed in the details below:

• Transformation and Pipeline class have been moved into loki.batch as they are the interface to the scheduler
• transformations has been moved into loki.transformations
  • the associated test have been moved in the same fashion
  • the additional conftest.py has been resolved and the runner utilities for CLOUDSC and ecWAM added to loki.tools
• The subpackages loki.transformations.single_column, loki.transformations.transpile and loki.transformations.build_system have been separated
  • DependencyTransformation and ModuleWrapTransformation have been mode to separate files inside trnasformations.build_system
  • CI And packaging has been changes as needed (please double-check this)
  • loki.transform has been moved into loki.transformations, often removing the transform_xxx prefix

And finally, apologies to @MichaelSt98 , as this one might conflict quite a bit with his current stuff... :grimacing:

[github-actions[bot]]

Documentation for this branch can be viewed at https://sites.ecmwf.int/docs/loki/296/index.html

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 95.71046% with 16 lines in your changes are missing coverage. Please review.

Project coverage is 95.09%. Comparing base (102dafd) to head (fb56455).

Files	Patch %	Lines
loki/tools/files.py	15.78%	16 Missing :warning:

Additional details and impacted files

```diff @@ Coverage Diff @@ ## main #296 +/- ## ========================================== + Coverage 94.97% 95.09% +0.12% ========================================== Files 151 165 +14 Lines 32407 35282 +2875 ========================================== + Hits 30779 33552 +2773 - Misses 1628 1730 +102 ``` | [Flag](https://app.codecov.io/gh/ecmwf-ifs/loki/pull/296/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs) | Coverage Δ | | |---|---|---| | [lint_rules](https://app.codecov.io/gh/ecmwf-ifs/loki/pull/296/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs) | `96.39% <ø> (ø)` | | | [loki](https://app.codecov.io/gh/ecmwf-ifs/loki/pull/296/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs) | `95.07% <95.71%> (-0.07%)` | :arrow_down: | | [transformations](https://app.codecov.io/gh/ecmwf-ifs/loki/pull/296/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs) | `?` | | Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=ecmwf-ifs#carryforward-flags-in-the-pull-request-comment) to find out more.

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[MichaelSt98]

Thanks, really nice! I'm happy with it and haven't discovered anything problematic. Nevertheless some comments:

I think in order to be consistent:

• transformations/build_system/tests/test_transform_dependency.py --> transformations/build_system/test_dependency.py
• transformations/tests/test_transform_derived_types.py --> transformations/tests/test_derived_types.py

Why not?

• transformations/transform_loop.py --> transformations/loop.py
• transformations/transform_region.py --> transfrormations/region.py

---

Me thinking out loud:

• introduce subdirectories within transformations ?
  • transformations/temporaries/ (or something like that) for hoist-variables, pool_allocator, ...
  • transformations/control_flow/ (or something like that) for loop, inline, region, ...
  • transformations/utilities/ for utilities, remove_code, drhook, ...
• it makes sense for transformation and pipeline to be within batch as they are interfacing the scheduler, however, I somehow would have expected them to be in transformations/

I guess that further repo reorganisation like "separate operations (inlinecall, subscript, sum, ...) from the symbols/literals" will be done in a future/separate PR?!

[mlange05]

@MichaelSt98 Many thanks for the review - to answer your questions:

1. transformations/build_system/tests/test_transform_dependency.py --> transformations/build_system/test_dependency.py
   transformations/tests/test_transform_derived_types.py --> transformations/tests/test_derived_types.py

   Yes, absolutely, that was an oversight! Will fix in the needed rebase!

2. transformations/transform_loop.py --> transformations/loop.py
   transformations/transform_region.py --> transfrormations/region.py

   Mostly because I actually thought "transform" to be a good high-level description for "fusion/fission/interchange" and "hoist/extract". Open for suggestions though, as the grouping by target "node type" seemed not scalable to me.

3. Agree in general on the proposed further sub-division or the transformations package; this will potentially also require some re-jigging of the stuff insied some of those transformation (eg. lots in SCCBase could go into utility) and I simply wanted to keep this PR manageable.

With regards to Transformation/Pipeline moving, this simply creates a clean import chain, as everything in loki.transformations should be "do something to code", and thus depend on the ir/expression and batch sub-packages. Hope this makes sense. :wink:

[mlange05]

Ok, apologies, this took a bit longer than anticipated. @reuterbal I think I've addressed all comments - many thanks for the suggestions!

I've also tried to address the naming inconsistencies pointed out by @MichaelSt98, but those generated some name clashes in the test base (test_derived_type existed twice). So instead I've opted to rename loki.transformations.derived_type to loki.transformations.transform_derived_types to match the test. I hope this works now, and I think we'll get to tidy this up a little further in a follow-on PR.