Amending vertical layer structure

[davidorme]

Description

This is a work in progress PR implementing the removal of the explicit subcanopy layer from the vertical layer structure.

The current progress is

• [X] The LayerStructure no longer has the subcanopy layer. The layer_roles attribute is now shorter by 1.
• [X] The LayerStructure.from_template(name) method is added, returning a fresh blank named DataArray with layer_roles and cell_id dimensions.
• [X] There are now two dictionaries LayerStructure.role_indices and LayerStructure.role_indices_bool, depending on whether you want integer indices or boolean indices onto the layer role dimension. Those are keyed by above, canopy, surface, soil.
• [X] Implement the specific topsoil indexing option - the soil layer roles are now explicitly split into a single topsoil role and then additional subsoil layers but an additional all_soil index is provided to refer to all soil layers collectively.
• [x] Add indices to identify the subset of soil layers that are biologically active along with the thicknesses of the layers and the biologically active thickness of each soil layer.

New location for Grid

• [X] As a side effect, the Grid object now also exists within CoreComponents and is inherited by models automatically as Model.grid via the BaseModel.__init__ mechanism. As noted in #376, this is saner place to put it now that CoreComponents exists, and the LayerStructure component needs some minimal information from the grid.

  This PR will not remove Data.grid so we will have two copies of that for a while, but this will be the long term home and it is easier to work with two copies than try and do this all in one go. At some point a separate PR will fix the other half of #376 and remove the dupe from Data.

Things to do:

Fix everything that this breaks:

• [ ] example data and tests - mainly the removal of the subcanopy layer.
• [ ] adoption of the new API across model code. This doesn't strictly have to be done here, but it would be good to pick off as much as possible.

These fixes can be by individual PR onto this branch to pick off issues until we have a passing PR to merge down.

Fixes #421 (issue)

Type of change

• [ ] New feature (non-breaking change which adds functionality)
• [X] Optimization (back-end change that speeds up the code)
• [ ] Bug fix (non-breaking change which fixes an issue)

Key checklist

• [ ] Make sure you've run the pre-commit checks: $ pre-commit run -a
• [ ] All tests pass: $ poetry run pytest

Further checks

• [ ] Code is commented, particularly in hard-to-understand areas
• [ ] Tests added that prove fix is effective or that feature works

[davidorme]

First review of WIP

@jacobcook1995 and @vgro - Do you need any of this implementation changing or extending? @alexdewar Anything obviously mad?

[jacobcook1995]

Yeah looks sensible to me, I'm not planning to continue using topsoil_index in the long run (I don't know if @vgro is though), so from my perspective I'm happy with any implementation that doesn't break the current tests

[vgro]

Looks sensible to me, also good job tidying up the conftest :-) One concern is that I have the parallel branch for including stats which and I don't know in which order we want to merge this in develop to make the least mess ?

[davidorme]

This PR is a little on the long side, as @vgro says. Not sure how easy it would have been to break this work up, but in general I think little PRs often is better than big PRs occasionally, especially when the work touches a lot of different areas of code.

@alexdewar. I completely agree - the problem here is that one of the core changes we want to make in this PR changes the vertical structure of the simulation so that the dimensions of the data change. That's not something that we can adopt incrementally, but we are trying to isolate the bits for different models in sub-branches. Thanks for the comments!

[davidorme]

@dalonsoa Any idea on the qa failure here? The pre-commit.ci is running the same thing with the same configuration - wait. Might be python version?

The only other thing I looked at is the mypy configuration in pyproject.toml specifies the numpy.testing.mypy-plugin, but that shouldn't affect the pre-commit configuration and that is identical between the two actions.

[davidorme]

Not obviously a python version thing. I've switch my poetry environment between 3.10, 3.11 and 3.12 and pre-commit run -a passes on all three.

[codecov-commenter]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 94.75%. Comparing base (d7298af) to head (778ba2b).

Additional details and impacted files

```diff @@ Coverage Diff @@ ## develop #441 +/- ## =========================================== + Coverage 94.66% 94.75% +0.08% =========================================== Files 70 70 Lines 3786 3791 +5 =========================================== + Hits 3584 3592 +8 + Misses 202 199 -3 ```

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[davidorme]

@vgro Can you approve this - we know it's a partial implementation of what is coming in #452 but we should bank it and then we can work on implementing #452 on a separate PR.

[davidorme]

@dalonsoa I think we need you to approve this too after your requested changes. We know it's a halfway house, but we want to be able to move on with other stuff in develop while the changes in #452 are adopted.