search

geoschem / integrated_methane_inversion

Integrated Methane Inversion workflow repository.
https://imi.readthedocs.org
MIT License
26 stars 23 forks source link

Feature/clustering options #88

Closed laestrada closed 1 year ago

laestrada commented 1 year ago

This PR adds the following clustering functionality to the config file:

ReducedDimensionStateVector: true
ClusteringPairs:
  - [1, 15]
  - [2, 24]
ForcedNativeResolutionPixels: 
  - [31.5, -104]

In this scenario the clustering algorithm would create 39 clusters in the domain of interest with 24 2-grid cell clusters and 15 native resolution pixels. The ForcedNativeResolutionPixels option allows you to specify cells as for native resolution by setting the averaging kernel estimate value to 1.0.

The main file changes for the clustering feature are within the following files:

- aggregation.py
- imi_preview.py
- statevector.sh

Additionally, this PR reorganizes the IMI directories and shell scripts into discrete functions in an attempt to 'componentize' the IMI source code. There are a number of file changes associated with this.

Initial testing with a 1-week inversion shows the following runtime results: Step 71 elements (v1.1.0) 71 elements (clustering branch)
Setup (s) 957 487
Preview (s) 214 121
Spinup (s) 710 710
Jacobian (s) 1530 1500
Inversion (s) 290 290
Posterior (s) 394 390
Total (s) 3881 3377
Posterior emissions (Tg/y) 0.344 0.344

Notably, this PR also reduces the setup time needed significantly from our v1.1.0 benchmark. I think this may be due to the componentization, which reduces some repetition of code (loading environments, etc) and performance enhancements to the IMI preview.

Comparing the 71 element inversions from v1.1.0 and this branch shows no differences as we would expect.

djvaron commented 1 year ago

I haven't grasped all of the intricacies of aggregate.py, but this pull request looks good to me. I really like the "componentization" update. That will be useful for better implementing the Kalman filter feature.