search

NVIDIA / modulus

Open-source deep-learning framework for building, training, and fine-tuning deep learning models using state-of-the-art Physics-ML methods
https://developer.nvidia.com/modulus
Apache License 2.0
797 stars 174 forks source link

🚀[FEA]: sort-based CRPS #455

Closed nbren12 closed 2 months ago

nbren12 commented 2 months ago

Is this a new feature, an improvement, or a change to existing functionality?

New Feature

How would you describe the priority of this feature request

Medium

Please provide a clear description of problem you would like to solve.

Can we add a sort-based implementation of exact CRPS to the metrics? In theory this should give the same answer as the kernel method, but in O(n log n) time and memory rather than O(n^2).

earth2mip has an implementation here: https://github.com/NVIDIA/earth2mip/blob/main/earth2mip/crps.py. There are also some tests. I've used this for large-scale scoring. Would a PR be welcome?

Describe any alternatives you have considered

No response

nbren12 commented 2 months ago

@dallasfoster

dallasfoster commented 2 months ago

I think such a PR would be welcomed. As far as tests, I think we are trying to keep properscoring out of the requirements.

dallasfoster commented 2 months ago

The other thought would be if you think the method should come in as crps_from_empirical_cdf or as _crps_from_empirical_cdf and be available under the crps(..., method = 'empirical_cdf)` method.

nbren12 commented 2 months ago

How about method="sort"? That would make it easier for people to discover probably.

nbren12 commented 2 months ago

I think such a PR would be welcomed. As far as tests, I think we are trying to keep properscoring out of the requirements.

Perhaps I can test against "kernel".

dallasfoster commented 2 months ago

I think the following tests would be helpful:

  1. Accuracy test against 'kernel'
  2. Scaling/accuracy tests on N(0,1) (same as other methods)
  3. Negative tests for shape, device, dtype mismatches