search

microsoft / aurora

Implementation of the Aurora model for atmospheric forecasting
https://microsoft.github.io/aurora
Other
247 stars 31 forks source link

Questions on CAMS experiments #20

Open patel-zeel opened 2 months ago

patel-zeel commented 2 months ago

Hi authors and @wesselb!

It is great to see such a powerful foundation model for air quality. Thank you for making the codebase available. I had a few questions about the CAMS experiments.

  1. In Figure 2 (Aurora outperforms operational CAMS across many targets) of the paper, what was the ground truth to compute the RMSE of CAMS and Aurora and show that Aurora has, at best, 30% better RMSE? For example, the Integrated Surface Database (ISD) was used as the ground truth for meteorology.
  2. Do you have more insights on using or not using emission inventory as input to Aurora? Did you do any small/large scale experiments with/without emission inventory as an input and know whether it is useful? The question is more from an ML point of view on intuitions about the usefulness of emission inventory.
wesselb commented 2 months ago

Hey @patel-zeel! Good to hear from you. :)

We're very excited about the air quality application, and hope to release the air quality version here soon too.

For the CAMS experiments, we used CAMS analysis as the ground truth. (To clarify, the CAMS system produces both forecasts and an analysis product. The forecasts are, well, forecasts; and the analysis product is the system's best estimate of the ground truth.) We did not compare to any station measurements. Such a comparison would be possible, but I think that the model's resolution is just too low for that. At 0.4 degrees, you really only capture average/background levels and no local effects.

Aurora would probably perform better if we also included estimates of anthropogenic factors. We did not do any ablation studies to see what the effect of including an emission inventory is. My intuition is that, given enough data, the model should be able to learn these effects automatically, at least to some extent. The problem is that CAMS data is very scarce, so it's likely that explicitly accounting for these factors will improve performance.

patel-zeel commented 2 months ago

Thank you for the clarification, @wesselb. This is useful.