mlangguth89 / downscaling_benchmark

6 stars 0 forks source link

Ablation study on t2m concerning different predictor variable sets #17

Closed epavel1 closed 1 month ago

epavel1 commented 3 months ago

In this issue, 2 files were added, namely 'ablation_study_predictor_variables.ipynb' and 'results_postprocess.txt'. The notebook visualizes various configurations of models and predictor variable sets to make a decision regarding the selection of predictor variables. A reduced number of predictor variables, denoted as lean, has been chosen. The exact selection is ["t2m_in", "slhf_in", "sshf_in", "sp_in", "z_in", "lsm_in", "t115_in", "t122_in", "t127_in", "t131_in", "t135_in", "fr_land_tar"]. The notebook itself serves as a more detailed documentation. The txt file contains the metric data visualized in the notebook. Based on the results, the best performing UNet and WGAN model were selected. The best achieving WGAN model achieves an rmse of 1.0409 K and a grad amplitude of 0.9478. The best achieving UNet model achieves an rmse of 1.0271 K and a grad amplitude of 0.8774. model_evaluation_season model_evaluation_year

mlangguth89 commented 1 month ago

grafik

For the transformer-based SwinIR-model however, the experiments without the 10m-wind components and the PBL height result into a slight degradation of the performance, mainly in terms of the RMSE. The averaged RMSE difference is about 0.015K which is small, but probably statistically significant.
A possible explanation is that transformers may scale better with an increased number of predictor variables, capturing better their interaction/correlation compared to ConvNets. Thus, it is probably good to stick to the original set of predictor varibales.