search

microsoft / ClimaX

Foundation model for weather & climate
https://microsoft.github.io/ClimaX/
MIT License
626 stars 82 forks source link

The replication issues with the downscaling task. #37

Open Tttizi opened 9 months ago

Tttizi commented 9 months ago

  1. In attempting the Downscaling task, following the publicly available code on GitHub did not yield the reported performance in the paper. Specifically, the Root Mean Squared Error (RMSE) for T2m was 6.08, whereas the paper reports 2.79. I am uncertain if there are key points I should be mindful of to address this discrepancy.

  2. I noticed some discrepancies between the descriptions in the paper and the provided code, such as the setting of the learning rate. Despite trying various combinations, I have been unable to obtain the correct results. I would appreciate your advice and guidance on this matter.

  3. I would like to inquire about the choice of the pre-training model—should I select the 1.40625-degree model? I have encountered some confusion during my attempts, and I am seeking your professional opinion on this matter.

tung-nd commented 9 months ago

Hi, thank you for your interest in ClimaX. I answer the questions as follows:

  1. Can you elaborate on what the differences are?
  2. Yes, you should use the 1.40625deg model. What issues did you run into when trying to use it?
Tttizi commented 9 months ago

Thank you very much for your response. I have noticed three differences between the paper and the code. First, in the paper, the learning rate for the downscaling task is 5e-5, while in the code, it is set to 5e-4. Second, the warmup setting in the paper is not explicitly mentioned, but from the code, it seems to have exceeded 5 epochs. Third, in the paper, it is stated that you trained different networks for different features, while in the code, these features are predicted together. I have attempted to adjust these settings, but the performance is still not satisfactory. Therefore, I hope you can provide more details on how each feature corresponds to specific settings or offer more detailed guidance on how to reproduce the results from the paper.

Tttizi commented 9 months ago

I have another question regarding the data. There are two issues with the data provided in the Hugging Face link. First, it lacks data for the features "10_m_u_component_of_wind" and "10_m_v_component_of_wind." Second, the data does not match the WeatherBench dataset. Since there are no timestamps, I extracted data for one day and compared it with the data corresponding to that year in the WeatherBench dataset. Unfortunately, I couldn't find matching data.

Tttizi commented 8 months ago

Hi, just wanted to check if there have been any updates on this issue.

Tttizi commented 7 months ago

Hi, just wanted to check if there have been any updates on this issue.

Tttizi commented 7 months ago

I've noticed in the code that during the network initialization, there is a feature called land_sea_mask. Is this feature used in the downscaling task? Where is this data obtained from?

Escape142 commented 4 months ago

@tung-nd is Cli-ViT from ClimaX paper is the same as ViT in Climate Learn paper?