search

NGEET / fates

repository for the Functionally Assembled Terrestrial Ecosystem Simulator (FATES)
Other
92 stars 89 forks source link

Model does not warn for NAs in cliamte driver and run as normal #1200

Open XiulinGao opened 1 month ago

XiulinGao commented 1 month ago

I recently discovered that FATES does not given warning messages and run as normal when there are significant amount of NAs in climate drivers. I noticed this because when I run FATES on my own branch (with some new code implmentation), I randomly get model failure with error message pointing to negative relative humidity from the HumanIndexMod on CTSM side. This made me to check the climate driver (which is subsetted point data from a regional climate dataset), and noticed that there are A LOT of NAs in the climate driver, and whole climate data literally has no useful information.... But the model will not fail if it's running on other branches. I am not totally sure what differences in model set up or between branches that triggered this, but it's worth noting that FATES can run when climate data is NA.

Although I have solved this for my own simulations by fixing the data subsetting problem, I think I need to bring this to your attention so people are aware of this. And maybe fix it?

ekluzek commented 1 month ago

I think you mean NaN's (Not a Number) rather than NA? If not I'm not sure what you mean by NA? NaN's happen when a mathematical calculation is done that doesn't have a defined result (sqrt of a negative number for example).

The production build version of the model typically allows NaN's to happen and lets the calculated result be expressed as a NaN. If the calculation is restricted this may be fine, but it can propagate out depending on what is going on. The reason the production code allows NaN's is for performance, checking for NaN's is slower. When we run the code with the DEBUG option turned on, the code will abort when a NaN is calculated. In testing we typically do shorter simulations in DEBUG mode and hence catch problems like this. And we do recommend that if you encounter NaN's that are problems you run in DEBUG mode so that these are caught.

Does that answer your query?

XiulinGao commented 1 month ago

Thanks Erik. by 'NA' I did mean NaN. Does that mean when in DATM mode the model allow most climate variables to be NaNs and only abort under debug mode? It is sort of concerning we can grow a forest when there is literally no precipitation or anything....