search

NordicESMhub / NEGI-Abisko-2019

Climate science at high latitudes: eScience for linking Arctic measurements and modeling
https://nordicesmhub.github.io/NEGI-Abisko-2019/
Other
6 stars 33 forks source link

dataframe with cftime issues #107

Open marikoll opened 5 years ago

marikoll commented 5 years ago

here is an example to reproduce the issue https://abisko.uiogeo-apps.sigma2.no/user/a23bf869-f10d-4e06-b63a-3b201c67dda6/lab/tree/Arctic_haze.ipynb

daliagachc commented 5 years ago

It has to do with the index being a cftime object rather than pd.Timestamp. we have similar issues with xarrays too. (see 102. Ive created a function to solve this inside negi-stuff (also instrucction to install update are here)

from negi_stuff.modules import koalas as kl
kl.check_transform_cftime_dim_2_timestamp(df_with_cftime)
daliagachc commented 5 years ago

@marikoll

marikoll commented 5 years ago

@daliagachc now it works with cftime.DatetimeNoLeap, but not cftime.Datetime360Day..

daliagachc commented 5 years ago

hi! i went into the file but im not sure where the error occurs. could you create a small notebook where you open a dataframe with cftime.Datetime360Day so that i can debug? the send me the shareable file again thanks!

marikoll commented 5 years ago

Thank you!

https://abisko.uiogeo-apps.sigma2.no/user/a23bf869-f10d-4e06-b63a-3b201c67dda6/lab/tree/Datetime360Day.ipynb

daliagachc commented 5 years ago

ok. now its fixed. the problem was that if cftime.Datetime360Day, then values like (2019,2,30) are allowed but that makes no sense in pd.Timestamp so we set it to 15 days to be in the middle and avoid the error. 

new_df = kl.check_transform_cftime_dim_2_timestamp(_dsUKESM, middle_of_month=True)

make sure to update negi_stuff. also you may want to resample to month after using the above line of code to make sure that all monthly data is synced e.g.

new_df = new_df.resample('M').mean()
marikoll commented 5 years ago

Great, now it works! Thank you so much!