search

jgieseler / solo-epd-loader

Data loader (and downloader) for Solar Orbiter/EPD energetic charged particle sensors EPT, HET, and STEP. Supports level 2 and low latency data provided by ESA's Solar Orbiter Archive.
BSD 3-Clause "New" or "Revised" License
15 stars 2 forks source link

Replacing FILLVALUES not working with pandas 1.5.0 #16

Closed jgieseler closed 1 year ago

jgieseler commented 1 year ago

At least until pandas 1.4.4 the replacement of FILLVAUES done by the following code worked: https://github.com/jgieseler/solo-epd-loader/blob/f92e4e995a273d5755792c3f02e4ea3c33cfc675/solo_epd_loader/__init__.py#L754-L761

But since pandas 1.5.0 it doesn't work anymore, and the values of -1e+31 are not replaced with np.nan's.

I don't know the reason, maybe it has to do with the fact that the corresponding DataFrames have a MultiIndex.

jgieseler commented 1 year ago

(Hopefully) fixed with https://github.com/jgieseler/solo-epd-loader/commit/7e200042337e4d1acd8b95a809b12a4744f92374

Pandas seems to became really picky and needs now np.float32(-1e+31) instead of just -1e+31 (which will be just float) in the replace function.

jgieseler commented 1 year ago

NB: Code changed with using sunpy 4.1.0 because that replaces FILLVALs directly, so no need anymore to do it manually here.