search

nickmckay / LiPD-utilities

Input/output and manipulation utilities for LiPD files in Matlab, R and Python
http://nickmckay.github.io/LiPD-utilities/
GNU General Public License v2.0
29 stars 9 forks source link

Python: Incorrect inferred data calculations #4

Closed chrismheiser closed 7 years ago

chrismheiser commented 7 years ago

"For some reason the min/max... of the resolution has NaN and it breaks the JSON and the LiPD uploader" MD97-2141.Rosenthal.2003.xlsx MD97-2141.Rosenthal.2003.zip

nickmckay commented 7 years ago

Do you know where this is calculated? Must be in the python utilities... Should be ignoring missing values there

chrismheiser commented 7 years ago

Yes it is a Python issue. I should have mentioned that.

CommonClimate commented 7 years ago

This is easy to fix with numpy. Are you working with lists or numpy arrays?

Julien Emile-Geay, associate professor University of Southern California http://https://scholar.google.com/citations?user=OVti4jEAAAAJ&hl=enclimdyn.usc.eduhttp://climdyn.usc.edu

On May 17, 2017, at 10:29, Christopher Heiser notifications@github.com<mailto:notifications@github.com> wrote:

Yes it is a Python issue. I should have mentioned that.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_nickmckay_LiPD-2Dutilities_issues_4-23issuecomment-2D302166219&d=DwMCaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=sm7R-KWGS-qL7K-XLvSPqQ&m=RAP7FSsiw1vhSt083CiVh4IloazXaGQz04Arc_y21T4&s=-fwSaniJmlbRKLdi2nnPOs4bv-8llbZKWmJJJFX3CKY&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ANH234XE1Hy2r6bYaR-2DKdUYQdkX-5F2djBks5r6y6TgaJpZM4NeM1C&d=DwMCaQ&c=clK7kQUTWtAVEOVIgvi0NU5BOUHhpN0H8p7CSfnc_gI&r=sm7R-KWGS-qL7K-XLvSPqQ&m=RAP7FSsiw1vhSt083CiVh4IloazXaGQz04Arc_y21T4&s=EsueFQk8qpHYuIdLOyz04PCHwqWwIs3Kt7v3mT9saD0&e=.

chrismheiser commented 7 years ago

@CommonClimate numpy was actually my first issue. Since it uses non-native data types it was giving me problems when trying to move back to python data types for writeLipds(). I switched to native python max/min/mean/median functions for this, but clearly it doesn't know how to handle NaNs properly.

khider commented 7 years ago

Also I have successfully transformed datasets with NaNs without any problems. Not sure what's so special about this file.

chrismheiser commented 7 years ago

Reverted to old code that uses numpy module, with some safeguards. Tested.