c-proof / pyglider

glider software
https://pyglider.readthedocs.io/
Apache License 2.0
17 stars 24 forks source link

Cannot parse real-time microstructure data #160

Open bastienqueste opened 7 months ago

bastienqueste commented 7 months ago

At line ~131, "pl.read_csv(f, separator=';')" in seaexplorer.py, read_csv cannot parse extreme microstructure values as i64:

ComputeError: Could not parse2.22e-101as dtypei64` at column 'MR1000G-RDL_EPS2' (column number 22). The current offset in the file is 490928 bytes.

You might want to try:

bastienqueste commented 7 months ago

Probably should state that this is not surprising. 2.22e-101 is a stupid value for Rockland and Alseamar to be returning.

jklymak commented 7 months ago

Hmm I'm surprised it won't parse that exponent. If you can link an example or suggest a PR I'm sure this is easy to fix

callumrollo commented 7 months ago

It probably shouldn't be parsing microstructure data as 64 bit integer. I assume it would read fine into a float? Maybe we should start catching and specifying dtypes

bastienqueste commented 7 months ago

I've just made it look a little further before inferring a data type - small slowdown but doesn't fail:

out = pl.read_csv(f, separator=';', infer_schema_length=10000)


Although I may work and communicate at strange hours, I do not expect replies outside of your own working hours. CLICK HERE TO BOOK A MEETING OR CHECK @.***?anonymous&ep=plink>

Dr Bastien Y. Queste Associate professor / Biträdande lektor Department of Marine Sciences, University of Gothenburg (UGOT) Natrium, Medicinaregatan 7B, SE-405 30 Gothenburg, Sweden Tel: +46-73-504-28-39 Home page: http://www.byqueste.com ORCID: https://orcid.org/0000-0002-3786-2275


From: Callum Rollo @.> Sent: 22 January 2024 14:13 To: c-proof/pyglider @.> Cc: Bastien Queste @.>; Author @.> Subject: Re: [c-proof/pyglider] Cannot parse real-time microstructure data (Issue #160)

It probably shouldn't be parsing microstructure data as 64 bit integer. I assume it would read fine into a float? Maybe we should start catching and specifying dtypes

— Reply to this email directly, view it on GitHubhttps://github.com/c-proof/pyglider/issues/160#issuecomment-1903981260, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AABHOCUWKU42PDBEJE5IH5LYPZQXDAVCNFSM6AAAAABCCFA64WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBTHE4DCMRWGA. You are receiving this because you authored the thread.Message ID: @.***>