Open bastienqueste opened 7 months ago
Probably should state that this is not surprising. 2.22e-101 is a stupid value for Rockland and Alseamar to be returning.
Hmm I'm surprised it won't parse that exponent. If you can link an example or suggest a PR I'm sure this is easy to fix
It probably shouldn't be parsing microstructure data as 64 bit integer. I assume it would read fine into a float? Maybe we should start catching and specifying dtypes
I've just made it look a little further before inferring a data type - small slowdown but doesn't fail:
out = pl.read_csv(f, separator=';', infer_schema_length=10000)
Although I may work and communicate at strange hours, I do not expect replies outside of your own working hours. CLICK HERE TO BOOK A MEETING OR CHECK @.***?anonymous&ep=plink>
Dr Bastien Y. Queste Associate professor / Biträdande lektor Department of Marine Sciences, University of Gothenburg (UGOT) Natrium, Medicinaregatan 7B, SE-405 30 Gothenburg, Sweden Tel: +46-73-504-28-39 Home page: http://www.byqueste.com ORCID: https://orcid.org/0000-0002-3786-2275
From: Callum Rollo @.> Sent: 22 January 2024 14:13 To: c-proof/pyglider @.> Cc: Bastien Queste @.>; Author @.> Subject: Re: [c-proof/pyglider] Cannot parse real-time microstructure data (Issue #160)
It probably shouldn't be parsing microstructure data as 64 bit integer. I assume it would read fine into a float? Maybe we should start catching and specifying dtypes
— Reply to this email directly, view it on GitHubhttps://github.com/c-proof/pyglider/issues/160#issuecomment-1903981260, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AABHOCUWKU42PDBEJE5IH5LYPZQXDAVCNFSM6AAAAABCCFA64WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMBTHE4DCMRWGA. You are receiving this because you authored the thread.Message ID: @.***>
At line ~131, "pl.read_csv(f, separator=';')" in seaexplorer.py, read_csv cannot parse extreme microstructure values as i64:
ComputeError: Could not parse
2.22e-101as dtype
i64` at column 'MR1000G-RDL_EPS2' (column number 22). The current offset in the file is 490928 bytes.You might want to try:
infer_schema_length
(e.g.infer_schema_length=10000
),dtypes
argumentignore_errors
toTrue
,2.22e-101
to thenull_values
list.`