search

shshemi / tabiew

A lightweight TUI application to view and query tabular data files, such as CSV, TSV, or parquet.
MIT License
541 stars 14 forks source link

csv parsing error #6

Closed Chleba closed 4 months ago

Chleba commented 4 months ago

Hi, Great tool, only I just getting problem parsing some csv files because of you trying to parse some collumns into a specific types that are maybe later not the same. I'm only trying to parse csv files from specific bug verified sources for example list of service names and transfer protocols ports (source is here: https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml?skey=-3). I can see that this project is not using csv crate for parsing and it's having it's own, so I have to make an issue for it, because I really loving design of this tool but I can't use it for csv files that I'm working with.

Error:

Error: ComputeError(ErrString("could not parse `5095-5098` as dtype `i64` at column 'Port Number' (column number 2)\n\nThe current offset in the file is 668780 bytes.\n\nYou might want to try:\n- increasing `infer_schema_length` (e.g. `infer_schema_length=10000`),\n- specifying correct dtype with the `dtypes` argument\n- setting `ignore_errors` to `True`,\n- adding `5095-5098` to the `null_values` list.\n\nOriginal error: ```remaining bytes non-empty```"))
Chleba commented 4 months ago

After looking into a code I noticed that You're using polar to parse csv into a DB. From a quick look it can be specified schema for certain columns, but as a user I can't do it.

Chleba commented 4 months ago

SORRY - I can see it now. I have to look at the code only to found out that you have the argument for it --infer-schema safe and that did solved my issue. Sorry about that :)