CSV Parser POC - Githubissues

exalate-issue-sync[bot] commented 1 year ago

There is a new CSV parser POC ready in seprate branch: https://github.com/h2oai/h2o-3/tree/pavel_csv_poc

This parser was inspired by https://github.com/osiegmar/FastCSV.

Parts of the original code are used, even thogh modified. FastCSV is not used as a dependency. We must definitely mention their code has inspired us, because the original code is licensed under Apache 2 License.

Parses documents per line. Recognizes line and its tokens first.
After a line is parsed, the tokens are for further categorization and parsing
Works on multiple chunks (fetches new chunk when a line is obviously not finished and there is still a possibility to read further).
Solves problems with carriage return characters inside quotes
Solves problems with multiple escaped characters and multiple quotes inside original quotes
Parses multi-line text columns correctly.
May be even faster for clean datasets, as the state machine is much simpler for common uses cases.
Memory consumption seems to be the same

New parser memory consumption: !Snímek z 2018-04-08 19-35-57.png|thumbnail! Original parser: !Snímek z 2018-04-08 19-34-29.png|thumbnail!

Cons:

Parser of separate tokens is very simple, uses native java means for Double.valueOf and String.valueOf. This is very expensive in terms of computing power. May be terribly slow, as there are exceptions thrown each time one of these parsers fails to parse a token (String is a fallback, there may be multiple exceptions thrown per token). This is quite on purpose, as this is just a POC to show parsing functionality may be eventually done better.
No optimization done yet. There are multiple spots where optimization would be suitable to speed up the parsing process even more. Especially in RowParser, where array of fields per line grows in the same way ArrayList grows (2x once limit is exceeded). However, we know line length beforehand and that line length is our limit.
Depends on guessSetup phase to guess delimiter and separator (works pretty good actually).

Can parse higgs_10, poets, TED, toxic_comments and other standard keggle datasets our current parser can not.

Stil a very early alpha version.

hasithjp commented 1 year ago

JIRA Issue Migration Info

Jira Issue: PUBDEV-5461 Assignee: New H2O Bugs Reporter: Pavel Pscheidl State: Open Fix Version: N/A Attachments: Available (Count: 2) Development PRs: N/A

Attachments From Jira

Attachment Name: Snímek z 2018-04-08 19-34-29.png Attached By: Pavel Pscheidl File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-5461/Snímek z 2018-04-08 19-34-29.png

Attachment Name: Snímek z 2018-04-08 19-35-57.png Attached By: Pavel Pscheidl File Link:https://h2o-3-jira-github-migration.s3.amazonaws.com/PUBDEV-5461/Snímek z 2018-04-08 19-35-57.png