search

audeering / audformat

Format to store media files and annotations
https://audeering.github.io/audformat/
Other
11 stars 1 forks source link

Load csv tables with pandas if pyarrow fails #450

Closed hagenw closed 4 months ago

hagenw commented 4 months ago

Closes #449

Unfortunately, loading csv files with pyarrow.csv.read_csv() as introduced in https://github.com/audeering/audformat/pull/419 is not as tolerant to malformed csv files as pandas.read_csv(). I have identified so far three cases in which loading of a csv file might fail (two of them are listed in #449):

  1. Loading a csv file can fail, if the csv file contains more columns, as mentioned in the header of a database
  2. Loading a csv file can also fail, if it is very long and contains a lot of special characters, like ", "", ,. I did not added a test for it, because it turns out that the syntax of the csv file is correct, and it works when splitting the file into smaller ones.
  3. Loading of a csv file can fail, if it contain some offsets in there date values, e.g. +00:00, which is the case for some of our older datasets.

As pyarrow.csv.read_csv() cannot be easily extended to handle those cases, I use now a try-except statement, that falls back to loading the file with pandas.read_csv(). This is very unfortunate as it means when implementing a new feature (e.g. streaming) it needs to be implemented for both cases. But I don't know a better solution at the moment.

In principle, we could solve 1. and 3. by updating the databases, but you will still no longer be able to load old versions then, which is not acceptable. How we could solve 2. otherwise, I don't know.

codecov[bot] commented 4 months ago

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 100.0%. Comparing base (e3fd511) to head (6f53c6b).

Additional details and impacted files | [Files](https://app.codecov.io/gh/audeering/audformat/pull/450?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=audeering) | Coverage Δ | | |---|---|---| | [audformat/core/table.py](https://app.codecov.io/gh/audeering/audformat/pull/450?src=pr&el=tree&filepath=audformat%2Fcore%2Ftable.py&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=audeering#diff-YXVkZm9ybWF0L2NvcmUvdGFibGUucHk=) | `100.0% <100.0%> (ø)` | |