search

cldf / pycldf

python package to read and write CLDF datasets
https://cldf.clld.org
Apache License 2.0
15 stars 7 forks source link

Add FIELD_SIZE_LIMIT option to cli #141

Closed xrotwang closed 3 years ago

xrotwang commented 3 years ago

Similar to the tools in csvkit, cldf commands should support an option

  -z FIELD_SIZE_LIMIT, --maxfieldsize FIELD_SIZE_LIMIT
                        Maximum length of a single field in the input CSV file.

to make sure we can also read CSV with large field content - such as GeoJSON data, for example.

xrotwang commented 3 years ago

Maybe setting a higher field size limit in csvw - say 1,000,000 - might make sense as well. But then - as long as we use Python's csv module for the low level parsing, we should probably stick with its defaults to not surprise users (even if it might be nice surprises).