search

beancount / beangulp

Importers framework for Beancount
GNU General Public License v2.0
54 stars 22 forks source link

Add mechanism to allow large files for ingest #18

Closed blais closed 3 years ago

blais commented 4 years ago

Original report by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).

I have a 32 MB CSV file (it contain all invalid transactions that have been attempted in addition to the valid ones that actually went through). bean-extract doesn’t like this because:

# A file size beyond which we will simply ignore the file. This is used to skip
# large files that are commonly co-present in a Downloads directory.
FILE_TOO_LARGE_THRESHOLD = 810241024

Would it be possible to add an option for individual importers to override the size limit (my importer copes with the 32 MB file just fine) or does this have side effects (caching, etc)?

blais commented 4 years ago

Original comment by Daniele Nicolodi (Bitbucket: daniele, GitHub: dnicolodi).

You can simply add something like

from beancount.ingest import identify
identify.FILE_TOO_LARGE_THRESHOLD = 64 * 1024 * 1024

to your importer script.

blais commented 4 years ago

Original comment by Martin Michlmayr (Bitbucket: tbm13, GitHub: tbm).

Thank you Daniele! That works great.