pudo / dataset

Easy-to-use data handling for SQL data stores with support for implicit table creation, bulk loading, and transactions.
https://dataset.readthedocs.org/
MIT License
4.78k stars 298 forks source link

insert_many: maximum chunk_size #295

Closed PlatonB closed 5 years ago

PlatonB commented 5 years ago

Please specify in the API documentation what maximum chunk_size can be used. For example, my laptop has 8 GB RAM, and if program tries to insert barely 100 MB list into SQLite DB, OS hangs.

while True:
    if sys.getsizeof(fragment) / 1e8 > done_commits + 1:
        table.insert_many(fragment, chunk_size=len(fragment))
        fragment.clear()
        done_commits += 1
pudo commented 5 years ago

Do you have a strict limit that can be correct all database errors? Otherwise I'd argue that this is an application-level concern to sort out and we cannot give guidance on this.