sensiblecodeio / scraperwiki-python

ScraperWiki Python library for scraping and saving data
https://scraperwiki.com
BSD 2-Clause "Simplified" License
159 stars 69 forks source link

`unique_keys` doesn't work for buffered data #107

Open jfilter opened 4 years ago

jfilter commented 4 years ago

The whole idea around unique_keys=["col"] doesn't work for buffered rows. Since not all modifications are committed directly to the database, this make the unique constraint working unreliably.

To make it work, you have to commit transactions manually after each modification.

scraperwiki.sqlite.commit_transactions()