search

sensiblecodeio / scraperwiki-python

ScraperWiki Python library for scraping and saving data
https://scraperwiki.com
BSD 2-Clause "Simplified" License
159 stars 69 forks source link

sqlalchemy.exc.OperationalError on writing to a table `details` #88

Closed IanHopkinson closed 9 years ago

IanHopkinson commented 9 years ago

As described in this issue:

https://bitbucket.org/scraperwikids/contract-finder/issues/2/sqlalchemyexcoperationalerror-for-most

This scraper fails on:

scraperwiki.sqlite.save(
            unique_keys=['URL'],
            data=record,
            table_name='details')

For version 0.5.1 with an error that begins:

sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: details [SQL:

pwaller commented 9 years ago

This usually means out of disk or a misnamed or corrupted file or something like that, in my experience. Can you give more detail? Is there any more information available?

pwaller commented 9 years ago

Is the bug reproducible, and can you reproduce it in more than one place?

IanHopkinson commented 9 years ago

I've reproduced the bug in multiple boxes on the platform. I'll try to reproduce on my Windows machine and make a minimal failing case.

pwaller commented 9 years ago

Ah, the interesting part is sqlalchemy.exc.OperationalError: no such table (missed from the issue title and body). Then we need to figure out why table creation isn't happening.

IanHopkinson commented 9 years ago

I was running the database writes inside a dshelpers batchprocessor when I disabled the batchprocessor I revealed a different error - I had specified columns with names Date_scraped and date_scraped which sqlalchemy identified as duplicates. Fixing this and re-applying the batchprocessor seems to have resolved the original problem.