sensiblecodeio / scraperwiki-python

ScraperWiki Python library for scraping and saving data
https://scraperwiki.com
BSD 2-Clause "Simplified" License
159 stars 69 forks source link

SQLAlchemy error storing empty strings #74

Closed psychemedia closed 9 years ago

psychemedia commented 9 years ago

I've just updated an old scraperwiki library for the latest pip version and one a breaking change I've come across seems to be the storage of empty strings when SQLalchemy is trying to cast to a float (sqlalchemy.exc.StatementError: could not convert string to float).

I've tidied up functions where I'm supposed to return a number from returning an empty string (which always used to get added as NULL, I think?) to returning a float('nan'), but it strikes me that in the general quick'n'dirty scruffy case where you're scraping and throwing as much stuff into SQLite as quickly as possible that for scraperwiki to be able to cast empty string to NaN or None/NULL as a default when SQLalchemy is trying to cast it to a float, rather than throwing an error, would be handy?

pwaller commented 9 years ago

@psychemedia can you provide a minimal example to reproduce?

psychemedia commented 9 years ago

Hi

Seems I can't... I updated a whole raft of old libraries, including sqlalchemy, and can't replicate the minimum example of what I thought the problem was offhand...

Apols about that... Will put it down to old libraries... I'll close the issue.