philipnye
commented
9 years ago Update: looking into this further, I realised the error was caused by SqlAlchemy, a library that Scraperwiki uses to handle setting up and accessing Sql databases.
I couldn't fully work out why this error was happening, though by changing my code so that all records have a published_recent value of the same type, rather than a mix as I had before (i.e. Booleans for most records, but string for the record that was causing the error) I've managed to stop the error occurring. See here for further details: http://stackoverflow.com/questions/28993750/sqlalchemy-exc-statementerror-invalid-literal-for-int-with-base-10-in-scraper
Although I still don't totally get why my code worked sometimes, and failed others (scraping exactly the same data), but this issue can be closed.
frabcus
Coding in my browser I get the following error:
<Traceback (most recent call last): File "./code/scraper", line 126, in <module> NumberOfOpenSec5Schools=ofsted_scraper(0) #passes scraper '0' for starting number of inspcted schools, runs scraper and saves number of sec 5 inspected schools to this variable File "./code/scraper", line 88, in ofsted_scraper scraperwiki.sql.save(["URN"], school, "School_details_2") File "/usr/local/lib/python2.7/dist-packages/scraperwiki/sql.py", line 195, in save connection.execute(insert.values(row)) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 729, in execute return meth(self, multiparams, params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/sql/elements.py", line 321, in _execute_on_connection return connection._execute_clauseelement(self, multiparams, params) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 826, in _execute_clauseelement compiled_sql, distilled_params File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 893, in _execute_context None, None) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 1160, in _handle_dbapi_exception exc_info File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/util/compat.py", line 199, in raise_from_cause reraise(type(exception), exception, tb=exc_tb) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/base.py", line 889, in _execute_context context = constructor(dialect, self, conn, *args) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/engine/default.py", line 573, in _init_compiled param.append(processors[key](compiled_params[key])) File "/usr/local/lib/python2.7/dist-packages/sqlalchemy/processors.py", line 56, in boolean_to_int return int(value) sqlalchemy.exc.StatementError: invalid literal for int() with base 10: 'none' (original cause: ValueError: invalid literal for int() with base 10: 'none') u'INSERT OR REPLACE INTO "School_details_2" (published_recent, inspection_rating2, schooltype, "LA", "URL", "URN", schoolname, open_closed, opendate_full, inspection_rating, opendate_short, phase, publication_date, include, notes, inspection_date) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)' []>when running<scraperwiki.sql.save(["URN"], school, "School_details_2")>on this line of data:<{u'published_recent': 'none', u'inspection_rating2': 'none', u'schooltype': u'Free school', u'LA': u'Tower Hamlets', u'URL': u'http://www.ofsted.gov.uk/inspection-reports/find-inspection-report/provider/ELS/138262', u'URN': u'138262', u'schoolname': u'City Gateway 14-19 Provision', u'open_closed': u'Open', u'opendate_full': u'2012-09-03 00:00:00.000000', u'inspection_rating': 'No section 5 inspection yet', u'opendate_short': u'September 2012', u'phase': u'Alternative provision', u'publication_date': 'none', u'include': False, u'notes': u'Not yet scraped', u'inspection_date': 'none'}>Saving has worked fine on the previous 48 lines of data I've saved and - to my mind, the oddest thing - when I copy my entire code into a new Code In Your Browser window it works fine with no errors there.
I've tried clearing the dataset, but that doesn't make any difference.
Thanks in advance