Switch to MySql has caused what appears to be some unicode issues with the scraper...

[chrislkeller]

Error is:

_mysql_exceptions.Warning: Incorrect string value: '\xC2\x96 Sto...' for column 'conditions' at row 1

Traceback is below…

Traceback (most recent call last):
  File "manage.py", line 10, in <module>
    execute_from_command_line(sys.argv)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/core/management/__init__.py", line 443, in execute_from_command_line
    utility.execute()
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/core/management/__init__.py", line 382, in execute
    self.fetch_command(subcommand).run_from_argv(self.argv)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/core/management/base.py", line 196, in run_from_argv
    self.execute(*args, **options.__dict__)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/core/management/base.py", line 232, in execute
    output = self.handle(*args, **options)
  File "/Users/ckeller/Programming/2kpcc/django-projects/firetracker/calfire_tracker/management/commands/scraper_wildfires.py", line 389, in handle
    iterate_through_urls_to_scrape()
  File "/Users/ckeller/Programming/2kpcc/django-projects/firetracker/calfire_tracker/management/commands/scraper_wildfires.py", line 25, in iterate_through_urls_to_scrape
    extract_details_link_if_present(url_query)
  File "/Users/ckeller/Programming/2kpcc/django-projects/firetracker/calfire_tracker/management/commands/scraper_wildfires.py", line 50, in extract_details_link_if_present
    extract_details_table(fire_name, details_link)
  File "/Users/ckeller/Programming/2kpcc/django-projects/firetracker/calfire_tracker/management/commands/scraper_wildfires.py", line 70, in extract_details_table
    extract_data_from_cells(fire_name, details_link, details_table)
  File "/Users/ckeller/Programming/2kpcc/django-projects/firetracker/calfire_tracker/management/commands/scraper_wildfires.py", line 85, in extract_data_from_cells
    save_data_from_dict_to_model(data_dict)
  File "/Users/ckeller/Programming/2kpcc/django-projects/firetracker/calfire_tracker/management/commands/scraper_wildfires.py", line 271, in save_data_from_dict_to_model
    'notes': notes,
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/models/manager.py", line 134, in get_or_create
    return self.get_query_set().get_or_create(**kwargs)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/models/query.py", line 449, in get_or_create
    obj.save(force_insert=True, using=self.db)
  File "/Users/ckeller/Programming/2kpcc/django-projects/firetracker/calfire_tracker/models.py", line 87, in save
    super(CalWildfire, self).save()
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/models/base.py", line 463, in save
    self.save_base(using=using, force_insert=force_insert, force_update=force_update)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/models/base.py", line 551, in save_base
    result = manager._insert([self], fields=fields, return_id=update_pk, using=using, raw=raw)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/models/manager.py", line 203, in _insert
    return insert_query(self.model, objs, fields, **kwargs)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/models/query.py", line 1576, in insert_query
    return query.get_compiler(using=using).execute_sql(return_id)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 910, in execute_sql
    cursor.execute(sql, params)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/backends/util.py", line 40, in execute
    return self.cursor.execute(sql, params)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/django/db/backends/mysql/base.py", line 114, in execute
    return self.cursor.execute(query, args)
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/MySQLdb/cursors.py", line 203, in execute
    if not self._defer_warnings: self._warning_check()
  File "/Users/ckeller/.virtualenvs/firetracker/lib/python2.7/site-packages/MySQLdb/cursors.py", line 117, in _warning_check
    warn(w[-1], self.Warning, 3)
_mysql_exceptions.Warning: Incorrect string value: '\xC2\x96 Sto...' for column 'conditions' at row 1

Fatal error: local() encountered an error (return code 1) while executing 'python manage.py scraper_wildfires'

Aborting.

[chrislkeller]

This seems to only exist in the local development environment...

[chrislkeller]

Changing the Collation on the MySql database to utf8_general_ci has solved the problem. Was set up as Latin by default. I must have missed a setting somewhere...