search

theanti9 / PyCrawler

A python web crawler
212 stars 103 forks source link

Code out of box works for while then gives this error #9

Open ghost opened 12 years ago

ghost commented 12 years ago

using linux box sqlite

[21:15:00] INFO::PyCrawler - Starting (http://www.dmoz.org)... [21:15:00] ERROR::PyCrawler - EXCEPTION: 'ascii' codec can't decode byte 0xc3 in position 31: ordinal not in range(128) Traceback (most recent call last): File "PyCrawler.py", line 88, in crawl() File "PyCrawler.py", line 70, in crawl if not cdb.checkCrawled(q): File "/home/oguz/Desktop/taraa/query.py", line 84, in checkCrawled s = select([self.crawl_table]).where(self.crawl_table.c.address == unicode(url)) UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 31: ordinal not in range(128)

theanti9 commented 12 years ago

If you do a search for this error, you'll find that this probably has something to do with the charset that the database uses (particularly if you're using it with mysql, i believe that is the driver i saw the post about with this similar error). You can read about it here:

http://stackoverflow.com/questions/10264774/unicodedecodeerror-ascii-codec-cant-decode-byte-0x96-in-position-10-ordinal

I'm actually not real sure about how to set that with SQLAlchemy, but I'm sure it's in the documentation somewhere. I don't really have time to work on this project right now unfortunately but I believe this fix is pretty simple. If you wouldn't mind, i'd love for you to fork it and make any changes you see fit to solve this, the other issue you opened, or any others you find, and I can merge them back later.