search

theanti9 / PyCrawler

A python web crawler
212 stars 104 forks source link

error on finishing #3

Closed sjparsons closed 12 years ago

sjparsons commented 13 years ago

I get the following error at the end of my execution of the script. It does, however, successfully store everything in the SQLite db.

Continuing without psyco JIT compilation! Traceback (most recent call last): File "/Users/sam/bin/pycrawler", line 210, in threader().run() File "/Users/sam/bin/pycrawler", line 97, in run self.crawl(crawling) File "/Users/sam/bin/pycrawler", line 170, in crawl self.queue_links(url, links, cid, curdepth) File "/Users/sam/bin/pycrawler", line 184, in queue_links cursor.execute("SELECT url FROM queue WHERE url=?", [link]) sqlite3.ProgrammingError: You must not use 8-bit bytestrings unless you use a text_factory that can interpret 8-bit bytestrings (like text_factory = str). It is highly recommended that you instead just switch your application to Unicode strings.

theanti9 commented 12 years ago

Closing old issues after completely rewriting the program