search

Queens-Hacks / qcumber-scraper

Scrapes SOLUS and generates structured data
3 stars 6 forks source link

The same course seems to be scraped multiple times #11

Closed mystor closed 10 years ago

mystor commented 10 years ago

I was fiddling with the scraper, and noticed that multiple different threads appear to be scraping the exact same course multiple times...

[2014-05-12 20:17:34,899][INFO][Process-2]: ----Course: PUNS - 0 Level Unspecified
[2014-05-12 20:17:34,899][DEBUG][Process-2]: COURSE DATA DUMP: {'basic': {'description': '', 'number': 'PUNS', 'title': '0 Level Unspecified'}, 'extra': {'CEAB': {}}}
[2014-05-12 20:17:34,914][DEBUG][Process-2]: Returning from a course
[2014-05-12 20:17:34,914][INFO][Process-1]: ----Course: PUNS - 0 Level Unspecified
[2014-05-12 20:17:34,915][DEBUG][Process-1]: COURSE DATA DUMP: {'basic': {'description': '', 'number': 'PUNS', 'title': '0 Level Unspecified'}, 'extra': {'CEAB': {}}}
[2014-05-12 20:17:34,929][DEBUG][Process-1]: Returning from a course

Any thoughts?

mystor commented 10 years ago

(I think that I solved the problem, I think that they were two identical looking courses from two different subjects... I'll confirm that this is correct and then close the issue... Oops)