search

rgarner / cma-tna-crawlers

Scraping old cases from TNA for CMA, no TLAs.
0 stars 3 forks source link

Prevent overwriting of some cases #2

Closed rgarner closed 9 years ago

rgarner commented 9 years ago

Already fixed, but leaving as a marker:

If we visit a page that has already been crawled, the crawler will know by URL and not visit it again. However, sometimes the same page can be at more than one URL. In this case, we were overwriting all the cases beginning with A because the query ending ?bytype=atoz&byid=a was equivalent to the different URL ending with ?bytype=atoz. On save, this would start new cases for all cases beginning with A, overwriting anything that had been added as detail (markup sections/dates) in the meantime.

rgarner commented 9 years ago

Fixed by 2e220bc0fbb728fba357af8bc7cb40939cc83985