search

openaustralia / morph

Take the hassle out of web scraping
https://morph.io
GNU Affero General Public License v3.0
461 stars 74 forks source link

something changed in requirements.txt? #784

Open mobeets opened 9 years ago

mobeets commented 9 years ago

I'm not sure exactly what's going on, but my scraper stopped being able to find its database, so any data scraped was immediately forgotten. (This started happening on 2015/4/28, in case that date coincides with some code change?)

My requirements.txt file ended up being the culprit, which had previously been working fine as follows:

scraperwiki==0.3.11
python-dateutil==2.1
BeautifulSoup==3.2.1
unidecode==0.04.16

For whatever reason, I had to change the scraperwiki==0.3.11 part to -e git+http://github.com/openaustralia/scraperwiki-python.git@morph_defaults#egg=scraperwiki, which I found on the python docs page as being the default.

Any ideas why this had to change? Ideally I'd like to use a normal scraperwiki version.

henare commented 9 years ago

Hmm, I've no idea why this would've changed. AFAIK it shouldn't have worked before because the scraperwiki library specified in requirements.txt would've written data to scraperwiki.sqlite and not data.sqlite.

Maybe @mlandauer can shed some light?

mobeets commented 9 years ago

Yeah, true, but for whatever reason I think it failed in different ways. So before, it couldn't find the database to load the data, but it could still write to it. So I'd always see things like "1554 records added, 1554 records removed in the database". But at some point around 3 months ago it just started saying "nothing changed in the database".