datatogether / archivertools

Python package for scraping websites into the Data Together pipeline via morph.io
GNU Affero General Public License v3.0
6 stars 1 forks source link

Morph requires a specific Scraperwiki version, causes pip install to not work #1

Closed jeffreyliu closed 7 years ago

jeffreyliu commented 7 years ago

The morph version of scraperwiki is different than the one on PyPI. Morph's template points to a custom version, which is forked and desynced from the main scraperwiki.

This is a pain because pip install doesn't follow dependency links by default, and the --process-dependency-links flag is deprecated and is scheduled for removal in a future release of pip

Since the bit of scraperwiki that we use is just for the database interaction bits, I propose that we just implement the bits of read/write ourselves, especially since scraperwiki doesn't seem to be actively maintained anymore

jeffreyliu commented 7 years ago

@danielballen, In trying your suggestion as a hotfix, I'm running into the the following error: error in archivertools setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers; Invalid requirement, parse error at "'-e git+h'"

the requirements.txt looks like: requirements.txt

I'd also initially tried with the syntax - git+https://... as given in the suggestion, but it gives the same error, except the end looks like Invalid requirement, parse error at "'- git+ht'" instead

jeffreyliu commented 7 years ago

Fixed by https://github.com/datatogether/archivertools/pull/3