search

datalad / datalad-crawler

DataLad extension for tracking web resources as datasets
http://datalad.org
Other
5 stars 16 forks source link

ENH: crawl stanford lib initial crawler #17

Closed yarikoptic closed 6 years ago

yarikoptic commented 6 years ago

Extract from larger #11 which was optimistically hoping also add simple indexes into it

vsoch commented 6 years ago

Should link this here https://github.com/datalad/datalad-crawler/issues/18

yarikoptic commented 6 years ago

The failures are the ghost of scrapy... Forgot what to do about them besides restarting so will merge as is

vsoch commented 6 years ago

hey @yarikoptic I was just trying to run the tests locally and it says datalad.api doesn't have module crawl - I'm guessing something about my setup is not finding /registering the module for datalad as a plugin? Is there a way I can run the tests to help debug travis, aside from cloning and setting up my own mirror of travis? I was following the commands in the .travis.yml but I didn't run the script to install neurodebian stuffs (so maybe that led to the issue).

yarikoptic commented 6 years ago

pip install -r requirements.txt

Or just

pip install -e .

Should suffice - both should install datalad