search

ScrapeOps / scrapeops-scrapy-sdk

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.
https://scrapeops.io/
BSD 3-Clause "New" or "Revised" License
37 stars 9 forks source link

Can't add spider into platform #7

Closed andreyluchianic closed 1 year ago

andreyluchianic commented 1 year ago

i have repository with 2 scrapy projects inside (one is very simple other one quite complex) in install script i do install requirements of both after i added repository i can add simple spider but not complex one weird thing is that connecting trough ssh i can successfully run scrapy list but ui says that can't

Screenshot 2023-05-09 at 12 23 07 Screenshot 2023-05-09 at 12 20 41
josephkearney91 commented 1 year ago

Hi there,

I would make sure you have the correct virtual environment activated when you run scrapy list.

If you manually installed all the Python modules that you need for the Scrapy project globally then it might work for you but then when the ScrapeOps dashboard first activates the virtual environment then it might still have a missing module which would cause the scrapy list command to fail.

I need to fix the bug of the dashboard not showing the full traceback though as then you would know what is missing for the command to work correctly.