search

ScrapeOps / scrapeops-scrapy-sdk

Scrapy extension that gives you all the scraping monitoring, alerting, scheduling, and data validation you will need straight out of the box.
https://scrapeops.io/
BSD 3-Clause "New" or "Revised" License
37 stars 8 forks source link

Unable to Connect Server #5

Closed wmullaney closed 1 year ago

wmullaney commented 1 year ago

I have attempted to add my server using both methods but have been unable to connect. Currently I am using the following shell script:

wget -O scrapeops_setup.sh "https://assets-scrapeops.nyc3.digitaloceanspaces.com/Bash_Scripts/scrapeops_setup.sh"; bash scrapeops_setup.sh

The only message I receive for troubleshooting is "connection error" I have scrapyd, scrapyd-client, scrapyd-api installed. scrapyd is running as a service and status is "successful". I have not deployed the scrapyd service to the public ip because none of the documentation says to do so. Hopefully someone can point me in the right direction to get something working because I am distraught and defeated at this point.

To reproduce the error have a fresh install of ubuntu pro 20.04 and follow the scrapeops instructions at the following:

https://scrapeops.io/python-scrapy-playbook/extensions/scrapy-scrapyd-guide/

I have exhausted all other options for a scrapy dashboard. I can get gerapy working but it won't deploy my projects. I might have to revisit gerapy as a solution and see if I can just manually deploy my projects. I wanted so bad for scrapeops to work. I was thinking to myself "I will pay/donate good money to the first solution that just friggin works!" However, it appears I can't even throw many at this problem. I am very close to forking gerapy and coding my own solution. I have wasted two days of life I will never get back.

josephkearney91 commented 1 year ago

Hi there, Sorry for the late reply - I wasn't getting notifications for this repo!

If you are just getting "connection error" after running the bash script the next thing to do would be to follow the steps here: https://scrapeops.io/docs/servers-scheduling/scrapyd-integration/#option-2-manual-install

In the manual steps it will bring you through making sure your server is allowing incoming connections from our server as more than likely the firewall is blocking connections - you server would also need a reverse proxy to let connection from scrapeops reach your scrapyd server. The steps in the documentation that I've linked also show you how to do that.