Include scrapyd as a dependency

[LexiconCode]

There is a good deal of confusion among users when they encounter the following errors due to scrapyd not being installed. See 1,2. It seems to me that this is a dependency that is been missed as it's not mentioned anywhere in the installation section of portia.readthedocs

When scraping is triggered through the portia GUI or Console shows - ConnectionError: HTTPConnectionPool(host='localhost', port=6800): Max retries exceeded

It seems based on the above threads that scrapyd is a requirement.

The docker file should include scrapyd as a dependency and should be mentioned in the install section of the documentation.

Thank you for your time and if I need to provide any more info please let me know.

[ruairif]

Hi @LexiconCode I totally agree that it is confusing and should be included. Would you consider making a PR with the updated docs and changes to docker? You could also consider including SpiderKeeper which is a nicer UI for scrapyd.

[LexiconCode]

A couple of questions. @ruairif Do you have a working install of portia and does it include scrapyd? If you do, for expediency's sake what commands did you use for install?

As for including something like another program like SpiderKeeper is really out of the scope of portia. We might include it as a complementary program with in the documentation. I'm under the impression that scrapyd is required to have a fully deployed portia. In that case scrapyd would be considered a dependency.

Sent from my Motorola XT1635-02 using FastHub

[cwichka]

Hi @LexiconCode
you're the only one i've found speaking about including spiderkeeper and portia. I want to do sthg like this, like scheduling spiders created with portia using the spiderkeeper dashboard. Did u find a solution about including these projects ?

Thanks in advance !

[LexiconCode]

@ZeroEffort

I have not found a solution for this and given up on portia. I'm looking for alternative currently since there is zero support.