Adding custom endpoints

[vazkir]

Hi,

I was wondering if it possible to add custom endpoints scrapyd? So to extend the current endpoints by allowing people to write one themselves?

Or perhaps if anyone knows if any workaround to achieve this?

Would very beneficial to developers like my who want to write some of their own logic by creating new endpoints, especially in a docker environment where you have only 1 ENTRYPOINT to interact with, and running only 1 program is recommanded

[jpmckinney]

Can you give an example of an endpoint you would like to add?

[vazkir]

Ofcourse; I wanted to create new spiders automatically based on api input. Which I got working, so for anyone interested:

You can use the WsResource to create similar endpoints by importing it and using it like this:

from scrapyd.webservice import WsResource 

# Examples of any existings scrapyd endpoints: https://github.com/scrapy/scrapyd/blob/master/scrapyd/webservice.py
class TestNewEndpoint(WsResource):

    def render_GET(self, txrequest):

        # projects = list(self.root.scheduler.list_projects())
        return {"node_name": "nodeje", "status": "ok", "projects": {"yeag":"fsf"}}

And then in the scrapyd.conf you want to add your custom endpoint under [services] like this:

...............
[services]
schedule.json     = scrapyd.webservice.Schedule
cancel.json       = scrapyd.webservice.Cancel
addversion.json   = scrapyd.webservice.AddVersion
listprojects.json = scrapyd.webservice.ListProjects
listversions.json = scrapyd.webservice.ListVersions
listspiders.json  = scrapyd.webservice.ListSpiders
delproject.json   = scrapyd.webservice.DeleteProject
delversion.json   = scrapyd.webservice.DeleteVersion
listjobs.json     = scrapyd.webservice.ListJobs
daemonstatus.json = scrapyd.webservice.DaemonStatus
test_endpoint.json     = extra_endpoints.create_spider.TestNewEndpoint

[vazkir]

Only now I am running into the problem of that I want to reload scrapyd so it picks up the new SPIDER_MODULES, without fully cancelling any jobs it might still have been running. Do you by any chance know of the best way to handle this?

I could only find this SO post, where multiple options were given, only not sure which ones would be best suited?

[jpmckinney]

Hmm, I haven't tested changing SPIDER_MODULES specifically, but I would have expected this workflow to work, without requiring a reload of the scrapyd service:

• Edit SPIDER_MODULES in your Scrapy project's settings.py
• Send a request to the addversion.json endpoint, with the updated project that changed SPIDER_MODULES
• Send a request to the schedule.json endpoint, with the new version