q-m / scrapyd-k8s

Scrapyd on container infrastructure
MIT License
12 stars 7 forks source link

Expose logs #12

Open wvengen opened 9 months ago

wvengen commented 9 months ago

Scrapyd's listjobs endpoint shows a logs path for finished jobs (as seen here):

{
    "status": "ok",
    // ...
    "finished": [
        {
            "id": "2f16646cfcaf11e1b0090800272a6d06",
            "project": "myproject", "spider": "spider3",
            "start_time": "2012-09-12 10:14:03.594664",
            "end_time": "2012-09-12 10:24:03.594664",
            "log_url": "/logs/myproject/spider3/2f16646cfcaf11e1b0090800272a6d06.log",
            "items_url": "/items/myproject/spider3/2f16646cfcaf11e1b0090800272a6d06.jl"
        }
    ]
}

We don't have this yet. First add serving logs to the API (just plaintext output, the container log messages, streaming if you manage to do it). Then add the extra entry to the listjobs endpoint for finished jobs.