q-m / scrapyd-k8s

Scrapyd on container infrastructure
MIT License
12 stars 7 forks source link

Add start_time and end_time in listjobs.json #11

Open wvengen opened 9 months ago

wvengen commented 9 months ago

For running jobs returned by the listjobs endpoint, add start_time. For finished jobs returned by the listjobs endpoint, add start_time and end_time.

See https://scrapyd.readthedocs.io/en/latest/api.html#listjobs-json for an example output:

{
    "status": "ok",
    "pending": [
        {
            "project": "myproject", "spider": "spider1",
            "id": "78391cc0fcaf11e1b0090800272a6d06"
        }
    ],
    "running": [
        {
            "id": "422e608f9f28cef127b3d5ef93fe9399",
            "project": "myproject", "spider": "spider2",
            "start_time": "2012-09-12 10:14:03.594664"
        }
    ],
    "finished": [
        {
            "id": "2f16646cfcaf11e1b0090800272a6d06",
            "project": "myproject", "spider": "spider3",
            "start_time": "2012-09-12 10:14:03.594664",
            "end_time": "2012-09-12 10:24:03.594664",
            "log_url": "/logs/myproject/spider3/2f16646cfcaf11e1b0090800272a6d06.log",
            "items_url": "/items/myproject/spider3/2f16646cfcaf11e1b0090800272a6d06.jl"
        }
    ]
}

(we'll omit items_url, and log_url has #12)

wvengen commented 9 months ago

Note that we'd want to get these dates from the container info. For Docker, try docker inspect <jobname> and you'll see StartedAt and FinishedAt in the State. For Kubernetes, try kubectl describe job <jobname> and you'll see Start Time and Completed At.

Make sure to stick to the date format as in the example.

RoelAdriaans commented 8 months ago

Worked on by Roel and Timmy