Swirrl / drafter

A clojure service and a client to it for exposing data management operations to PMD
Other
0 stars 0 forks source link

Job start time changes when it completes #647

Closed RickMoynihan closed 1 year ago

RickMoynihan commented 1 year ago

This could be a drafter bug; but I'll report here first pending investigation.

ONS asked us to investigate some stalled jobs. I looked in their admin panel and saw the following:

Screenshot 2022-10-18 at 15 12 57

Note that this appears to indicate an append to GSS data/ClimateChange/csvcubed/BEIS-2005-to-2020-local-authority-greenhouse-gas-emissions/upload-to-pnd followed by two publish operations (which look like they're on the same draftset - this in itself is slightly strange; but I asked Shannon at ONS who did this and it seems like it was an accident caused by inadverdently using two browser sessions. So I don't think this is relevant to this issue).

So finally the append finished and then the publish finished, and the completed jobs screen then looked like this:

Screenshot 2022-10-18 at 15 59 42

Note that the start times of the append job has now changed and that the ordering is different. I suspect this is because internally the jobs are split into "joblets" and that the start time for them is their processing time rather than their submission time; so we're seeing the start time be the time the last joblet for that append started.