Closed igarny closed 6 years ago
I have found another example ... different from the above. Despite the successful submission of the task pscheduler fails with a strange error. This results to no measurements started, which again leads to the reported problem: "no additional attempts are made to sustain the latencybg tests running"
Apr 9 03:25:33 psmp-gn-bw-02-fra-de runner INFO 626037: With powstream: latencybg --dest psmp-gn-owd-01-ham-de-v4.geant.net --packet-padding 0 --source psmp-gn-owd-02.fra.de.geant.net --ip-version 4 --packet-interval 0.1 --duration PT86400S --packet-count 1200
Apr 9 03:25:33 psmp-gn-bw-02-fra-de runner ERROR 626037: Failed to set running state: Invalid transition between states (5 to 3).
Both of your comments are two separate issues. Probably need @mfeit for both of them and may need some pscheduler issues open instead of this one:
mesh-config actually does try to detect when a latencybg fails and recreate it. latencybg tasks can fail a little differently than others, so one special thing it checks is that after the start time of a background-multi task plus some delta, if there is only 1 run, it assumes the task failed. Since it doesn't know anything else about how to read tests, like how frequent a task should be creating runs this is about the best it can do, but should capture most of the failure cases and correct things within about an hour as opposed to 24 hours. It determines this by looking at the task JSON's detail
section's runs
field. Notice that in the task I shared above runs is actually set to 2. What's strange is if you go to https://psmp-gn-owd-02.fra.de.geant.net/pscheduler/tasks/3aa993ef-600d-49c0-a630-10bdf4e3cd6b/runs, there is only 1 run listed. It seems like that the value provided in "runs" is incorrect. I'm wondering if maybe the runs are getting miscounted by pScheduler? I am actually able to see this is a couple ESnet tasks this morning as well, so its likely a more general issue.
A helpful query for finding tasks in the state where "runs" says 2 but if you look at the runs list it only shows one is as follows: https://HOSTNAME/pscheduler/tasks?pretty&expanded&detail&json={%22test%22:{%22type%22:%22latencybg%22},"detail":{"runs":2}}
Casey Russell is also seeing the run count issue: https://ps-psu-lt.perfsonar.kanren.net/pscheduler/tasks?pretty&expanded&detail&json={%22test%22:{%22type%22:%22latencybg%22},%22detail%22:{%22runs%22:2}}
Mark added a new field to help detect stuck processes so they don't hang around for 24 hours. I have tested pretty extensively and have seen this work in production so going to close for now. Fix is part of 4.1.
Hello Andy,
I am observing these for some time, but kind of now have found a way to report them.
From the pscheduler log: the problem:
the running:
No direct errors observed in:
Again the issue:
The meshconfig events:
My hope is to have this automation in pSconfig, that would identify missing measurement runs of latencybg and attempt to restart them on the next configuration re-read.
Should I open a ticket in pSconfig?