LSSTDESC / desc-gen3-prod

Desc-prod wrapper for pipeline production using gen3_workflow.
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Task state counts reported by runapp_g3wfpipe are inaccurate #12

Open dladams opened 8 months ago

dladams commented 8 months ago

I see small discrepancies between the task state counts reported by g3wfpipe and those found by the monexp notebook.

E.g. in run 1930, progress.log contains

...
      32: Running. Found parsl pickle file.
      42: Running. Finished 0 of 2985 tasks. 2985 pending.
      52: Running. Finished 0 of 2985 tasks. 2910 pending. 11 launched. 64 running.
      63: Running. Finished 0 of 2985 tasks. 2910 pending. 11 launched. 64 running.
      73: Running. Finished 0 of 2985 tasks. 2910 pending. 11 launched. 64 running.
      84: Running. Finished 0 of 2985 tasks. 2910 pending. 11 launched. 64 running.
      94: Running. Finished 0 of 2985 tasks. 2910 pending. 11 launched. 64 running.
     104: Running. Finished 0 of 2985 tasks. 2910 pending. 11 launched. 64 running.
...

While the isr curve in the monexp "running Task Count" plot has these values:

0      0.000000
1      0.000000
2      0.000000
3      0.000000
4      0.000000
5      0.000000
6      0.000000
7     44.109747
8     75.000000
9     75.000000
10    75.000000
11    75.000000
12    75.000000

in its first few monitoring lines and the recently updated monexp notebook show 117 running isr tasks at the start of then job. The time stamps in the isr stderr logs agree with the latter value, i.e. 117 tasks start in the first few seconds.