search

flux-framework / flux-core

core services for the Flux resource management framework
GNU Lesser General Public License v3.0
167 stars 50 forks source link

apparent hello storm after scheduler reloaded #6294

Open grondo opened 2 weeks ago

grondo commented 2 weeks ago

In the logs captured below, the scheduler was apparently loadedat 10:58, then we see the messages:

[ +45.189807] job-manager[0]: alloc: stop due to disconnect: Success
[ +47.946744] sched-fluxion-resource[0]: disconnect_request_cb: a notify request aborted

Followed by what appears to be a hello storm, with at least 35 hello requests sent to the job manager if the logs can be believed. Full logs here:

[Sep19 10:58] job-manager[0]: scheduler: hello
[Sep19 10:59] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[  +3.152109] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[  +4.710717] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[  +6.724351] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[  +7.308041] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[  +7.921133] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +7.955663] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +7.983040] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.007288] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +8.030388] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.070941] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.073458] job-manager[0]: scheduler: ready unlimited
[Sep19 11:00] job-exec[0]: job-exception: id=f7fZxjpetmd: resource allocation expired
[  +5.002067] job-exec[0]: Sending SIGKILL to job f7fZxjpetmd
[  +5.972579] job-manager[0]: housekeeping: f7fZxjpetmd started
[ +45.189807] job-manager[0]: alloc: stop due to disconnect: Success
[ +47.947512] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947585] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947646] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947707] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947767] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947827] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.946744] sched-fluxion-resource[0]: disconnect_request_cb: a notify request aborted
[ +47.947885] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.947947] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948005] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948066] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948124] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948185] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948244] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948301] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948358] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948428] sched-fluxion-qmanager[0]: jobmanager_cancel_cb: queue not found for job (id=49532748996380672): No such file or directory
[ +47.948643] job-manager[0]: scheduler: hello
[ +47.951330] job-manager[0]: scheduler: hello
[ +47.951430] job-manager[0]: scheduler: hello
[ +47.954927] job-manager[0]: scheduler: hello
[ +47.955021] job-manager[0]: scheduler: hello
[ +47.955696] job-manager[0]: scheduler: hello
[ +47.955788] job-manager[0]: scheduler: hello
[ +47.955884] job-manager[0]: scheduler: hello
[ +47.957521] job-manager[0]: scheduler: hello
[ +47.957697] job-manager[0]: scheduler: hello
[ +47.957812] job-manager[0]: scheduler: hello
[ +47.957880] job-manager[0]: scheduler: hello
[ +47.957952] job-manager[0]: scheduler: hello
[ +47.959617] job-manager[0]: scheduler: hello
[ +47.959925] job-manager[0]: scheduler: hello
[ +47.959999] job-manager[0]: scheduler: hello
[ +47.960738] job-manager[0]: scheduler: hello
[ +47.960803] job-manager[0]: scheduler: hello
[ +47.961847] job-manager[0]: scheduler: hello
[ +47.963804] job-manager[0]: scheduler: hello
[ +47.966084] job-manager[0]: scheduler: hello
[ +47.966188] job-manager[0]: scheduler: hello
[ +47.966276] job-manager[0]: scheduler: hello
[ +47.966767] job-manager[0]: scheduler: hello
[ +47.966838] job-manager[0]: scheduler: hello
[ +47.966899] job-manager[0]: scheduler: hello
[ +47.966968] job-manager[0]: scheduler: hello
[ +47.967031] job-manager[0]: scheduler: hello
[ +47.967099] job-manager[0]: scheduler: hello
[ +47.967167] job-manager[0]: scheduler: hello
[ +47.967233] job-manager[0]: scheduler: hello
[ +47.967301] job-manager[0]: scheduler: hello
[ +49.118986] job-manager[0]: housekeeping: f7fZxjpetmd complete
[ +50.025627] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.025716] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.025807] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.025895] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.025974] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026049] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026120] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026193] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026266] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026342] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026429] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026503] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026577] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026645] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026712] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026788] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026855] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.026925] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027000] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027072] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027143] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027210] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027277] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027345] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027424] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027492] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027580] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027650] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027717] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027795] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027871] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +50.027949] sched-fluxion-resource[0]: update_request_cb: jobid (49434508833686528) with matching R exists
[ +53.302036] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.302466] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.302844] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.303211] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.303579] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.303934] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.304288] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.304655] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.305011] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.305362] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.305725] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.306081] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.306445] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.306798] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.307152] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.307525] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.308063] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.308641] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.309197] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.309775] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.310351] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.310929] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.311496] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.312057] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.312585] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.313087] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.313588] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.314077] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.314579] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.315070] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.315597] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.316095] sched-fluxion-resource[0]: update_request_cb: jobid (49492102835177472) with matching R exists
[ +53.316235] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316296] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316340] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316386] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316436] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316477] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316521] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316577] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316631] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316676] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316719] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316761] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316804] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316845] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316884] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316931] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.316977] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.317008] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.318064] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.319966] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.320115] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.320217] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.320318] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.320426] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.321449] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.321961] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.322055] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.322718] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.322821] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.323029] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.323104] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.323179] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.323989] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.324272] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324352] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324428] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324505] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324563] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324618] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324672] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324729] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324849] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.324915] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.325133] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.325241] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.325478] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.325661] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.325724] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.326122] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.326534] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.326623] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.327162] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.327469] sched-fluxion-resource[0]: update_request_cb: jobid (49401880554341376) with matching R exists
[ +53.327532] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.328196] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.328262] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.328430] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.329002] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.329222] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.329287] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.329426] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.329559] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.329767] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.330115] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.330183] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331042] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.331280] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331348] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331478] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331548] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331620] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331686] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.331746] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332003] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332069] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.332169] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332260] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332398] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332468] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332530] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[ +53.332586] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.332644] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.333515] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.334151] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.334217] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.334273] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.334524] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.335621] sched-fluxion-resource[0]: update_request_cb: jobid (48377370145293312) with matching R exists
[ +53.335686] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.335804] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.336159] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[ +53.336246] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.336311] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.336398] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.337001] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.337070] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.337345] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.338276] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.338789] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.339201] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.339449] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[ +53.339586] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.339749] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.339935] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340046] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340158] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340266] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340365] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340472] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340653] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340756] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340879] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.340986] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.341142] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.341515] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.341694] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.341950] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.342157] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.342504] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[ +53.342817] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.342928] sched-fluxion-resource[0]: update_request_cb: jobid (49410619017628672) with matching R exists
[ +53.343284] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.343390] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.343495] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.343946] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.344426] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.344533] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.344797] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.345288] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.345397] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.345504] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.345598] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.345695] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.345792] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.345889] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.345982] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346080] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.346175] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346267] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346360] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346471] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346640] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.346971] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.347073] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.347180] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.347396] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.347519] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.348542] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[ +53.348841] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[ +53.348943] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.349180] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[ +53.349286] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[Sep19 11:01] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.000405] job-manager[0]: scheduler: ready unlimited
[  +0.000519] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.000628] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.000897] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.001004] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.001327] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[  +0.001453] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.001555] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.001895] sched-fluxion-resource[0]: update_request_cb: jobid (49552984399250432) with matching R exists
[  +0.002004] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.002106] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.002205] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.003214] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.003340] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.003643] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.003799] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.004314] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.004666] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.004798] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.004902] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005005] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005105] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005204] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005310] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.005692] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.006067] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.006316] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.006819] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.007014] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.007552] sched-fluxion-resource[0]: update_request_cb: jobid (49532439171531776) with matching R exists
[  +0.007636] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.007826] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.007911] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.008208] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.008371] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.009506] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.009700] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.009844] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.010418] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.010652] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.010765] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.010939] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.011246] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +0.011308] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.011630] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +0.011898] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821248] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821317] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +2.821400] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821448] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821503] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821548] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821594] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821645] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +2.821689] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821733] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821777] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.821822] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821865] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821907] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.821948] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822005] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +2.822059] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822102] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822146] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822203] sched-fluxion-resource[0]: update_request_cb: jobid (48706744979653632) with matching R exists
[  +2.822257] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822299] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822342] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822393] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822439] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +2.822626] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822670] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822711] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822752] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822796] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +2.822838] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.298808] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305264] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305345] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305414] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305469] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305520] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305584] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305641] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305692] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.305770] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305816] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.305864] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.305910] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.305956] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306003] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.306046] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306094] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306191] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.306252] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.306436] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306488] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.306539] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.306584] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306665] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306710] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306756] sched-fluxion-resource[0]: update_request_cb: jobid (48522298464764928) with matching R exists
[  +6.306812] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306895] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +6.306947] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.306993] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +6.307036] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.517900] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.518481] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.518645] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.519121] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.519254] job-manager[0]: scheduler: ready unlimited
[  +8.519692] job-manager[0]: scheduler: ready unlimited
[  +8.519710] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.520158] job-manager[0]: scheduler: ready unlimited
[  +8.520277] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.520435] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520497] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520552] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520621] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520678] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520715] job-manager[0]: scheduler: ready unlimited
[  +8.520734] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520788] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520847] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.520903] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.520953] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.521019] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521075] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521130] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521186] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521246] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521324] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521416] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521511] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.521840] job-manager[0]: scheduler: ready unlimited
[  +8.522091] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +8.522286] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.522370] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.522463] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.522533] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.522608] sched-fluxion-resource[0]: update_request_cb: jobid (48523417085641728) with matching R exists
[  +8.522673] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +8.523446] job-manager[0]: scheduler: ready unlimited
[  +9.146982] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.147564] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.148109] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.148350] job-manager[0]: scheduler: ready unlimited
[  +9.148679] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.148692] job-manager[0]: scheduler: ready unlimited
[  +9.149229] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.149329] job-manager[0]: scheduler: ready unlimited
[  +9.149525] job-manager[0]: scheduler: ready unlimited
[  +9.149776] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.150314] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.150688] job-manager[0]: scheduler: ready unlimited
[  +9.150871] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.150927] job-manager[0]: scheduler: ready unlimited
[  +9.151269] job-manager[0]: scheduler: ready unlimited
[  +9.151416] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.151951] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.152182] job-manager[0]: scheduler: ready unlimited
[  +9.152493] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.152632] job-manager[0]: scheduler: ready unlimited
[  +9.152942] job-manager[0]: scheduler: ready unlimited
[  +9.153028] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.153570] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.154103] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.154109] job-manager[0]: scheduler: ready unlimited
[  +9.154149] job-manager[0]: scheduler: ready unlimited
[  +9.154607] job-manager[0]: scheduler: ready unlimited
[  +9.154646] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.155067] job-manager[0]: scheduler: ready unlimited
[  +9.155179] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.155720] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.155751] job-manager[0]: scheduler: ready unlimited
[  +9.156252] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.156793] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.156824] job-manager[0]: scheduler: ready unlimited
[  +9.156936] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +9.157030] job-manager[0]: scheduler: ready unlimited
[  +9.157427] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.157520] job-manager[0]: scheduler: ready unlimited
[  +9.157578] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +9.157942] job-manager[0]: scheduler: ready unlimited
[  +9.158059] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.158200] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +9.158268] sched-fluxion-resource[0]: update_request_cb: jobid (49432290516632576) with matching R exists
[  +9.158586] job-manager[0]: scheduler: ready unlimited
[  +9.159212] job-manager[0]: scheduler: ready unlimited
[  +9.736240] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.736825] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.737290] job-manager[0]: scheduler: ready unlimited
[  +9.737377] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.737772] job-manager[0]: scheduler: ready unlimited
[  +9.737934] sched-fluxion-resource[0]: update_request_cb: jobid (49505126920858624) with matching R exists
[  +9.738474] job-manager[0]: scheduler: ready unlimited
[  +9.739052] job-manager[0]: scheduler: ready unlimited
garlick commented 2 weeks ago

@trws - is there any way the exception wrapper fluxion puts around mod_main() could be causing it to be called repeatedly or recursively? That woud be

https://github.com/flux-framework/flux-sched/blob/master/src/common/c%2B%2Bwrappers/eh_wrapper.hpp

which is way over my head.

trws commented 2 weeks ago

It certainly shouldn't be. The passed function object is called only once. If there's a way the call from the event loop can be retried, like if it's not reset or something, then it's possible an exception could be causing it to exit before hitting that point and the event loop is re-calling it but it would have to be something like that.