search

hercules-ci / hercules-ci-agent

https://hercules-ci.com build and deployment agent
Apache License 2.0
99 stars 19 forks source link

Crash on service stop #417

Closed brainrake closed 2 years ago

brainrake commented 2 years ago

Description

Stopping the agent service crashes the agent.

To Reproduce

systemctl stop hercules-ci-agnent-...

Expected behavior

clean shutdown

Logs

Apr 18 21:33:34 ci systemd[1]: Stopping hercules-ci-agent-mlabs-haskell.service...
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232886][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:395:5] Syncing
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232780][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:395:5] Syncing
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232686][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:395:5] Syncing
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232831][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:395:5] Syncing
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1233047][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:395:5] Syncing
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232735][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:395:5] Syncing
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232942][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:395:5] Syncing
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232996][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:395:5] Syncing
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232831][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:399:5] Logger done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232831][ThreadId 14][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:204:30] runner done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1233047][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:399:5] Logger done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1233047][ThreadId 14][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:204:30] runner done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232942][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:399:5] Logger done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][][Info][ci][PID 1232656][ThreadId 78][agent-version:0.9.3][task:bf468818-a62e-45d6-833e-a1482e9d38c5][worker:1232831][main:Hercules.Agent.Log hercules-ci-agent/Hercules/Agent/Log.hs:42:5] evaluator: hercules-ci-agent-worker: zdmainzdHerculesziAgentziWorkerziHerculesStorezdHerculeszziAgentzziWorkerzziHerculesStorezumkBuilderCallback: interrupted
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][][Info][ci][PID 1232656][ThreadId 120][agent-version:0.9.3][task:f920245d-0053-42bb-bcb1-a77a1a613b0c][worker:1233047][main:Hercules.Agent.Log hercules-ci-agent/Hercules/Agent/Log.hs:42:5] evaluator: hercules-ci-agent-worker: zdmainzdHerculesziAgentziWorkerziHerculesStorezdHerculeszziAgentzziWorkerzziHerculesStorezumkBuilderCallback: interrupted
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232942][ThreadId 14][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:204:30] runner done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][][Info][ci][PID 1232656][ThreadId 99][agent-version:0.9.3][task:8a5a5503-6324-42a8-9b77-da9a439fc1a1][worker:1232942][main:Hercules.Agent.Log hercules-ci-agent/Hercules/Agent/Log.hs:42:5] evaluator: hercules-ci-agent-worker: zdmainzdHerculesziAgentziWorkerziHerculesStorezdHerculeszziAgentzziWorkerzziHerculesStorezumkBuilderCallback: interrupted
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232780][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:399:5] Logger done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232780][ThreadId 14][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:204:30] runner done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][][Info][ci][PID 1232656][ThreadId 67][agent-version:0.9.3][task:feb7c4ee-dcf8-428b-add9-69033fb56738][worker:1232780][main:Hercules.Agent.Log hercules-ci-agent/Hercules/Agent/Log.hs:42:5] evaluator: hercules-ci-agent-worker: zdmainzdHerculesziAgentziWorkerziHerculesStorezdHerculeszziAgentzziWorkerzziHerculesStorezumkBuilderCallback: interrupted
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232886][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:399:5] Logger done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232886][ThreadId 14][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:204:30] runner done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][][Info][ci][PID 1232656][ThreadId 88][agent-version:0.9.3][task:a9b32eeb-4a9f-4c0a-90a1-cae379a589c6][worker:1232886][main:Hercules.Agent.Log hercules-ci-agent/Hercules/Agent/Log.hs:42:5] evaluator: hercules-ci-agent-worker: zdmainzdHerculesziAgentziWorkerziHerculesStorezdHerculeszziAgentzziWorkerzziHerculesStorezumkBuilderCallback: interrupted
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232996][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:399:5] Logger done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232996][ThreadId 14][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:204:30] runner done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][][Info][ci][PID 1232656][ThreadId 109][agent-version:0.9.3][task:865a9b1d-acb4-4903-bb57-9cc0f568b6a5][worker:1232996][main:Hercules.Agent.Log hercules-ci-agent/Hercules/Agent/Log.hs:42:5] evaluator: hercules-ci-agent-worker: zdmainzdHerculesziAgentziWorkerziHerculesStorezdHerculeszziAgentzziWorkerzziHerculesStorezumkBuilderCallback: interrupted
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232686][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:399:5] Logger done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232686][ThreadId 14][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:204:30] runner done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][][Info][ci][PID 1232656][ThreadId 45][agent-version:0.9.3][task:38d7e86f-d693-476d-bd8e-69bff1f2f8e5][worker:1232686][main:Hercules.Agent.Log hercules-ci-agent/Hercules/Agent/Log.hs:42:5] evaluator: hercules-ci-agent-worker: zdmainzdHerculesziAgentziWorkerziHerculesStorezdHerculeszziAgentzziWorkerzziHerculesStorezumkBuilderCallback: interrupted
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232735][ThreadId 19][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:399:5] Logger done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][Worker][Debug][ci][PID 1232735][ThreadId 14][main:Hercules.Agent.Worker hercules-ci-agent-worker/Hercules/Agent/Worker.hs:204:30] runner done
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: [2022-04-18 21:33:34][][Info][ci][PID 1232656][ThreadId 56][agent-version:0.9.3][task:fdaa573b-fb26-4ba0-8839-04407b570818][worker:1232735][main:Hercules.Agent.Log hercules-ci-agent/Hercules/Agent/Log.hs:42:5] evaluator: hercules-ci-agent-worker: zdmainzdHerculesziAgentziWorkerziHerculesStorezdHerculeszziAgentzziWorkerzziHerculesStorezumkBuilderCallback: interrupted
Apr 18 21:33:34 ci hercules-ci-agent[1232656]: NixInfo {nixLibVersion = "2.7.0", nixPlatforms = ["i686-linux","x86_64-linux","x86_64-v1-linux","x86_64-v2-linux","x86_64-v3-linux"], nixSystemFeatures = ["benchmark","big-parallel","kvm","nixos-test"], nixSubstituters = ["https://public-plutonomicon.cachix.org","https://hercules-ci.cachix.org","https://iohk.cachix.org","https://cache.nixos.org/","https://cache.nixos.org/"], nixTrustedPublicKeys = ["cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=","public-plutonomicon.cachix.org-1:3AKJMhCLn32gri1drGuaZmFrmnue+KkKrhhubQk/CWc=","hercules-ci.cachix.org-1:ZZeDl9Va+xe9j+KqdzoBZMFJHVQ42Uu/c/1/KMC5Lw0=","iohk.cachix.org-1:DpRUyj7h7V830dp/i6Nti+NEO2/nhblbov/8MW7Rqoo=","cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY="], nixNarinfoCacheNegativeTTL = 0, nixNetrcFile = Just "/etc/nix/netrc"}
Apr 18 21:33:35 ci systemd-coredump[1275038]: [🡕] Process 1232831 (hercules-ci-age) of user 996 dumped core.
Apr 18 21:33:35 ci systemd-coredump[1275048]: [🡕] Process 1232735 (hercules-ci-age) of user 996 dumped core.
Apr 18 21:33:35 ci systemd-coredump[1275044]: [🡕] Process 1232886 (hercules-ci-age) of user 996 dumped core.
Apr 18 21:33:35 ci systemd-coredump[1275040]: [🡕] Process 1233047 (hercules-ci-age) of user 996 dumped core.
Apr 18 21:33:35 ci systemd-coredump[1275041]: [🡕] Process 1232942 (hercules-ci-age) of user 996 dumped core.
Apr 18 21:33:35 ci systemd-coredump[1275043]: [🡕] Process 1232780 (hercules-ci-age) of user 996 dumped core.
Apr 18 21:33:35 ci systemd-coredump[1275046]: [🡕] Process 1232996 (hercules-ci-age) of user 996 dumped core.
Apr 18 21:33:35 ci systemd-coredump[1275047]: [🡕] Process 1232686 (hercules-ci-age) of user 996 dumped core.
Apr 18 21:33:35 ci systemd[1]: hercules-ci-agent-mlabs-haskell.service: Deactivated successfully.
Apr 18 21:33:35 ci systemd[1]: Stopped hercules-ci-agent-mlabs-haskell.service.

Platform / Version NixOS unstable, hercules-ci-agent 0.9.3 rev 56a8925fb519ce25a4ff8376ab76607a7d1b49ec, multi-agent nixos module via flake

roberth commented 2 years ago

Is it possible to get stack traces from one of these? For example:

agent# nix-shell -p gdb
nix-shell# coredumpctl debug 1232686
(gdb) thread apply all bt
brainrake commented 2 years ago

gdb.txt