search

hercules-ci / support

User feedback, questions and our public roadmap. help@hercules-ci.com
5 stars 1 forks source link

Job stuck in "import from derivation" #30

Open blitz opened 4 years ago

blitz commented 4 years ago

Hi,

I have a job [1] that seems to be stuck in "Import from derivation" for 24h. Restarting the agent has had no effect. The logs from the restarted daemon are

Oct 30 00:27:25 first-temple hercules-ci-agent[3372]: [2019-10-29 23:27:25][][Info][first-temple][3372][ThreadId 44][task:6c225c50-0ff3-4254-8ae3-3b2da0a286e8][task-type:eval][worker:3433][agent-version:0.5.0][main:Hercules.Agent.Evaluate hercules-ci-agent/Hercules/Agent/Evaluate.hs:416:7] Evaluator: Building "/nix/store/s2ri5n5kf97n2yrly82gg7y10jm6dicb-haskell-lib.drv!out"
Oct 30 00:27:25 first-temple hercules-ci-agent[3372]: [2019-10-29 23:27:25][][Debug][first-temple][3372][ThreadId 49][cache:blitz][task:6c225c50-0ff3-4254-8ae3-3b2da0a286e8][task-type:eval][agent-version:0.5.0][derivation:/nix/store/s2ri5n5kf97n2yrly82gg7y10jm6dicb-haskell-lib.drv][main:Hercules.Agent.Evaluate hercules-ci-agent/Hercules/Agent/Evaluate.hs:281:64] Pushing ifd drvs to cachix
Oct 30 00:27:25 first-temple hercules-ci-agent[3372]: [2019-10-29 23:27:25][][Info][first-temple][3372][ThreadId 44][task:6c225c50-0ff3-4254-8ae3-3b2da0a286e8][task-type:eval][worker:3433][agent-version:0.5.0][main:Hercules.Agent.Evaluate hercules-ci-agent/Hercules/Agent/Evaluate.hs:416:7] Evaluator: Awaiting "/nix/store/s2ri5n5kf97n2yrly82gg7y10jm6dicb-haskell-lib.drv"

The agent is idle and there is no further output from the hercules-ci service. Any advice is appreciated!

[1] https://hercules-ci.com/github/blitz/obiwan/jobs/26

roberth commented 4 years ago

Hi blitz,

Import from derivation requires the ability to run >1 concurrent tasks in your cluster. In your case of a single machine, you need to set

services.hercules-ci-agent.concurrentTasks = 2; # nixos/nixops/nix-darwin

or

concurrentTasks = 2  # manual deployment with toml file

Thank you for reporting this. We will clarify this in the documentation and in the dashboard.

blitz commented 4 years ago

Makes sense. Increasing concurrentTasks has unstuck my pipeline. The update to the dashboard to make this condition more visible is much appreciated.

domenkozar commented 4 years ago

Let's keep this open until it's fixed.