NixOS / hydra

Hydra, the Nix-based continuous build system
http://nixos.org/hydra
GNU General Public License v3.0
1.19k stars 303 forks source link

Default GitInput timeout is too low for nixpkgs #1181

Open JJJollyjim opened 2 years ago

JJJollyjim commented 2 years ago

Describe the bug

By default, git commands run by GitInput.pm time out after 600 seconds. This is not enough time to perform the first fetch of nixpkgs (2.19GiB) on a VDSL (~15mbps) connection.

Additionally, the timeout doesn't seem to be documented outside of the GitInput.pm file itself, and it is hard to find from the error message (which is just the word "timeout") creating a pretty poor experience for first-time users.

Even on my gigabit connection, I only get 80mbps from GitHub*, which means a fresh nixpkgs clone takes 329.71s wall time to transfer the necessary 2.19GiB. This provides very little wiggle room for nixpkgs to get larger or for GitHub to get slower.

*Not sure if this is a universal experience, or if GitHub just has poor geo-distribution and it's only an issue for non-US people.

https://github.com/NixOS/hydra/blob/01fb23ddf647ee7b6dc3abc1f012eaf9f5306c88/src/lib/Hydra/Plugin/GitInput.pm#L80-L83

Happy to PR bumping the timeout (3600s?) if that's an acceptable approach. An analogue of nix's max-silent-time might be a more elegant solution if the goal is to detect hanging/stuck checkouts, though.

To Reproduce Steps to reproduce the behavior:

  1. Limit internet speed to 20mbps, e.g. tc qdisc add dev eth0 root tbf rate 20000kbit burst 20000
  2. Create a jobset with a git input of nixpkgs
  3. Observe that evaluation fails

Logs

Mar 23 08:06:48 nixos hydra-evaluator[79757]: error fetching latest change from git repo at `https://github.com/nixos/nixpkgs.git':
Mar 23 08:06:48 nixos hydra-evaluator[79757]: timeout

Hydra Server:

crawford commented 2 years ago

This can be worked around for now using the following configuration snippet:

{
  services.hydra.extraConfig = ''
    <git-input>
      timeout = 3600
    </git-input>
  '';
}