Investigate hydra-queue-runner S3 downloads

edolstra commented 8 years ago

Need to figure out why this happens:

Mar 09 18:57:48 chef hydra-queue-runner[11303]: uploaded ‘s3://nix-cache/nar/190m0683g4c5gqsy69ixc0hqm53kgblj2y6dj9cracf4k8vys65k.nar.xz’ (426701500 bytes) in 83605 ms
...
Mar 09 18:58:08 chef hydra-queue-runner[11303]: fetching ‘s3://nix-cache/nar/190m0683g4c5gqsy69ixc0hqm53kgblj2y6dj9cracf4k8vys65k.nar.xz’...
Mar 09 19:00:34 chef hydra-queue-runner[11303]: downloaded ‘s3://nix-cache/nar/190m0683g4c5gqsy69ixc0hqm53kgblj2y6dj9cracf4k8vys65k.nar.xz’ (426701500 bytes) in 145942 ms
...
Mar 09 19:00:41 chef hydra-queue-runner[11303]: exporting path ‘/nix/store/8w4bmwqk4cpg8zi43lmzz35jqki500fw-iso9660-image’ (439354208 bytes)
...
Mar 10 03:57:47 chef hydra-queue-runner[11303]: loading build 33064542 (nixos:release-16.03-small:nixos.iso_minimal.x86_64-linux)
Mar 10 03:57:47 chef hydra-queue-runner[11303]: fetching ‘s3://nix-cache/8w4bmwqk4cpg8zi43lmzz35jqki500fw.narinfo’...
Mar 10 03:57:47 chef hydra-queue-runner[11303]: downloaded ‘s3://nix-cache/8w4bmwqk4cpg8zi43lmzz35jqki500fw.narinfo’ (713 bytes) in 126 ms
...
Mar 10 04:05:10 chef hydra-queue-runner[11303]: exporting path ‘/nix/store/8w4bmwqk4cpg8zi43lmzz35jqki500fw-iso9660-image’ (439354208 bytes)

The first download is probably due to caching of negative lookups in Cloudfront, causing a build machine to be unable to fetch the path directly. This can be fixed by having the build machines access the underlying S3 bucket directly, rather than Cloudfront. (This would also save some money for EC2 build machines.)

The second download is probably due to the queue monitor thread encountering a new build of a cached path, requiring it to fetch the path to determine the build products. This could probably be avoided by copying the build products from the previous build in the Hydra database.

edolstra commented 8 years ago

Queue runner status after 76,000s of uptime:

      "s3" : {
         "costDollarApprox" : 12.7399,
         "get" : 346118,
         "getBytes" : 143014408725,
         "getSpeed" : 3.77933,
         "getTimeMs" : 36088182,
         "head" : 117607,
         "put" : 113421,
         "putBytes" : 389521856099,
         "putSpeed" : 2.44061,
         "putTimeMs" : 152206399
      }

domenkozar commented 8 years ago

So this is fixed?

NixOS / hydra

Investigate hydra-queue-runner S3 downloads #276