input-output-hk / iohk-nix

nix scripts shared across projects
62 stars 28 forks source link

Strange cache misses #237

Closed deepfire closed 4 years ago

deepfire commented 4 years ago

Some pretty weird cache misses in the current tip of https://github.com/input-output-hk/cardano-node/tree/serge/mainnet-ci (commit 8b6f615506f2de42c339f16aa1f5fe6fbcd0c808):

  1. Caches filled on Hydra: https://hydra.iohk.io/build/1377499 (note the .drv: /nix/store/9ihjzh425p2s3wrk06767amclc9ab8an-vm-test-run-chairmans-cluster-test.drv)

  2. The attrbute evaluates locally to the same .drv:

$ nix-instantiate -A nixosTests.chairmansCluster
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/9ihjzh425p2s3wrk06767amclc9ab8an-vm-test-run-chairmans-cluster-test.drv
  1. Yet Nix fails at caching:
$ nix-build -A nixosTests.chairmansCluster
these derivations will be built:
  /nix/store/cqad1a89vy2lqsv963vj087clm8ax876-ghc-8.4.4.drv
  /nix/store/x4qg7x969vb7lwiay7idcmll820vwlz9-default-Setup.drv
  /nix/store/j2gxy6rw6rmfrrf8mb59wpvckllv7xpj-bytestring-0.10.8.2-lib-bytestring.drv
  /nix/store/md70yw3ghhgla0ihkga015fbhcgy9ylf-containers-0.5.11.0-lib-containers.drv
...
don't know how to build these paths:
  /nix/store/0kah289565r372gnyw5r426h418hxjiw-yaml-0.11.0.0-lib-yaml
  /nix/store/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz-ghc-8.4.4
  /nix/store/9m173vkvd6r8i706nykyvxfhlqjlhzb8-ghc-8.4.4-doc
  /nix/store/bhnwnvrv94q1lm072v68wswyc1gkw438-nonempty-containers-0.1.1.0-lib-nonempty-containers
  /nix/store/dihzx42kyjh51hgbv582775qnij2mgi2-conduit-1.3.1.1-lib-conduit
  /nix/store/iywqv861qhkrii08ypk3ysgbfaqzrn4p-libyaml-0.1.1.0-lib-libyaml
  /nix/store/vxhvbfjqzpf1gzi2vikczzsc8bl95pd1-mono-traversable-1.0.11.0-lib-mono-traversable

..and note the failure to build the paths.

  1. The particular GHC derivation is cached:
$ curl https://hydra.iohk.io/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
  1. The caches are configured (& mostly hit for other .drvs):
$ nix show-config | grep 'substituters '
extra-substituters = 
substituters = https://cache.nixos.org/ https://hydra.iohk.io/ https://mantis-hydra.aws.iohkdev.io/ https://cache.nixos.org/
trusted-substituters = https://cache.nixos.org/ https://hydra.iohk.io/ https://mantis-hydra.aws.iohkdev.io/
deepfire commented 4 years ago

Output of nix-build -vvv -A nixosTests.chairmansCluster:

$ grep 'hydra.iohk.io.*26rbz3cbjcgdyh9zv0x1ic8bfablpkiz' build.log 
checking substituter 'https://hydra.iohk.io' for path '/nix/store/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz-ghc-8.4.4'
downloading 'https://hydra.iohk.io/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo'...
starting download of https://hydra.iohk.io/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo
finished download of 'https://hydra.iohk.io/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo'; curl status = 0, HTTP status = 404, body = 304 bytes

..however:

$ curl -vv https://hydra.iohk.io/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo
*   Trying 18.197.159.233:443...
* TCP_NODELAY set
* Connected to hydra.iohk.io (18.197.159.233) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=hydra.iohk.io
*  start date: Nov 10 23:50:21 2019 GMT
*  expire date: Feb  8 23:50:21 2020 GMT
*  subjectAltName: host "hydra.iohk.io" matched cert's "hydra.iohk.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x14977f0)
> GET /26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo HTTP/2
> Host: hydra.iohk.io
> User-Agent: curl/7.66.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 301 
< server: nginx
< date: Tue, 19 Nov 2019 18:24:31 GMT
< content-type: text/html
< content-length: 178
< location: https://iohk-nix-cache.s3-eu-central-1.amazonaws.com/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo
< 
<html>
<head><title>301 Moved Permanently</title></head>
<body bgcolor="white">
<center><h1>301 Moved Permanently</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host hydra.iohk.io left intact
deepfire commented 4 years ago

Ok, after following redirects it's dead indeed:

$ curl -vL https://hydra.iohk.io/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo
*   Trying 18.197.159.233:443...
* TCP_NODELAY set
* Connected to hydra.iohk.io (18.197.159.233) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=hydra.iohk.io
*  start date: Nov 10 23:50:21 2019 GMT
*  expire date: Feb  8 23:50:21 2020 GMT
*  subjectAltName: host "hydra.iohk.io" matched cert's "hydra.iohk.io"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x14167f0)
> GET /26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo HTTP/2
> Host: hydra.iohk.io
> User-Agent: curl/7.66.0
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 301 
< server: nginx
< date: Tue, 19 Nov 2019 18:34:00 GMT
< content-type: text/html
< content-length: 178
< location: https://iohk-nix-cache.s3-eu-central-1.amazonaws.com/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo
< 
* Ignoring the response-body
* Connection #0 to host hydra.iohk.io left intact
* Issue another request to this URL: 'https://iohk-nix-cache.s3-eu-central-1.amazonaws.com/26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo'
*   Trying 52.219.72.168:443...
* TCP_NODELAY set
* Connected to iohk-nix-cache.s3-eu-central-1.amazonaws.com (52.219.72.168) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server did not agree to a protocol
* Server certificate:
*  subject: C=US; ST=Washington; L=Seattle; O=Amazon.com, Inc.; CN=*.s3.eu-central-1.amazonaws.com
*  start date: Nov  9 00:00:00 2019 GMT
*  expire date: Dec 10 12:00:00 2020 GMT
*  subjectAltName: host "iohk-nix-cache.s3-eu-central-1.amazonaws.com" matched cert's "*.s3-eu-central-1.amazonaws.com"
*  issuer: C=US; O=DigiCert Inc; OU=www.digicert.com; CN=DigiCert Baltimore CA-2 G2
*  SSL certificate verify ok.
> GET /26rbz3cbjcgdyh9zv0x1ic8bfablpkiz.narinfo HTTP/1.1
> Host: iohk-nix-cache.s3-eu-central-1.amazonaws.com
> User-Agent: curl/7.66.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< x-amz-request-id: 6E7BAD5BE8C9C7D9
< x-amz-id-2: spOer5W7tU9KzT7lJw7e/N0QtMgXqPKzK2nf1FsrmvZpmWhDhX0GjFaZcID2f2yw3fW6vDHEjGQ=
< Content-Type: application/xml
< Transfer-Encoding: chunked
< Date: Tue, 19 Nov 2019 18:34:00 GMT
< Server: AmazonS3
< 
<?xml version="1.0" encoding="UTF-8"?>
* Connection #1 to host iohk-nix-cache.s3-eu-central-1.amazonaws.com left intact
<Error><Code>NoSuchBucket</Code><Message>The specified bucket does not exist</Message><BucketName>iohk-nix-cache</BucketName><RequestId>6E7BAD5BE8C9C7D9</RequestId><HostId>spOer5W7tU9KzT7lJw7e/N0QtMgXqPKzK2nf1FsrmvZpmWhDhX0GjFaZcID2f2yw3fW6vDHEjGQ=</HostId></Error>
deepfire commented 4 years ago

@cleverca22 says:

there is a .narinfo that points to a nar/<hash>.xz which is missing

..which is based on a clean (post nix-collect-garbage's) run of:

deepfire@andromedae:~/cardano-node$ nix-instantiate -A nixosTests.chairmansCluster -Q
unpacking 'https://github.com/input-output-hk/iohk-nix/archive/5f67c8d5b6f4e0c202c6c949d91ff1cb4d7050a3.tar.gz'...
unpacking 'https://github.com/input-output-hk/nixpkgs/archive/55db2a95eade4a80fadf85ae7fce7fa293f87f50.tar.gz'...
copying path '/nix/store/mw5a5mf54c5mcgzw5zhhpx2ixy6zp9r1-mirrors-list' from 'https://cache.nixos.org'...
copying path '/nix/store/gyf5cwnbwa4lfkz9m2mhfqar6dmhk3dy-curl-7.66.0-man' from 'https://cache.nixos.org'...
copying path '/nix/store/my7m2sf9n8d4b5ss80z3glwlbd2035w5-nghttp2-1.39.2' from 'https://cache.nixos.org'...
copying path '/nix/store/vs14q7rcmjsy5gxmbds7fij484sdam0z-libkrb5-1.17' from 'https://cache.nixos.org'...
copying path '/nix/store/z60n71gvr0jn05046l5yd2vf50dg1c6b-nghttp2-1.39.2-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/qjgifg6nzqmvl5814s2yiks7nj408ydi-patchelf-0.9' from 'https://cache.nixos.org'...
copying path '/nix/store/hal0sxabwi18m232s3qindpzxlzbhwxv-openssl-1.1.1d' from 'https://cache.nixos.org'...
copying path '/nix/store/npsin1jlnzs5sqxpsv2mk5gbnm4hk4kr-stdenv-linux' from 'https://cache.nixos.org'...
copying path '/nix/store/98hjgzby078n0l70zjqjzw729yl8xhc1-libkrb5-1.17-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/3fa2dny19fhs2qz863csaql4sr1fip31-libssh2-1.9.0' from 'https://cache.nixos.org'...
copying path '/nix/store/sk5jnqfj9gkcjfwc6bis3fi1d4z0nwyq-nghttp2-1.39.2-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/wil7rx6ggax77wikzk1hjrsylgqq101r-openssl-1.1.1d-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/lnspgzs99ic6yci4zizjv1l8kvqficq1-curl-7.66.0' from 'https://cache.nixos.org'...
copying path '/nix/store/kicfbq4jqsy7syc3daaf1vy1xi48fz6s-libssh2-1.9.0-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/n02qmp9h7lm7k3g1bw3x9v2xq4qf1a24-openssl-1.1.1d-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/b26lw9pn2x8y19bswwxahrcmgx9cb124-nghttp2-1.39.2-dev' from 'https://cache.nixos.org'...
copying path '/nix/store/n70zk2s410gs238rli9fbb8wm1240ivw-curl-7.66.0-bin' from 'https://cache.nixos.org'...
copying path '/nix/store/bv8wjil273x13aavx0ackkq6d6lya6cn-curl-7.66.0-dev' from 'https://cache.nixos.org'...
building '/nix/store/p85wn80802vbp72sv1635ywbm1inhhia-source.drv'...
unpacking 'https://github.com/input-output-hk/haskell.nix/archive/aa10a0b7de7f0d6e82c7a10bb8cd3ca390821feb.tar.gz'...
unpacking 'https://github.com/input-output-hk/stackage.nix/archive/866f9bafcb884c0a92bd708232caa808fc48080c.tar.gz'...
warning: dumping very large path (> 256 MiB); this may run out of memory
unpacking 'https://github.com/input-output-hk/hackage.nix/archive/dd740b03ea26c1d574dc6edab24357987602153b.tar.gz'...
building '/nix/store/d6bhh5zkh4dqsknwcs9qphb6fj7vx3bl-source.drv'...
copying path '/nix/store/xbxca9msc2pgc3zwdl53w2msk5q524dg-stdenv-linux' from 'https://hydra.iohk.io'...
copying path '/nix/store/1vli9fs1lxqnbhakb834yjfwr1viz2xj-lndir-1.0.3' from 'https://hydra.iohk.io'...
copying path '/nix/store/w1r8r5lylqb237fgydk45n09v2ybbw7r-default-Setup' from 'https://hydra.iohk.io'...
copying path '/nix/store/2b25y0nkp2ch4r0z56pvgr858hd4jhds-hook' from 'https://hydra.iohk.io'...
file 'nar/08ppd3a2328iivn28hjmmp0bs86r4n4lis9fxf71k7k3bis7pj33.nar.xz' does not exist in binary cache 'https://hydra.iohk.io'
building '/nix/store/p46j2csg16z3dcr0b6575p1vbzpj62xj-stdenv-linux.drv'...
file 'nar/1gb5kyjwr0m63a1imwf4qdcbcnxisiwqd9d7lvgqan6r6hmafdgs.nar.xz' does not exist in binary cache 'https://hydra.iohk.io'
file 'nar/0b0dp6i8cw0ar9mx1fp9kj39q1hjlc1s2srrm63bcwxvi21xm39s.nar.xz' does not exist in binary cache 'https://hydra.iohk.io'
file 'nar/0996wz0y4a3lc5ql6wb3a2my44zs2szvv7bgaznq9gizbxapxj7b.nar.xz' does not exist in binary cache 'https://hydra.iohk.io'
copying path '/nix/store/rhwljq7yyq4gka6jan564ywahjjx8jxr-lndir-1.0.3.tar.bz2' from 'https://cache.nixos.org'...
copying path '/nix/store/ggm022dh0wh2an74vxypk4jlls3bsmxm-xproto-7.0.31' from 'https://hydra.iohk.io'...
copying path '/nix/store/2b25y0nkp2ch4r0z56pvgr858hd4jhds-hook' from 'https://hydra.iohk.io'...
copying path '/nix/store/h062q7p4ay0kknjwsklncvah0ymaak5b-pkg-config-0.29.2' from 'https://hydra.iohk.io'...
building '/nix/store/zp73vw2vb0rf0sprk9p679z0izs5hcj3-cardano-config-0.1.0.0-lib-cardano-config-config.drv'...
building '/nix/store/5q0d62dmiq4mm0m814hwcfa83fm1gb8r-default-Setup.drv'...
building '/nix/store/7c28g1k1ax54xfxprfvv4g2m1yzmcd9f-remove-references-to.drv'...
file 'nar/05apwkvlxf9ghylamp6df8wsvc8bwmcczagwlxfnrmpca6663wmp.nar.xz' does not exist in binary cache 'https://hydra.iohk.io'
copying path '/nix/store/vh9hkvjvsh9l5d3plghb0rb7292gwiij-xproto-7.0.31.tar.bz2' from 'https://cache.nixos.org'...
file 'nar/0b0dp6i8cw0ar9mx1fp9kj39q1hjlc1s2srrm63bcwxvi21xm39s.nar.xz' does not exist in binary cache 'https://hydra.iohk.io'
file 'nar/1lsmhiw5ydik1bx9m9998ajwgqadbkpg8lv2xyk4s9a49p7yfv06.nar.xz' does not exist in binary cache 'https://hydra.iohk.io'
copying path '/nix/store/83haczzqmdmjjaf73y4fnk1zkcfai9zf-pkg-config-0.29.2.tar.gz' from 'https://cache.nixos.org'...
building '/nix/store/z9crpqg5f5isqzfqwlwkrdw42dxdwj4n-hook.drv'...
copying path '/nix/store/68mm5n8xpv2pzrlc918m350z86aq1f77-hook' from 'https://hydra.iohk.io'...
file 'nar/1lpa4skkq4il8cf3i15z6mn0p9y6gvlvfjd31jfsvvi63awdgvdj.nar.xz' does not exist in binary cache 'https://hydra.iohk.io'
building '/nix/store/r5w8gml2df7dsbl9yp9k5caa8vmqc414-hook.drv'...
building '/nix/store/a2j6i2j6rm81xs9pyn58z3pawg25rxpi-pkg-config-0.29.2.drv'...
building '/nix/store/zli8hgams1s7rv5mafs5dilhxdj1rc4b-xproto-7.0.31.drv'...
building '/nix/store/4340mdiq4azc55lvs6cl3akhax9kwbrb-lndir-1.0.3.drv'...
building '/nix/store/pkf2g1bwxxqsmqc2p4v1nwqg06vs3zsi-cardano-config-0.1.0.0-lib-cardano-config-ghc-8.6.5.drv'...
building '/nix/store/zrs680dqfzjr9aszs3ysv4qd2k4cj7hi-cardano-config-0.1.0.0-lib-cardano-config.drv'...
building '/nix/store/94bf33z8s68z9sn90z95gssk02w1s50k-cardano-node-1.0.0-lib-cardano-node-config.drv'...
building '/nix/store/qvdjhfcpm53vxldn2da5wdbwxm3p88w1-cardano-node-1.0.0-lib-cardano-node-ghc-8.6.5.drv'...
building '/nix/store/qmqpbxvdh2cjnx3psd440qfyr35bbrkz-cardano-node-1.0.0-lib-cardano-node.drv'...
building '/nix/store/1ran23q2n83wb11kfmhii8f9sppaypvx-cardano-node-1.0.0-exe-cardano-cli-config.drv'...
building '/nix/store/7p4dr6nnim9lm043x725yx84xbnljfql-cardano-node-1.0.0-exe-cardano-node-config.drv'...
building '/nix/store/fky7k2318p8l1lmk2hcn96xb1jac0jjv-cardano-node-1.0.0-exe-chairman-config.drv'...
building '/nix/store/m3jxm4a38f0l8hmdbl74smvks3l651k3-cardano-node-1.0.0-exe-trace-acceptor-config.drv'...
building '/nix/store/w2hv9dnapsk5xxrrsfy9xipqwldy3xkd-cardano-node-1.0.0-exe-wallet-client-config.drv'...

building '/nix/store/6q882j97x886skav569lb9mlyl7a2izr-cardano-node-1.0.0-exe-cardano-cli-ghc-8.6.5.drv'...
building '/nix/store/vcy1c8mzqrdsz95fvd6ksqd8mry08mbs-cardano-node-1.0.0-exe-trace-acceptor-ghc-8.6.5.drv'...
building '/nix/store/fvbvlxh94hvjf677mgzb7r2rpj6h9hgh-cardano-node-1.0.0-exe-cardano-cli.drv'...
building '/nix/store/rb65cvrcbrl2dvxxq7h7263lsd9k1sjw-cardano-node-1.0.0-exe-chairman-ghc-8.6.5.drv'...
building '/nix/store/3knskr16mjff30lwp7z6kk3w30zqcgd8-cardano-node-1.0.0-exe-wallet-client-ghc-8.6.5.drv'...
building '/nix/store/p93icx86r5xclrkgb0fc6hmb6bz4l0jf-cardano-node-1.0.0-exe-trace-acceptor.drv'...
building '/nix/store/fv78xljk481prkrw09kws4w0i2zbkry7-cardano-node-1.0.0-exe-chairman.drv'...
building '/nix/store/22xs5268z8nxmvf3kv4icf1pvlx68qiv-cardano-node-1.0.0-exe-wallet-client.drv'...
building '/nix/store/jzsmcr5ik6zzbi67imd09lnmm744dvcy-cardano-node-1.0.0-exe-cardano-node-ghc-8.6.5.drv'...
building '/nix/store/bd8sdxywhk3fw45dhsw2f11s7ig9rm8d-cardano-node-1.0.0-exe-cardano-node.drv'...
building '/nix/store/2jbkqmcd1q0j9f4k0zx1syiq6qlc90pq-cardano-node-exes.drv'...
building '/nix/store/v4gmnjnyfn50ljqcmy23l6c18mwgyk8h-genesis-hash.drv'...
building '/nix/store/hjgssp0ygxfhnpfg58vn3i6hlbjb7j47-migrated-leaked-secret-key.sk.drv'...
building '/nix/store/hg1zliw5kzin215g4yxqq22mhxlpipp1-key.pub.drv'...
building '/nix/store/p6rq2z0nisjy6dc5nzqgx68853brpgm0-migrated-leaked-secret-key.sk.drv'...
building '/nix/store/0gr9x5sj5y83j05d2k4d7gr6zvi5splj-key.pub.drv'...
building '/nix/store/i091jw1ypj45k6m4mk5y6yhpa0rab80d-migrated-leaked-secret-key.sk.drv'...
building '/nix/store/h77jx37l6dcg98cryff395bsdwkadsia-key.pub.drv'...
warning: you did not specify '--add-root'; the result might be removed by the garbage collector
/nix/store/9ihjzh425p2s3wrk06767amclc9ab8an-vm-test-run-chairmans-cluster-test.drv
deepfire commented 4 years ago

Ok, these errors disappeared when I changed geographical location of the Hydra HTTP client.

deepfire commented 4 years ago

Thank you, AWS S3!