search

c0c0n3 / kitt4sme.live

On a mission to bring AI to the shop floor: https://kitt4sme.eu/
MIT License
1 stars 28 forks source link

Nix Flake can't download Kubeseal deps #167

Closed c0c0n3 closed 2 years ago

c0c0n3 commented 2 years ago

Describe the bug

The Nix Flake fails to build the Kubeseal package. It looks like the problem is that the Go build can't download some of the Kubeseal deps from proxy.golang.org.

To Reproduce

  1. Create a multipass VM: multipass launch --name test --cpus 2 --mem 4G --disk 40G 20.04
  2. Shell into it: multipass shell test
  3. Install Nix: sh <(wget -qO- https://nixos.org/nix/install)
  4. Set up Nix env: . /home/ubuntu/.nix-profile/etc/profile.d/nix.sh
  5. Create Nix config dir: mkdir -p ~/.config/nix
  6. Enable Nix Flakes: echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf
  7. Clone KITT4SME live repo: git clone https://github.com/c0c0n3/kitt4sme.live
  8. Enter KITT4SME's Nix shell: cd kitt4sme.live/nix && nix shell

You should see an error similar to the one below:

error: builder for '/nix/store/4vzgbmn0mqqadi7swf278qdj97y26ysn-kubeseal-0.17.5.drv' failed with exit code 1;
       last 10 log lines:
       > pkg/apis/sealed-secrets/v1alpha1/sealedsecret_expansion.go:14:2: k8s.io/apimachinery@v0.23.4: Get "https://proxy.golang.org/k8s.io/apimachinery/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:47564->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:28:2: k8s.io/apimachinery@v0.23.4: Get "https://proxy.golang.org/k8s.io/apimachinery/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:47564->[::1]:53: read: connection refused
       > pkg/apis/sealed-secrets/v1alpha1/register.go:7:2: k8s.io/apimachinery@v0.23.4: Get "https://proxy.golang.org/k8s.io/apimachinery/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:47564->[::1]:53: read: connection refused
       > pkg/apis/sealed-secrets/v1alpha1/register.go:8:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51224->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:30:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51224->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:43:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51224->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:31:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51224->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:32:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51224->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:33:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51224->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:34:2: k8s.io/klog/v2@v2.40.1: Get "https://proxy.golang.org/k8s.io/klog/v2/@v/v2.40.1.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:55715->[::1]:53: read: connection refused
       For full logs, run 'nix log /nix/store/4vzgbmn0mqqadi7swf278qdj97y26ysn-kubeseal-0.17.5.drv'.
error: 1 dependencies of derivation '/nix/store/gcxa0icdrv6d5zy9qh84zlyrx8hi1vz9-kitt4sme-cluster-shell.drv' failed to build

Expected behavior

Nix should build the Kubeseal package and you should be able to enter the KITT4SME's shell.

Additional context

See

c0c0n3 commented 2 years ago

As a temp workaround you can exclude Kubeseal from the Nix Flake and install it manually. Here's how:

$ git clone https://github.com/c0c0n3/kitt4sme.live
$ cd kitt4sme.live/nix

Now edit flake.nix to delete kubeseal from this line

i.e. after editing the above line should look like this

          paths = [ git kubectl istioctl argocd kustomize opa ];

Now install the kubeseal client. Important: make sure you install version 0.17.5:

c0c0n3 commented 2 years ago

Notice if you've followed the cluster bootstrap procedure and tools.sh didn't work for you because of this Kubeseal issue (see error above), then you should not use tools.sh to start the KITT4SME's Nix shell. Instead, use the workaround outlined above and enter the Nix shell with

$ cd kitt4sme.live/nix
$ nix shell

This has basically the same effect as running tools.sh.

ds2268 commented 2 years ago

@c0c0n3 nix shell does not enter the interactive nix CMD. It just outputs the warning: warning: Git tree '/home/k4s-live/kitt4sme.live' is dirty. It looks like it does enter the nix cmd, but it doesn't have any new name appended, so it looks like it didn't enter. In normal nix usage, you get a "conda-like" env name appended. It's fine if it's not...but mentioned that maybe in README. It's a dummy, but someone that does not blindly follow the steps in README...would expect different behavior....even though it works :)

berserkrambo commented 2 years ago

@c0c0n3, same error, after the above fix I get "warning: Git tree '/home/ubuntu/kitt4sme.live' is dirty." but...version of this repo before Apr 13, 2022 works, bye :)

c0c0n3 commented 2 years ago

@ds2268, @berserkrambo you can ignore that warning, Nix should work regardless. Anyhoo, we're working on a fix, since this affected quite a lot of peeps out there. Stay tuned :-)

c0c0n3 commented 2 years ago

So I've done some extensive testing and debugging, but the reason why we're getting that "connection refused" error is still shrouded in mystery. For the record, here's what to do to reproduce the error in an isolated environment with just the kubeseal package.

VM

Get Multipass to spin up a 20.4 Ubuntu VM and shell into it. (Architecture: x86_64)

$ multipass launch --name debug --cpus 2 --mem 4G --disk 40G 20.04
$ multipass shell debug

The rest of the steps below assume you're running the commands inside this shell on the Ubuntu VM.

Nix

Install Nix on the VM and enable Flakes.

$ sh <(wget -qO- https://nixos.org/nix/install)
$ . /home/ubuntu/.nix-profile/etc/profile.d/nix.sh
$ mkdir -p ~/.config/nix
$ echo 'experimental-features = nix-command flakes' >> ~/.config/nix/nix.conf

Test repo

Create a git repo with the flake.nix and kubeseal.nix files below in it.

$ mkdir repo && cd repo
$ git init

flake.nix

{
  description = "Debug me!";

  inputs = {
    nixpkgs.url = "github:NixOs/nixpkgs/nixos-21.11";
  };

  outputs = { self, nixpkgs }:
    let
      system = "x86_64-linux";
      pkgs = nixpkgs.legacyPackages.${system};
      kubeseal = pkgs.callPackage ./kubeseal.nix {};
    in {
      packages.${system}.kubeseal = kubeseal;
      defaultPackage.${system} = pkgs.buildEnv {
        name = "shell";
        paths = [ kubeseal ];
      };
    };
}

kubeseal.nix

#
# Custom Sealed Secrets package.
#
# So I've rolled out a bare build recipe whipped together by looking
# at what the kubeseal Makefile target does. To be replaced with the
# official Nixpkgs expression.
#
{ lib, stdenv, go_1_17, fetchFromGitHub }:

stdenv.mkDerivation rec {
  pname = "kubeseal";
  version = "0.17.5";

  src = fetchFromGitHub {
    owner = "bitnami-labs";
    repo = "sealed-secrets";
    rev = "v${version}";
    sha256 = "sha256-cqOSMAagefKQiKKtgVbk1UFKYGBXQleJ1pgcJ/VyOnM=";
  };

  GO_LD_FLAGS = "-s -w -X main.VERSION=${version}";

  buildInputs = [ go_1_17 ];

  buildPhase = ''
    mkdir -p $out/mod-cache
    export GOMODCACHE=$out/mod-cache

    mkdir -p $out/build-cache
    export GOCACHE=$out/build-cache

    go build -o kubeseal -ldflags "$GO_LD_FLAGS" ./cmd/kubeseal
  '';
  installPhase = ''
    mkdir -p $out/bin
    cp kubeseal $out/bin
  '';

}

Now add the two files to the git index.

$ git add .

Flake build

The moment we've been eagerly waiting for. Drum roll...

$ nix build

It should bomb out with an error similar to this:

error: builder for '/nix/store/h0bnm7dvj7jg6vhg86a9lalyqnz687wb-kubeseal-0.17.5.drv' failed with exit code 1;
       last 10 log lines:
       > pkg/apis/sealed-secrets/v1alpha1/sealedsecret_expansion.go:14:2: k8s.io/apimachinery@v0.23.4: Get "https://proxy.golang.org/k8s.io/apimachinery/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51547->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:28:2: k8s.io/apimachinery@v0.23.4: Get "https://proxy.golang.org/k8s.io/apimachinery/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51547->[::1]:53: read: connection refused
       > pkg/apis/sealed-secrets/v1alpha1/register.go:7:2: k8s.io/apimachinery@v0.23.4: Get "https://proxy.golang.org/k8s.io/apimachinery/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:51547->[::1]:53: read: connection refused
       > pkg/apis/sealed-secrets/v1alpha1/register.go:8:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:58276->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:30:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:58276->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:43:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:58276->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:31:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:58276->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:32:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:58276->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:33:2: k8s.io/client-go@v0.23.4: Get "https://proxy.golang.org/k8s.io/client-go/@v/v0.23.4.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:58276->[::1]:53: read: connection refused
       > cmd/kubeseal/main.go:34:2: k8s.io/klog/v2@v2.40.1: Get "https://proxy.golang.org/k8s.io/klog/v2/@v/v2.40.1.zip": dial tcp: lookup proxy.golang.org on [::1]:53: read udp [::1]:55080->[::1]:53: read: connection refused
       For full logs, run 'nix log /nix/store/h0bnm7dvj7jg6vhg86a9lalyqnz687wb-kubeseal-0.17.5.drv'.
error: 1 dependencies of derivation '/nix/store/zw75adh8k7kdqr3c938769mj55576sib-shell.drv' failed to build

Ha! Must be a firewall issue. Let's open port 53.

$ sudo ufw allow 53/udp
$ sudo ufw allow 53/tcp

But if you run nix build again...you should get the same error as before. Ouch!

Surprise

Let's do some debugging. We're going to get an environment similar to what Nix uses to build the package and then run the Nix standard build in there---i.e. pretty much what mkDerivation does.

$ nix develop .#kubeseal
$ source $stdenv/setup
$ genericBuild

This build actually works

$ ./kubeseal --version
kubeseal version: 0.17.5
$ exit

What the heck?! Let's try again but with a cleaner environment.

$ sudo rm -rf outputs source
$ nix develop -i .#kubeseal
$ source $stdenv/setup
$ genericBuild
unpacking sources
unpacking source archive /nix/store/ilv8ribp7ic29hql6lszpp35qyxjn429-source
source root is source
patching sources
configuring
no configure script, doing nothing
building
missing $GOPATH

Fair enough, let's set GOPATH.

$ sudo rm -rf outputs source
$ mkdir ignore-me
$ nix develop -i .#kubeseal
$ export GOPATH=`pwd`/ignore-me
$ source $stdenv/setup
$ genericBuild

This actually works

$ ./kubeseal --version
kubeseal version: 0.17.5
$ exit

But why?! Hang on though. It gets better. Given that setting GOPATH worked in what's supposedly an environment as close as it gets to the actual Nix build, I thought adding these two lines to the build phase in kubeseal.nix would to the trick

...
  buildPhase = ''
    mkdir ignore-me
    export GOPATH=`pwd`/ignore-me
...

Turns out I was so wrong. Running nix build again with the above change outputs the same "connection refused" error as before. Back to square one.

c0c0n3 commented 2 years ago

@ds2268, @berserkrambo we've just merged #171 which hopefully should fix this issue. If you could please test on your side so we can close this issue if you're happy with the fix. Notice you'll have to undo your changes in git before merging in the code from upstream. Thanks!

berserkrambo commented 2 years ago

Ok I re-run all in a clean multipass vm and it seems to work correctly until section "Continuous delivery" included. TOP !!

Now, I'm trying to go forward on "Post-install steps" section where you say "The only unhappy service should be Keycloak", but I get: default profilers-774dbfc5fc-t5sxq 0/2 Init:CreateContainerConfigError 0 11m default keycloak-645f7df959-wpxff 1/2 CreateContainerConfigError 0 11m default postgres-74487cff86-8jc4x 1/2 CreateContainerConfigError 0 11m default quantumleap-7dcb7cff57-h4d8t 0/2 Init:CreateContainerConfigError 0 11m

It is because I have to generate new secrets?

Thank you and forgive me if I ask you this question not specifically related to the above issue, I'm a total newbe on all thoose "cluster-nix-kube-microk8s..ecc..ecc" stuff

ds2268 commented 2 years ago

I can also confirm that the bug is now fixed.

ds2268 commented 2 years ago

@berserkrambo I think that multiple people will come to that problem and the README can be hard to follow on the security part. Please check https://github.com/c0c0n3/kitt4sme.live/issues/177, where I have described what you need to do in a more direct language :)

berserkrambo commented 2 years ago

Thanks for the help @ds2268 xD, now it's more clear but..still I cannot log into argocd server -.-

I edited kustomization.yaml uncommenting last section, result:

- _replacements_/

# replacements:
- path: _replacements_/argocd-sso.yaml
- path: _replacements_/argocd-webapp.yaml
- path: _replacements_/gitops-repo.yaml

Edited custom_urls.yaml resulting:

data:
  argocd.repo: https://github.com/berserkrambo/kitt4sme.live
  ...
 argocd.webapp: http://127.0.0.1/argocd
  ...
  argocd.sso: |
    name: Keycloak
    issuer: http://127.0.0.1/auth/realms/master
    clientID: argocd
    clientSecret: $oidc.keycloak.clientSecret
    requestedScopes: ["openid", "profile", "email", "groups"]

Then...i generated new secrets using templates to generate keycloak-builtin-admin.yaml and postgres-users.yaml and pushed them on my forked kitt4sme repo..

Finally I deleted the old vm and created a new one and followed the readme till the Post-install steps obviously always replacing c0c0n3 git links with mine but....still cannot login or ping or see argocd and btw continuing to have same CreateContainerConfigError on quantumleap, keycloak, profilers and postgres

@c0c0n3 any ideas? what am I missing / doing wrong?

thanks!

berserkrambo commented 2 years ago

ok I finally managed to enter argocd, I think I messed up something with multipass lol, now I'll try to get rid of "CreateContainerConfigError" message, maybe I did something wrong during the process of creating new secrets...

c0c0n3 commented 2 years ago

@berserkrambo glad you came right in the end, good stuff. If you have issues or want to report a bug, please use the discussion forum for questions or open a new GitHub issue for bug reports or to request enhancements. So we can keep each thread focussed on just one topic :-)

Speaking of which, this issue was about Kubeseal. #171 fixed the problem, so I'm closing this issue :-)