Docker Hub login does not work

[kingdonb]

I found a related issue (https://github.com/getkuby/kuby-core/issues/5), but I still had trouble in the latest version of kuby (0.14.0)

$ bundle exec kuby -e production push
Pushing image kingdonb/kuby-test with tags 20211206144259, latest
Attempting to log in to registry at index.docker.io:443
Error response from daemon: login attempt to https://index.docker.io:443/v2/ failed with status: 503 Service Unavailable
Couldn't log in to the registry at index.docker.io:443
build failed: docker command exited with status code 1
Pushing image kingdonb/kuby-test with tags 20211206144259-assets, latest-assets
Attempting to log in to registry at index.docker.io:443
Error response from daemon: login attempt to https://index.docker.io:443/v2/ failed with status: 503 Service Unavailable
Couldn't log in to the registry at index.docker.io:443
build failed: docker command exited with status code 1

The issue appears to be some obscure detail about the index.docker.io service that prohibits accessing it in this way:

https://index.docker.io:443/v2/

If you hit it this way, it doesn't have the error:

https://index.docker.io/v2/

No idea why, but updating this method to look like that solved the issue for me:

sig { returns(String) }
def image_host
  # @image_host ||= "#{image_uri.host}:#{image_uri.port}"
  @image_host ||= "#{image_uri.host}"
end

This is not a reasonable change so I have obviously not packed it into a PR as it will not help anyone other than Docker Hub users, but maybe an appropriate workaround can be incorporated somehow anyway?

Now inexplicably I am getting an error when I try to push the image to the registry, but I get the same error outside of kuby, so it is unlikely to be kuby's fault ;)

[camertron]

Hmm, thanks for filing an issue @kingdonb. What URL are you using for your image? If possible it would be helpful to see your entire kuby.rb config, but if not just the URL would be fine.

[kingdonb]

Sure thing! There is nothing unique about my app, it is the standard "rails new" with a bunch of flags disabling various things, I am actually here trying to learn from kuby about the best way to install a Rails app in production on Kubernetes.

The image_url setting is just my dockerhub username / repo, so kingdonb/kuby-tester. I read enough of the code to figure out how to work around it, but haven't gotten very deep into the rest of the kuby code yet. Will keep reading.

require 'active_support/core_ext'
require 'active_support/encrypted_configuration'

# Define a production Kuby deploy environment
Kuby.define('KubyTest') do
  environment(:production) do
    # Because the Rails environment isn't always loaded when
    # your Kuby config is loaded, provide access to Rails
    # credentials manually.
    app_creds = ActiveSupport::EncryptedConfiguration.new(
      config_path: File.join('config', 'credentials.yml.enc'),
      key_path: File.join('config', 'master.key'),
      env_key: 'RAILS_MASTER_KEY',
      raise_if_missing_key: true
    )

    docker do
      credentials do
        username app_creds[:KUBY_DOCKER_USERNAME]
        password app_creds[:KUBY_DOCKER_PASSWORD]
        email app_creds[:KUBY_DOCKER_EMAIL]
      end

      image_url 'kingdonb/kuby-tester'
    end

    kubernetes do
      provider :digitalocean do
        access_token app_creds[:KUBY_DIGITALOCEAN_ACCESS_TOKEN]
        cluster_id app_creds[:KUBY_DIGITALOCEAN_CLUSTER_ID]
      end

      # Add a plugin that facilitates deploying a Rails app.
      add_plugin :rails_app do
        hostname 'kuby-test.hephy.pro'

        # configure database credentials
        database do
          user app_creds[:KUBY_DB_USER]
          password app_creds[:KUBY_DB_PASSWORD]
        end
      end

    end
  end
end

Once I have a better handle on exactly what kuby is and how it does what it does, I'd like to try contributing a GitOps provider? If that's a reasonable approach.

Maybe there is already something like an "apply-method agnostic YAML manifest exporter" provider that can provide the YAML manifests to be inspected before apply, or to be side-loaded in git through any GitOps tooling.

I work on the FluxCD project under CNCF and I use Rails for some of my personal projects. I have barely used kuby enough to know if it does what I want, (but I think it likely does, or at least mostly!) I spent a lot of time building Ruby docker tooling and came to the conclusion that it's a chore. So I'm glad to see some general-purpose tooling like Kuby that looks like a pretty solid "conceptual compression" application!

[kingdonb]

The workaround I found is not viable across OSes...

error: couldn't determine what port to connect to for 'index.docker.io'

There are evidently some places where image_host must include the port number.

I wound up working around it again in the Linux/amd64 build machine that I commandeered by putting 'index.docker.io' directly into the login CLI command, like this:

      def login(url:, username:, password:)
        cmd = [
          executable, 'login', 'index.docker.io', '--username', username, '--password-stdin'
        ]

        open3_w(cmd) do |stdin, _wait_threads|
...

(Obviously this is even worse than my original suggestion in terms of mergeability but it proved that it can work...)

It seems the original version of image_host was correct, but the docker login command in particular should not take index.docker.io:443 when the default docker hub index is used. You can replicate the trouble directly with docker login index.docker.io:443 --user <username> --password-stdin which also returns the same 503 (no surprise there)

I am now hung up on database issues (edit: it was this: https://github.com/getkuby/kuby-redis/issues/1#issuecomment-972861121) but I will keep reading, looks like my postgres didn't come up on its own. (Not related to the original issue, just excited that I almost have it working...)

I had to back down to k8s 1.19 but eventually my postgres did come up 👍 so close now 😬 (Edit: it works! the rest was 99% database credentials being supplied incorrectly. This is super smooth. A+ 💯 Kuby)

[camertron]

Awesome, thanks for providing all these details 😄

Right, so your image URL looks 100% fine, and after poking around a bit this afternoon I think I know what's going on here. As you noted, docker login seems to not like being given a port at all. Some googling reveals that the Docker registry API is supposed to operate entirely over HTTPS, so I suppose port 443 is just assumed and therefore not required (or even allowed). I was able to fix that problem pretty easily by using image_hostname instead of image_host in tasks.rb.

There's another problem too though. Apparently you can do docker login docker.io or docker login index.docker.io, but docker push only works with docker.io. I have no idea why. I guess it's reasonable for part of the Docker registry API to be available at one URL and another part at a different URL? Whatever, that's apparently how it works in practice, so who am I to argue. I ended up adding another option to Kuby's Docker spec called registry_metadata_url that defaults to index.docker.io. Seems to work as expected. I should be able to publish a new release of kuby-core tomorrow.

Once I have a better handle on exactly what kuby is and how it does what it does, I'd like to try contributing a GitOps provider? If that's a reasonable approach.

Hmm, that sounds interesting! From my (very rudimentary) understanding, GitOps is more a philosophy rather than a set of tooling though, right? What problems would a GitOps plugin solve?

Maybe there is already something like an "apply-method agnostic YAML manifest exporter" provider that can provide the YAML manifests to be inspected before apply, or to be side-loaded in git through any GitOps tooling.

You can run kuby resources to get Kuby to spit out all the YAML files it would send to Kubernetes. Is that helpful?

I work on the FluxCD project under CNCF and I use Rails for some of my personal projects. I have barely used kuby enough to know if it does what I want, (but I think it likely does, or at least mostly!)

Cool! Happy to chat more about any ideas you have :)

I had to back down to k8s 1.19

Ugh yeah, sorry about that. I need to document the supported k8s version on getkuby.io.

This is super smooth. A+ 💯 Kuby

Thanks!! I really appreciate the feedback :)

[kingdonb]

You can run kuby resources to get Kuby to spit out all the YAML files it would send to Kubernetes.

That is helpful, indeed I found that as I was exploring! I also found that kuby setup doesn't really have an equivalent command to export the resources that it would apply. That would be helpful to include!

What problems would a GitOps plugin solve?

I imagine how it would work is, by including the GitOps plugin, your manifests are written to a file instead of applied to a cluster. There can be helpers related to committing those manifests, or validation, but I don't want to take any control away from kuby deploy – I just want to force it through GitOps.

It doesn't really need much to be changed! In my idea for how to update kuby for GitOps MVP style, there would be a new "generic" provider that outputs manifests to a directory and leaves the rest up to you.

One more thing it should do; if it's writing resources out to files, it can use one file or many files, but secrets should not be mixed together with deployments and other resources.

We will use sops encryption to save the kind: Secret resources like RAILS_MASTER_KEY, commit them to the repository (encrypted just the same), and apply them to the cluster via another encryption method like KMS, PGP, or age with Mozilla SOPS.

To encrypt a secret in place with sops -e -i manifest.yaml it should be a file with ONLY secrets in it. In this sense, it's a little less than helpful that kuby resources emits a stream of both.

Happy to chat more

I am anticipating that many Kubernetes developers or users in the future won't have direct access to their clusters and they will come preinstalled with a GitOps operator like Flux or ArgoCD for them, to allow and facilitate their safe updating of the cluster.

Then instead of applying resources directly to their clusters, devs will push them to a branch and commit via pull request, with all the benefits of Git and safety of CI checks around their changeset. Policy engines, validators, dry-run and previews can confirm the commit can be applied, and the success or failure is logged as a CI check.

For a tool like kuby to be GitOps-friendly I think it should provide an option (like flux create --export) to emit manifests as files in the filesystem for any create or apply operations, and let them be captured as git commits. Then kuby can be used in a CI process or gitops bootstrap operation, without requiring it to have direct access to the cluster.

[kingdonb]

I was able to use kuby to do the deployment via GitOps, with the exception of the kuby setup stuff – I have a KubeDB enterprise license but not on this cluster anyway, so I'm likely to use the external database capability and reach across the network. I'm also super familiar with cert-manager and already had it installed, or facilities to have it installed. I'm not 100% sure what else was in there that I might still be missing from setup...

https://github.com/kingdonb/bootstrap-repo/blob/staging/secrets/moo-cluster/kuby-test-secret.yaml and https://github.com/kingdonb/bootstrap-repo/blob/staging/clusters/moo-cluster/kuby-test/manifests.yaml

[camertron]

That is helpful, indeed I found that as I was exploring! I also found that kuby setup doesn't really have an equivalent command to export the resources that it would apply. That would be helpful to include!

Ah, that's because kuby setup is mostly just installing helm charts. It does apply some YAML files that live in the certmanager repo also. I suppose kuby resources could print out their contents. Helm supports rendering resources to files as well. Definitely worth exploring. I think we'd need to add some interface methods to Kuby::Plugin. If you're up for a PR, I would gladly take a look :)

It doesn't really need much to be changed! In my idea for how to update kuby for GitOps MVP style, there would be a new "generic" provider that outputs manifests to a directory and leaves the rest up to you.

Cool that makes sense! Would that mean using a different deployer, or would you just not use Kuby for deployment? The krane gem Kuby uses now does some nice things like applying global resources first, waiting for deployments, etc that you wouldn't get by simply running kubectl apply.

In this sense, it's a little less than helpful that kuby resources emits a stream of both.

Right, I think another nice change would be an option to write all the resources out to individual files. We could potentially use the name of each resource as the file name. Might be a good idea to include the namespace too to avoid naming collisions.

I am anticipating that many Kubernetes developers or users in the future won't have direct access to their clusters and they will come preinstalled with a GitOps operator like Flux or ArgoCD for them, to allow and facilitate their safe updating of the cluster.

That sounds very cool, and I agree. Definitely room for some nice tooling here.

I'm not 100% sure what else was in there that I might still be missing from setup

In a basic Rails app with no other plugins, setup only does three things:

1. Installs certmanager
2. Installs ingress-nginx
3. Installs kubedb

So I think the only thing you're missing is ingress-nginx?

[kingdonb]

would you just not use Kuby for deployment? The krane gem Kuby uses now does some nice things like applying global resources first, waiting for deployments, etc that you wouldn't get by simply running kubectl apply.

That's right, in the GitOps model the cluster's security posture is such that developers nor CI ever get direct (write) access.

Flux does those things you mention too, with kubernetes go-client's server side apply, and a notification pipeline :D

CRDs are applied ahead of instances of CRs, and namespaces are created before the resources that are namespaced. Flux comes with an alertmanager rule for prometheus to monitor (and alert) for un-ready Kustomizations.

Then, a spec.wait directive in Flux Kustomization resource when set to enabled will monitor any Deployment, Service, Ingress, (HelmRelease, what have you) or any other resource that supports Kstatus. The Kustomization will report itself as un-ready if any subordinate resources are not ready.

So I think the only thing you're missing is ingress-nginx?

Cool! And I've actually already got two ingress-nginx deployed with IngressClasses, and a third ingress controller Traefik to handle the defaults. So kuby is served up with traefik, and it's all good 😎

Yeah, with the generic/file provider you just wouldn't use kuby for deployment anymore. The CI process that runs kuby build and kuby push can run kuby deploy as a third step, commit the changes that result, push them and do "Open Pull Request" or push them directly to the staging environment, or both. I haven't really figured out CI yet but I will probably use GitHub Actions. I see you have developed some too.

Is there a GitHub Actions example for kuby (to use to deploy directly to clusters) that I can maybe adapt? (In that case I probably won't use Docker Hub anyway, since ghcr is right on the local network of GitHub Actions...)

[kingdonb]

I would definitely use kuby infra as the kuby resources to kuby setup, vs kuby deploy

(eg. I would not include kuby setup stuff in kuby resources, I think the separation is good. We do a similar thing in Flux, if you see any of our guides where there is an apps Kustomization and a separate infrastructure Kustomization 👍 )

[kingdonb]

would you just not use Kuby for deployment?

I've had a bit further look through Kuby docs and I understood that there are some features for which a filesystem provider does not make sense.

Moreover I noticed that bundle exec kuby -e production resources does not work when there is no active kubeconfig (that might be unnecessary?) but lots of commands here (After the Deploy) might need an active kubeconfig even if you are using gitops for the deploy, which is something that I didn't consider.

If there was a "bring your own ~/.kube/config" provider type to go along with the file output mode that kuby resources kind of already provides, that would be ideal! (I'm using a home lab setup, where my Kubernetes cluster is provisioned either via cluster-api or by kubeadm by hand)

[camertron]

That's right, in the GitOps model the cluster's security posture is such that developers nor CI ever get direct (write) access.

Ok that's a really interesting model. Sounds like flux has the "smarts" to know how to work with raw YAML :)

Is there a GitHub Actions example for kuby (to use to deploy directly to clusters) that I can maybe adapt?

Unfortunately not yet, but it's something that's been on my radar for a little while now. I can't imagine it would be that difficult, but it's one of those things that will take some research, etc, and I haven't had the chance to invest the time. If you'd like to work on this then perhaps we can make it like the official Kuby GitHub action?

I've had a bit further look through Kuby docs and I understood that there are some features for which a filesystem provider does not make sense.

Right yeah, the providers abstraction is really meant to wrap an actual provider like Linode, DigitalOcean, etc. Your suggestion of kuby infra is interesting, but what about simply improving the kuby resources command so it outputs YAML files to disk?

If there was a "bring your own ~/.kube/config" provider type to go along with the file output mode that kuby resources kind of already provides, that would be ideal!

You might check out the bare metal provider which allows you to supply the path to your kubeconfig.

[camertron]

Apologies the fix for this Docker hub issue is taking so long. It's in master now, but I'm waiting to release until I've landed a few other things.

[camertron]

Fixed in v0.15.0.

[kingdonb]

You might check out the bare metal provider which allows you to supply the path to your kubeconfig.

Thanks for that! I hadn't found this on my own, I should really try reading all of the source code, it's going to be hard to avoid it now! >:)

what about simply improving the kuby resources command so it outputs YAML files to disk?

That's really what's going to solve the issue for me.

If you'd like to work on this then perhaps we can make it like the official Kuby GitHub action?

I've got this repo going,

• https://github.com/kingdonb/kuby_test

which a sorta complete not really working GitHub Action, a few things are stuck that I can't really understand why?

I also tried to use buildx, I've patched kuby-core with my own changes at https://github.com/kingdonb/kuby-core - but I'll probably take that out now that you've released these fixes (thank you very much!)

[kingdonb]

Hey I spoke too soon, about my example "not really working"

Turns out it works, when I remove my weird changes and adopt your version 0.15.0 release instead 😁

• https://github.com/kingdonb/kuby_test/runs/4493451566?check_suite_focus=true ✅ Passing!

I believe I can make a gitops example out of this now, even if it's going to be a little less automated for now...

Pushing image ghcr.io/kingdonb/kuby-tester with tags 20211211153400-assets, latest-assets

For now at least it is... Image builds on GitHub Actions, via kuby! Feast your eyes 👍 https://github.com/kingdonb/kuby_test

and the images are public now:

https://github.com/users/kingdonb/packages/container/kuby-tester/versions

[camertron]

Nice, that's excellent! Do you know how to turn it into an action anyone can use via marketplace?

[kingdonb]

Not really, but there isn't much machinery in there - right now anyone can take it and use it just by copying kuby.yml from .github/workflows

https://github.com/kingdonb/kuby_test/blob/main/.github/workflows/kuby.yml

... but they will have to contend with the issue under actions/cache#698 until it gets picked up by someone at GitHub

It is mostly based on the ruby/setup-ruby action which I think can be considered "Canon" and is well-maintained from what I can see!

Pretty much just needs two secrets to be added to GitHub, a PAT with the capability to push packages, the rails_master_key to decrypt credentials, (and the secrets that should be populated into credentials per the existing Kuby guides).

I will write up a walkthrough, but for me the purpose is dual, also showing off the GitOps deployment capability and image update automation, which I have added some examples here:

https://github.com/kingdonb/bootstrap-repo/tree/staging/apps/kuby-test and https://github.com/kingdonb/bootstrap-repo/tree/staging/clusters/moo-cluster/kuby-test

I have this set up so that when a new image gets pushed to the packages at ghcr.io it gets immediately added into a commit here in config.yaml and then deployed straight away.

It seems to be working great, but I'm going to put it through some more paces before I spend any time documenting or promoting it! Will probably put out a blog over the holidays and be sure to copy you 👍

On Sat, Dec 11, 2021, 4:36 PM Cameron Dutro @.***> wrote:

Nice, that's excellent! Do you know how to turn it into an action anyone can use via marketplace?

[yebyen]

Should actually be able to do away with the PAT now that the package repo is appropriately associated with the git repo, and lean on that permissions directive section to grant the access to the ambient GITHUB_TOKEN instead 👍