search

BCDevOps / OpenShift4-Migration

Scripts and info for Ministry teams migration from OpenShift 3.11 to 4.x
Apache License 2.0
3 stars 0 forks source link

Docker & Artifactory implementation details #51

Closed arcshiftsolutions closed 2 years ago

arcshiftsolutions commented 3 years ago

What's the issue?

Several teams are now hitting the docker rate-limit issue; only 200 images can be pulled every 6 hours.

Solutions

Two solutions were recommended by the platform team:

Platform Docs

https://developer.gov.bc.ca/Artifact-Repositories-(Artifactory)

Artifactory Repository

Automated Solution

This solution automates the artifactory process and avoids having to explicitly define the pullSecret or imagePullSecret in your BCs or DCS; run initOSProjects.sh

If the pull secrets are created and properly linked to the correct service accounts in the given environments the use of the Artifactory proxy to pull images becomes completely transparent to the project teams; i.e. there is NO need to explicitly define them in the BCs and DCs.

Here are some recent updates we did to the BCDevOps/openshift-developer-tools; https://github.com/BCDevOps/openshift-developer-tools/pull/128/files.

Those functions are part of the initOSProjects.sh, which automates the initial setup of the roles and secrets on a given project set. It automatically detects the artifacts-default-* creds in the tools project and automatically sets up the pull credentials and links them to the service accounts.

Example run (on an environment that already been setup, but it gives you the gist):

Found secret artifacts-default-ugnrgl, would you like to use this as a pull secret? (y/n)
y

Pull secret, artifactory-creds already exists in 583dbf-tools ...
Linking pull secret, artifactory-creds, to the default service account in 583dbf-tools ...
Linking pull secret, artifactory-creds, to the builder service account in 583dbf-tools ...

Pull secret, artifactory-creds already exists in 583dbf-dev ...
Linking pull secret, artifactory-creds, to the default service account in 583dbf-dev ...
Linking pull secret, artifactory-creds, to the builder service account in 583dbf-dev ...

Pull secret, artifactory-creds already exists in 583dbf-test ...
Linking pull secret, artifactory-creds, to the default service account in 583dbf-test ...
Linking pull secret, artifactory-creds, to the builder service account in 583dbf-test ...

Pull secret, artifactory-creds already exists in 583dbf-prod ...
Linking pull secret, artifactory-creds, to the default service account in 583dbf-prod ...
Linking pull secret, artifactory-creds, to the builder service account in 583dbf-prod ...

If you are using the automated scripts to setup the secrets and you still run into authentication issues pulling images, we have found, in some (limited) cases, the order the secrets are listed under the service account is significant:

This does not work in some cases (artifactory-creds listed last):

kind: ServiceAccount
apiVersion: v1
metadata:
  name: builder
  namespace: 069465-tools
  selfLink: /api/v1/namespaces/069465-tools/serviceaccounts/builder
  uid: 7b4cf3e7-c4fe-4d87-89ec-e6623535a040
  resourceVersion: '885860320'
  creationTimestamp: '2021-04-21T00:19:56Z'
secrets:
  - name: builder-dockercfg-2lxkk
  - name: builder-token-dwqrb
  - name: artifactory-creds
imagePullSecrets:
  - name: builder-dockercfg-2lxkk
kind: ServiceAccount
apiVersion: v1
metadata:
  name: default
  namespace: 069465-tools
  selfLink: /api/v1/namespaces/069465-tools/serviceaccounts/default
  uid: 5bde1f20-4b45-490e-aa14-2dfd9baec5ac
  resourceVersion: '885860288'
  creationTimestamp: '2021-04-21T00:19:56Z'
secrets:
  - name: default-token-8v8gl
  - name: default-dockercfg-7g9xf
imagePullSecrets:
  - name: default-dockercfg-7g9xf
  - name: artifactory-creds

In this case the solution is to change the order so the artifactory-creds are listed first:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: builder
  namespace: 069465-tools
  selfLink: /api/v1/namespaces/069465-tools/serviceaccounts/builder
  uid: 7b4cf3e7-c4fe-4d87-89ec-e6623535a040
  resourceVersion: '885860320'
  creationTimestamp: '2021-04-21T00:19:56Z'
secrets:
  - name: artifactory-creds
  - name: builder-dockercfg-2lxkk
  - name: builder-token-dwqrb
imagePullSecrets:
  - name: builder-dockercfg-2lxkk
kind: ServiceAccount
apiVersion: v1
metadata:
  name: default
  namespace: 069465-tools
  selfLink: /api/v1/namespaces/069465-tools/serviceaccounts/default
  uid: 5bde1f20-4b45-490e-aa14-2dfd9baec5ac
  resourceVersion: '885860288'
  creationTimestamp: '2021-04-21T00:19:56Z'
secrets:
  - name: default-token-8v8gl
  - name: default-dockercfg-7g9xf
imagePullSecrets:
  - name: artifactory-creds
  - name: default-dockercfg-7g9xf

Manual Artifactory Solution

For Artifactory, you will need to use the username & password from the secret created by the platform team in the TOOLS env. Your secret should be in the format of default-[namespacename]-[plate] or artifacts-default-[plate]. We also needed one additional step in our build configurations:

Create Secret

oc create secret docker-registry artifactory-creds \
    --docker-server=docker-remote.artifacts.developer.gov.bc.ca \
    --docker-username=<our username from secret> \
    --docker-password=<our password from secret> \
    --docker-email=unused

Link Secret for Build & Pulls

oc secrets link default artifactory-creds --for=pull
oc secrets link builder artifactory-creds

Change required to Dockerfile (include docker-remote.artifacts.developer.gov.bc.ca)

FROM docker-remote.artifacts.developer.gov.bc.ca/node:lts-alpine

Change required to Build Config (note the pullSecret)

strategy:
      dockerStrategy:
        pullSecret:
          name: artifactory-creds
        env:
        - name: BUILD_LOGLEVEL
          value: '2'
        - name: NPM_CONFIG_LOGLEVEL
          value: notice
      type: Docker

Docker Auth

For Docker auth, you will need your own docker username and password. We ran the following commands in our tools namespace:

Create Secret

oc create secret docker-registry docker-creds \
    --docker-server=docker.io \
    --docker-username=<docker username> \
    --docker-password=<docker password> \
    --docker-email=unused

Link Secret for Build & Pulls

oc secrets link default docker-creds --for=pull
oc secrets link builder docker-creds

Change the BuildConfig to specify the image (include docker.io)

  strategy:
    dockerStrategy:
      from:
        kind: DockerImage
        name: docker.io/node:lts-alpine

Change required to Dockerfile (include docker.io)

FROM docker.io/node:lts-alpine
WadeBarnes commented 3 years ago

Another solution that automates the above process and avoids having to explicitly define the pullSecret or imagePullSecret in your BCs or DCS; run initOSProjects.sh

Details:

If the pull secrets are created and properly linked to the correct service accounts in the given environments the use of the Artifactory proxy to pull images becomes completely transparent to the project teams; i.e. there is NO need to explicitly define them in the BCs and DCs.

Here are some recent updates we did to the BCDevOps/openshift-developer-tools; https://github.com/BCDevOps/openshift-developer-tools/pull/128/files.

Those functions are part of the initOSProjects.sh, which automates the initial setup of the roles and secrets on a given project set. It automatically detects the artifacts-default-* creds in the tools project and automatically sets up the pull credentials and links them to the service accounts.

Example run (on an environment that already been setup, but it gives you the gist):

Found secret artifacts-default-ugnrgl, would you like to use this as a pull secret? (y/n)
y

Pull secret, artifactory-creds already exists in 583dbf-tools ...
Linking pull secret, artifactory-creds, to the default service account in 583dbf-tools ...
Linking pull secret, artifactory-creds, to the builder service account in 583dbf-tools ...

Pull secret, artifactory-creds already exists in 583dbf-dev ...
Linking pull secret, artifactory-creds, to the default service account in 583dbf-dev ...
Linking pull secret, artifactory-creds, to the builder service account in 583dbf-dev ...

Pull secret, artifactory-creds already exists in 583dbf-test ...
Linking pull secret, artifactory-creds, to the default service account in 583dbf-test ...
Linking pull secret, artifactory-creds, to the builder service account in 583dbf-test ...

Pull secret, artifactory-creds already exists in 583dbf-prod ...
Linking pull secret, artifactory-creds, to the default service account in 583dbf-prod ...
Linking pull secret, artifactory-creds, to the builder service account in 583dbf-prod ...
rstens commented 3 years ago

@WadeBarnes Solution does not work for regular admins:

Loading settings ...
Loading settings from /h/biohubbc/settings.sh ...
/c/openshift-developer-tools/bin/settings.sh: line 374: ./settings.sh: No such file or directory
Granting deployment configuration access from '-dev', to '-tools', for service account 'default' ...
Assigning role [system:image-puller], to user [system:serviceaccount:-dev:default], in project [-tools] ...
Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "rstens@github" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "-tools"
WadeBarnes commented 3 years ago

@WadeBarnes Solution does not work for regular admins:

Loading settings ...
Loading settings from /h/biohubbc/settings.sh ...
/c/openshift-developer-tools/bin/settings.sh: line 374: ./settings.sh: No such file or directory
Granting deployment configuration access from '-dev', to '-tools', for service account 'default' ...
Assigning role [system:image-puller], to user [system:serviceaccount:-dev:default], in project [-tools] ...
Error from server (Forbidden): rolebindings.rbac.authorization.k8s.io is forbidden: User "rstens@github" cannot list resource "rolebindings" in API group "rbac.authorization.k8s.io" in the namespace "-tools"

@rstens, You're missing a settings.sh file for the BCDevOps/openshift-developer-tools. The file is used to define project's namespaces, for example; https://github.com/bcgov/orgbook-configurations/blob/master/openshift/settings.sh#L1. Therefore the script does not know how to connect to your projects.

garywong-bc commented 3 years ago

Could we please:

  1. Rename "Option 2 - Artifactory Auth" to "Option 1- Artifactory Auth" and move to the top of the choices?
  2. Add "Option 2 - OpenShift Developer Tools" and copy the text in from https://github.com/BCDevOps/OpenShift4-Migration/issues/51#issuecomment-780035982
  3. Rename "Option 1 - Docker Auth" to "Option 3 - Docker Auth" and move this to the bottom?

Lastly, at https://developer.gov.bc.ca/Artifact-Repositories-(Artifactory) update all references to default-[namespacename]-[plate] to:

default-[namespacename]-[plate] or artifacts-default-[plate]

I think that'll make things much clearer, and lessen the odds of a new dev choosing 'Docker Auth' as the first choice.

thanks gary

WadeBarnes commented 3 years ago

If you are using the automated scripts to setup the secrets and you still run into authentication issues pulling images, we have found, in some (limited) cases, the order the secrets are listed under the service account is significant:

This does not work in some cases (artifactory-creds listed last):

kind: ServiceAccount
apiVersion: v1
metadata:
  name: builder
  namespace: 069465-tools
  selfLink: /api/v1/namespaces/069465-tools/serviceaccounts/builder
  uid: 7b4cf3e7-c4fe-4d87-89ec-e6623535a040
  resourceVersion: '885860320'
  creationTimestamp: '2021-04-21T00:19:56Z'
secrets:
  - name: builder-dockercfg-2lxkk
  - name: builder-token-dwqrb
  - name: artifactory-creds
imagePullSecrets:
  - name: builder-dockercfg-2lxkk
kind: ServiceAccount
apiVersion: v1
metadata:
  name: default
  namespace: 069465-tools
  selfLink: /api/v1/namespaces/069465-tools/serviceaccounts/default
  uid: 5bde1f20-4b45-490e-aa14-2dfd9baec5ac
  resourceVersion: '885860288'
  creationTimestamp: '2021-04-21T00:19:56Z'
secrets:
  - name: default-token-8v8gl
  - name: default-dockercfg-7g9xf
imagePullSecrets:
  - name: default-dockercfg-7g9xf
  - name: artifactory-creds

In this case the solution is to change the order so the artifactory-creds are listed first:

kind: ServiceAccount
apiVersion: v1
metadata:
  name: builder
  namespace: 069465-tools
  selfLink: /api/v1/namespaces/069465-tools/serviceaccounts/builder
  uid: 7b4cf3e7-c4fe-4d87-89ec-e6623535a040
  resourceVersion: '885860320'
  creationTimestamp: '2021-04-21T00:19:56Z'
secrets:
  - name: artifactory-creds
  - name: builder-dockercfg-2lxkk
  - name: builder-token-dwqrb
imagePullSecrets:
  - name: builder-dockercfg-2lxkk
kind: ServiceAccount
apiVersion: v1
metadata:
  name: default
  namespace: 069465-tools
  selfLink: /api/v1/namespaces/069465-tools/serviceaccounts/default
  uid: 5bde1f20-4b45-490e-aa14-2dfd9baec5ac
  resourceVersion: '885860288'
  creationTimestamp: '2021-04-21T00:19:56Z'
secrets:
  - name: default-token-8v8gl
  - name: default-dockercfg-7g9xf
imagePullSecrets:
  - name: artifactory-creds
  - name: default-dockercfg-7g9xf
arcshiftsolutions commented 3 years ago

@garywong-bc @WadeBarnes - I've updated the issue description with your recommendations; please let me know if I've missed anything. Thanks!

WadeBarnes commented 3 years ago

@arcshiftsolutions, Looks good. I'd promote the automated approach more, but hopefully the platform team gets things setup in such a way that this all becomes transparent to all of us soon; https://github.com/bcgov/platform-services-registry/issues/359

arcshiftsolutions commented 3 years ago

Thanks @WadeBarnes - I've made further updates to promote the automated approach.

garywong-bc commented 3 years ago

Looks good.. any way to sync up https://developer.gov.bc.ca/Artifact-Repositories-(Artifactory) ? TIA

arcshiftsolutions commented 3 years ago

@garywong-bc I've updated the URL above to the new link - thanks for noting that. :)