Open smerle33 opened 9 months ago
Starting with LDAP:
Starting with LDAP:
* The image seems old and should be updated to meet our usual "update/release" system
Todo:
[x] Rename jenkins-infra/ldap to jenkins-infra/docker-ldap to fulfill convention
[x] Add the jenkins-infra/docker-ldap repository to infra.ci to start building / releasing the container image
[x] Set the principal branch of jenkins-infra/docker-ldap
to main
- (see https://github.com/jenkins-infra/helpdesk/issues/2671#issuecomment-1898470338)
[x] Open a PR in enkins-infra/docker-ldap with the required elements for the first build (Jenkinsfile_k8s
, release drafter setup, hadolint fixes). The checks should be OK before merging. - https://github.com/jenkins-infra/docker-ldap/pull/33
[x] Ensure the first release is made (eventually add PRs/hotfixes). DoD: both GH release (and git tag) + DockerHub tag
1.0.0
was created manually through GH release on the latest tag, and the associated image was deployed to DockerHub from the current production imagesskopeo copy --all docker://docker.io/jenkinsciinfra/ldap@sha256:d662790b4a5f1ca979c186241005e816ff86ccedf8161d9f11df43501f077f31 docker://docker.io/jenkinsciinfra/ldap:cron-1.0.0
skopeo copy --all docker://docker.io/jenkinsciinfra/ldap@sha256:a1518d683d3098be912a684892d10f909dd6c5bf2a65dce3f477a2caab73ef22 docker://docker.io/jenkinsciinfra/ldap:1.0.0
[x] Update jenkins-infra/kubernetes-management to use the new image and track it
1.0.1
: https://github.com/jenkins-infra/kubernetes-management/pull/4881[x] (eventually) add updatecli tracking
ldap-1.0.2
]()[x] Remove the Docker LDAP job from trusted.ci and ci.jenkins.io (were already deleted)
Then:
* Then we should be able to build it for arm64
Update:
arm64
image: https://github.com/jenkins-infra/docker-ldap/pull/35arm64
image when needed: https://github.com/jenkins-infra/kubernetes-management/pull/4884arm64
will require to migrate a copy of the data in the same availability "Zone" as the arm64
node pool in the same way we did for weekly.ciImportant note: these migrations are triggering SNAT exhaustion problem, which we though was fixed earlier this week. We have to suspend tentatives until we've fixed it (even temporarily): https://github.com/jenkins-infra/helpdesk/issues/3908
Update: let's start with ACP migration, using the same technique as https://github.com/jenkins-infra/helpdesk/issues/4044 for data migration
Update on ACP (artifact-caching-proxy)
arm64
node pool on the cijenkinsio-agents-1
AKS cluster => no blockers to migrate this one as wellarm64
requires changing the availability zone (from 3 to 1)of the service. As such, the disk must be either re-created in the zone 1 OR migrated to a ZRS type (instead of LRS)Proposed migration plan:
2024-07-05-acp0-data
and 2024-07-05-acp1-data
arm64
scheduling#jenkins-infra
Pod manifests to migrate data using rsync:
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-artifact-caching-proxy-0
namespace: artifact-caching-proxy
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "32Gi"
storageClassName: statically-provisionned
volumeName: repo-azure-jenkins-io-pv-0
---
apiVersion: v1
kind: Pod
metadata:
name: migrate-volume-0
namespace: artifact-caching-proxy
spec:
securityContext:
runAsUser: 0
runAsGroup: 0
fsGroup: 0
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: statefulset.kubernetes.io/pod-name
operator: In
values:
- artifact-caching-proxy-0
topologyKey: kubernetes.io/hostname
containers:
- image: jenkinsciinfra/packaging:latest
name: migrate-volume-script
command: ["rsync"]
args: ["-v", "-a", "--delete", "/src-0/", "/dest-0/"]
volumeMounts:
- mountPath: /src-0
name: old
readOnly: true
- mountPath: /dest-0
name: new
nodeSelector:
kubernetes.io/arch: amd64
restartPolicy: Never
volumes:
- name: old
persistentVolumeClaim:
claimName: nginx-cache-artifact-caching-proxy-0
- name: new
persistentVolumeClaim:
claimName: data-artifact-caching-proxy-0
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: data-artifact-caching-proxy-1
namespace: artifact-caching-proxy
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: "32Gi"
storageClassName: statically-provisionned
volumeName: repo-azure-jenkins-io-pv-1
---
apiVersion: v1
kind: Pod
metadata:
name: migrate-volume-1
namespace: artifact-caching-proxy
spec:
securityContext:
runAsUser: 0
runAsGroup: 0
fsGroup: 0
containers:
- image: jenkinsciinfra/packaging:latest
name: migrate-volume-script
command: ["rsync"]
args: ["-v", "-a", "--delete", "/src-1/", "/dest-1/"]
volumeMounts:
- mountPath: /src-1
name: old
readOnly: true
- mountPath: /dest-1
name: new
nodeSelector:
kubernetes.io/arch: amd64
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: statefulset.kubernetes.io/pod-name
operator: In
values:
- artifact-caching-proxy-1
topologyKey: kubernetes.io/hostname
restartPolicy: Never
volumes:
- name: old
persistentVolumeClaim:
claimName: nginx-cache-artifact-caching-proxy-1
- name: new
persistentVolumeClaim:
claimName: data-artifact-caching-proxy-1
Update on LDAP
arm64
architecture available:
arm64
requires changing the availability zone (from 3 to 1)of the service. As such, the disk must be either re-created in the zone 1 OR migrated to a ZRS type (instead of LRS)We also want to adapt the disk kind to the real life usage:
Current disk is Standard SSD, LRS (zone 3), 50 Gb
Current disk usage metrics are:
Usage is around 386Mb:
kubectl -n ldap exec ldap-0 -c slapd -- df -h /var/lib/ldap
Filesystem Size Used Avail Use% Mounted on
/dev/sdd 49G 386M 49G 1% /var/lib/ldap
IOPS averages at 3 IOPS with peaks around 12 IOPS
Bandwidth averages at 45-50 Kb/s with peaks at 2Mb/s
Let's use a Standard ZRS 8Gb disk (ref. https://azure.microsoft.com/en-us/pricing/details/managed-disks/)
Proposed migration plan:
2024-07-05-ldap-data
arm64
scheduling#jenkins-infra
Pod manifests to migrate data using rsync:
---
apiVersion: v1
kind: Pod
metadata:
name: migrate-volume
namespace: ldap
spec:
securityContext:
runAsUser: 0
runAsGroup: 0
fsGroup: 0
affinity:
podAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: statefulset.kubernetes.io/pod-name
operator: In
values:
- ldap-0
topologyKey: kubernetes.io/hostname
containers:
- image: jenkinsciinfra/packaging:latest
name: migrate-volume-script
command: ["rsync"]
args: ["-v", "-a", "--delete", "/src/", "/dest/"]
volumeMounts:
- mountPath: /src
name: old
readOnly: true
- mountPath: /dest
name: new
nodeSelector:
kubernetes.io/arch: amd64
restartPolicy: Never
volumes:
- name: old
persistentVolumeClaim:
claimName: ldap-data
- name: new
persistentVolumeClaim:
claimName: ldap-jenkins-io-data
Update:
arm64
with its new data disksarm64
image exists but it is buggy: the tini
binary in it is x86 (bug fix)arm64
image, but our custom theme image does not (todo)Update:
arm64
. But authentications have weird behaviors: I'm not seen as "Admin" in accountapp anymore and ci.jenkins.io auth. says "auth error". No logs difference on LDAP side between x86 and arm64
except the message mdb_equality_candidates: (member) not indexed
only present on x86 (when it is working).Update:
mirrorbits
, as the arm64
most probably will appear with a 1.x version, we should resume @lemeurherve's work in https://github.com/jenkins-infra/docker-mirrorbits/pull/18 where we could build the image on both amd64 and arm64, with custom build of the 0.5.1 tag of mirrorbits
Update:
We see a visible cost decrease thanks to:
arm64
(x86 node pool is now 2 node instead of 3):Update:
keycloak
is now running on arm64
nodes 🥳
Update:
* LDAP is now starting with `arm64`. But authentications have weird behaviors: I'm not seen as "Admin" in accountapp anymore and ci.jenkins.io auth. says "auth error". No logs difference on LDAP side between x86 and `arm64` except the message `mdb_equality_candidates: (member) not indexed` only present on x86 (when it is working).
seems to say that the indexing is happening on x86 and not on arm64 (https://github.com/cveda/cveda_databank/issues/1)
for reminder: we got a mock ldap in our repositories : https://github.com/jenkins-infra/mock-ldap
I did try to launch the ldap 1.1.1 on my ARM M1 machine and got :
Status: Downloaded newer image for jenkinsciinfra/ldap:1.1.1
qemu-x86_64: Could not open '/lib64/ld-linux-x86-64.so.2': No such file or directory
Update:
Update: We are the proud owners of an arm64
image of mirrorbits \o/ Let's move mirrorbits to arm64
!
Update: We are the proud owners of an
arm64
image of mirrorbits \o/ Let's move mirrorbits toarm64
!
Update:
arm64
stays at 3 nodes
Service(s)
Azure
Summary
following https://github.com/jenkins-infra/helpdesk/issues/3619 we still have :
Reproduction steps
No response