Closed DnPlas closed 2 years ago
ca-scribner
commented
2 years ago I am unable to reproduce on microk8s 1.22, ckf-1.6/beta goes to active for me and has an attached PVC.
I wonder if this is a charmed k8s thing. Do you have a default storage class? If you haven't already, try inspecting the pvcs and storageclass and see if there's something wrong there
ca-scribner
commented
2 years ago Yeah this works for me as well in microk8s 1.23. This feels like something specific to charmed k8s. If it is a storage class thing though I have no idea why some minios would work and others would not, unless something has changed in juju?
DnPlas
commented
2 years ago I checked the storage class, nothing seems off.
```log
ubuntu@charm-dev:~$ juju status --storage
Model Controller Cloud/Region Version SLA Timestamp
minio-test juju-aws charmedk8s/default 2.9.33 unsupported 14:06:21-05:00
App Version Status Scale Charm Channel Rev Address Exposed Message
minio res:oci-image@1755999 waiting 1 minio ckf-1.6/beta 95 10.152.183.244 no waiting for container
Unit Workload Agent Address Ports Message
minio/0* waiting idle waiting for container
Storage Unit Storage ID Type Mountpoint Size Status Message
minio/0 minio-data/0 filesystem pending
ubuntu@charm-dev:~$ kubectl get storageclass
NAME PROVISIONER RECLAIMPOLICY VOLUMEBINDINGMODE ALLOWVOLUMEEXPANSION AGE
cdk-ebs kubernetes.io/aws-ebs Delete WaitForFirstConsumer false 25m
ubuntu@charm-dev:~$ kubectl get pvc -A
NAMESPACE NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE
minio-test minio-data-7cc89ba9-minio-0 Pending cdk-ebs 39s
ubuntu@charm-dev:~$ kubectl describe pvc minio-data-7cc89ba9-minio-0 -nminio-test
Name: minio-data-7cc89ba9-minio-0
Namespace: minio-test
StorageClass: cdk-ebs
Status: Pending
Volume:
Labels: app.kubernetes.io/managed-by=juju
app.kubernetes.io/name=minio
storage.juju.is/name=minio-data
Annotations: controller.juju.is/id: da23ae37-5f6d-444d-8c5c-2fba9890bf22
juju-storage-owner: minio
model.juju.is/id: bb0e61b7-1b5f-4e4a-8f70-8490d70b1498
storage.juju.is/name: minio-data
Finalizers: [kubernetes.io/pvc-protection]
Capacity:
Access Modes:
VolumeMode: Filesystem
Used By:
ubuntu@charm-dev:~$ juju debug-log --replay
controller-0: 14:05:14 INFO juju.worker.apicaller [bb0e61] "machine-0" successfully connected to "localhost:17070"
controller-0: 14:05:14 INFO juju.worker.logforwarder config change - log forwarding not enabled
controller-0: 14:05:14 INFO juju.worker.logger logger worker started
controller-0: 14:05:14 INFO juju.worker.pruner.statushistory status history config: max age: 336h0m0s, max collection size 5120M for minio-test (bb0e61b7-1b5f-4e4a-8f70-8490d70b1498)
controller-0: 14:05:14 INFO juju.worker.pruner.action status history config: max age: 336h0m0s, max collection size 5120M for minio-test (bb0e61b7-1b5f-4e4a-8f70-8490d70b1498)
controller-0: 14:05:36 INFO juju.worker.caasapplicationprovisioner.runner start "minio"
controller-0: 14:05:39 INFO juju.worker.caasprovisioner started operator for application "minio"
application-minio: 14:05:41 INFO juju.cmd running jujud [2.9.33 e83d2a73f904080c5cdf4aaed2821abd4f58253a gc go1.18.5]
application-minio: 14:05:41 DEBUG juju.cmd args: []string{"/var/lib/juju/tools/jujud", "caasoperator", "--application-name=minio", "--debug"}
application-minio: 14:05:41 DEBUG juju.agent read agent config, format "2.0"
application-minio: 14:05:41 INFO juju.worker.upgradesteps upgrade steps for 2.9.33 have already been run.
application-minio: 14:05:41 INFO juju.cmd.jujud caas operator application-minio start (2.9.33 [gc])
application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-23 19:05:41.080298513 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "clock" manifold worker started at 2022-08-23 19:05:41.081384086 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrade-steps-gate" manifold worker started at 2022-08-23 19:05:41.081609017 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.introspection introspection worker listening on "@jujud-application-minio"
application-minio: 14:05:41 DEBUG juju.worker.dependency "agent" manifold worker started at 2022-08-23 19:05:41.081759277 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker completed successfully
application-minio: 14:05:41 DEBUG juju.worker.introspection stats worker now serving
application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-23 19:05:41.090215139 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.apicaller connecting with old password
application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrade-steps-flag" manifold worker started at 2022-08-23 19:05:41.092658526 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "api-config-watcher" manifold worker started at 2022-08-23 19:05:41.092976327 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "migration-fortress" manifold worker started at 2022-08-23 19:05:41.104384422 +0000 UTC
application-minio: 14:05:41 DEBUG juju.api successfully dialed "wss://172.31.17.77:17070/model/bb0e61b7-1b5f-4e4a-8f70-8490d70b1498/api"
application-minio: 14:05:41 INFO juju.api connection established to "wss://172.31.17.77:17070/model/bb0e61b7-1b5f-4e4a-8f70-8490d70b1498/api"
application-minio: 14:05:41 INFO juju.worker.apicaller [bb0e61] "application-minio" successfully connected to "172.31.17.77:17070"
application-minio: 14:05:41 DEBUG juju.api RPC connection died
application-minio: 14:05:41 DEBUG juju.worker.dependency "api-caller" manifold worker completed successfully
application-minio: 14:05:41 DEBUG juju.worker.apicaller connecting with old password
application-minio: 14:05:41 DEBUG juju.api successfully dialed "wss://3.101.105.248:17070/model/bb0e61b7-1b5f-4e4a-8f70-8490d70b1498/api"
application-minio: 14:05:41 INFO juju.api connection established to "wss://3.101.105.248:17070/model/bb0e61b7-1b5f-4e4a-8f70-8490d70b1498/api"
application-minio: 14:05:41 INFO juju.worker.apicaller [bb0e61] "application-minio" successfully connected to "3.101.105.248:17070"
application-minio: 14:05:41 DEBUG juju.worker.dependency "api-caller" manifold worker started at 2022-08-23 19:05:41.147930424 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker completed successfully
application-minio: 14:05:41 DEBUG juju.worker.dependency "caas-units-manager" manifold worker started at 2022-08-23 19:05:41.157414467 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrader" manifold worker started at 2022-08-23 19:05:41.158587465 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "log-sender" manifold worker started at 2022-08-23 19:05:41.158667493 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "migration-minion" manifold worker started at 2022-08-23 19:05:41.158725745 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker started at 2022-08-23 19:05:41.158782836 +0000 UTC
application-minio: 14:05:41 DEBUG juju.worker.dependency "upgrade-steps-runner" manifold worker completed successfully
application-minio: 14:05:41 DEBUG juju.worker.dependency "migration-inactive-flag" manifold worker started at 2022-08-23 19:05:41.160326745 +0000 UTC
application-minio: 14:05:41 INFO juju.worker.caasupgrader abort check blocked until version event received
application-minio: 14:05:41 DEBUG juju.worker.caasupgrader current agent binary version: 2.9.33
application-minio: 14:05:41 INFO juju.worker.caasupgrader unblocking abort check
application-minio: 14:05:41 INFO juju.worker.migrationminion migration phase is now: NONE
application-minio: 14:05:41 DEBUG juju.worker.logger initial log config: "
DnPlas
commented
2 years ago After closer inspection to minio (in particular the minio StatefulSet), the following appears to be the reason why the charm cannot be deployed.
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal SuccessfulCreate 54m statefulset-controller create Claim minio-data-7cc89ba9-minio-0 Pod minio-0 in StatefulSet minio success
Warning FailedCreate 2m21s (x28 over 54m) statefulset-controller create Pod minio-0 in StatefulSet minio failed error: Pod "minio-0" is invalid: spec.containers[0].securityContext.privileged: Forbidden: disallowed by cluster policyTurns out the kubernetes.securityContext.privileged setting, added by this commit is conflicting with my cluster. Turning that to false is a good workaround.
natalian98
commented
2 years ago On microk8s the kube-apiserver is started with --allow-privileged=true by default which allows for that security context, while it's disabled by default in charmed k8s.
I wonder if it's required for the recently added SSL support. If so, allow-privileged config option should be set to true in kubernetes-master. But otherwise we should remove the security context as you explained in #70.
@jardon is there a reason for keeping it?
DnPlas
commented
2 years ago @natalian98 I have asked @jardon offline and he agreed there is no need for the privileged setting. We can close this issue with #70 . Thanks for checking!
Observed behaviour
minio
ckf-1.6/betahangs in aWaitingStatusfor a long time and the storage that is attached to the unit remains in apendingstatus also. This causes minio to never be active.Steps to reproduce
Environment
Workaround
Remove the application and deploy an older version