Open FTS152 opened 1 year ago
Thanks for reporting this!
I personally haven't any first hand experience with kubectl rollout restart
and it's not part of our testing suite. So I suspect that it a large portion of the cause of the problem. The approach it takes to restarting the pods may be triggering an issue with the order of operations used to bring up the pods or perhaps there's an additional health check we need to prevent it from acting upon pods.
That's all speculation so far. First question - can we assume things were working OK prior to running the restart command? Next, the persistent tdb files should be stored on a PV. Can you help us confirm that the pods are mounting a persistent volume for the relevant paths under /var/lib - a full kubectl get -o yaml
of the pods should suffice there.
Finally, it would be helpful to see some of logs from the init containers from both before and after kubectl rollout restart
is run.
Thanks!
Thanks for the reply! I'm still researching what is causing the tdb files to disappear after rollout restart. It seems that this problem only happens when restarting a Samba server with CTDB feature enabled. Maybe it is the difference in the behavior of rollout restart in deployment
and statefulset
that causes this problem.
Samba server works fine before kubectl rollout restart
, one can connect to the server and read/write data from/to the share normally:
root@fts152-QV99:~# smbclient '//10.20.91.127/smbshare1' -U smbuser1 -p 445
Enter WORKGROUP\smbuser1's password:
Try "help" to get a list of possible commands.
smb: \> ls
. D 0 Tue Oct 11 16:03:32 2022
.. D 0 Thu Oct 13 10:07:11 2022
104857600 blocks of size 1024. 104857600 blocks available
smb: \> mkdir test
smb: \> cd test
smb: \test\> put /root/myfile myfile
putting file /root/myfile as \test\myfile (30684.3 kb/s) (average 30684.3 kb/s)
smb: \test\> ls
. D 0 Thu Oct 13 10:19:15 2022
.. D 0 Thu Oct 13 10:19:02 2022
myfile A 1073741824 Thu Oct 13 10:19:49 2022
104857600 blocks of size 1024. 104857600 blocks available
smb: \test\>
I checked samba with kubectl describe
and found that /var/lib/samba mounted from samba-state-dir
, but samba-state-dir
is not a PV. Could this be the problem? :
samba:
Image: quay.io/samba.org/samba-server:latest
Port: 445/TCP
Host Port: 0/TCP
Command:
samba-container
Args:
run
smbd
--setup=users
--setup=smb_ctdb
Liveness: tcp-socket :445 delay=0s timeout=1s period=10s #success=1 #failure=3
Readiness: tcp-socket :445 delay=0s timeout=1s period=10s #success=1 #failure=3
Environment:
SAMBA_CONTAINER_ID: smbshare1
SAMBACC_CONFIG: /etc/container-config/config.json:/etc/container-users/users.json
SAMBA_POD_NAME: (v1:metadata.name)
SAMBA_POD_NAMESPACE: (v1:metadata.namespace)
Mounts:
(...)
/var/lib/samba from samba-state-dir (rw)
/var/run/ctdb from ctdb-sockets (rw)
(...)
Volumes:
smbshare1-pvc-smb:
Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
ClaimName: smbshare1-pvc
ReadOnly: false
samba-container-config:
Type: ConfigMap (a volume populated by a ConfigMap)
Name: smbshare1
Optional: false
samba-state-dir:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
the full YAML file for samba Statefulset is as below:
root@k8s-test4:~/rook# kubectl get statefulset smbshare1 -o yaml -n samba-operator-system
apiVersion: apps/v1
kind: StatefulSet
metadata:
creationTimestamp: "2022-10-13T02:04:07Z"
generation: 1
labels:
app: samba
app.kubernetes.io/component: smbd
app.kubernetes.io/instance: samba-smbshare1
app.kubernetes.io/managed-by: samba-operator
app.kubernetes.io/name: samba
app.kubernetes.io/part-of: samba
samba-operator.samba.org/service: smbshare1
name: smbshare1
namespace: samba-operator-system
ownerReferences:
- apiVersion: samba-operator.samba.org/v1alpha1
blockOwnerDeletion: true
controller: true
kind: SmbShare
name: smbshare1
uid: 94a42780-a8f8-4ae1-b9b1-debac6bd5520
resourceVersion: "11256600"
uid: f3d02b0c-3cfa-47d6-bb19-cd2ad946f251
spec:
podManagementPolicy: OrderedReady
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: samba
app.kubernetes.io/component: smbd
app.kubernetes.io/instance: samba-smbshare1
app.kubernetes.io/managed-by: samba-operator
app.kubernetes.io/name: samba
app.kubernetes.io/part-of: samba
samba-operator.samba.org/service: smbshare1
serviceName: ""
template:
metadata:
annotations:
kubectl.kubernetes.io/default-container: samba
kubectl.kubernetes.io/default-logs-container: samba
creationTimestamp: null
labels:
app: samba
app.kubernetes.io/component: smbd
app.kubernetes.io/instance: samba-smbshare1
app.kubernetes.io/managed-by: samba-operator
app.kubernetes.io/name: samba
app.kubernetes.io/part-of: samba
samba-operator.samba.org/service: smbshare1
spec:
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: samba-operator.samba.org/service
operator: In
values:
- smbshare1
topologyKey: kubernetes.io/hostname
containers:
- args:
- run
- ctdbd
- --setup=smb_ctdb
- --setup=ctdb_config
- --setup=ctdb_etc
- --setup=ctdb_nodes
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBACC_CTDB
value: ctdb-is-experimental
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: IfNotPresent
name: ctdb
readinessProbe:
exec:
command:
- samba-container
- check
- ctdb-nodestatus
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /etc/container-users
name: users-config
- mountPath: /etc/ctdb
name: ctdb-config
- mountPath: /var/lib/ctdb/persistent
name: ctdb-persistent
- mountPath: /var/lib/ctdb/volatile
name: ctdb-volatile
- mountPath: /var/run/ctdb
name: ctdb-sockets
- mountPath: /var/lib/ctdb/shared
name: smbshare1-state-ctdb
- args:
- ctdb-manage-nodes
- --hostname=$(HOSTNAME)
- --take-node-number-from-hostname=after-last-dash
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBACC_CTDB
value: ctdb-is-experimental
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: IfNotPresent
name: ctdb-manage-nodes
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /etc/container-users
name: users-config
- mountPath: /etc/ctdb
name: ctdb-config
- mountPath: /var/run/ctdb
name: ctdb-sockets
- mountPath: /var/lib/ctdb/shared
name: smbshare1-state-ctdb
- args:
- run
- smbd
- --setup=users
- --setup=smb_ctdb
command:
- samba-container
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 445
timeoutSeconds: 1
name: samba
ports:
- containerPort: 445
name: smb
protocol: TCP
readinessProbe:
failureThreshold: 3
periodSeconds: 10
successThreshold: 1
tcpSocket:
port: 445
timeoutSeconds: 1
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /mnt/94a42780-a8f8-4ae1-b9b1-debac6bd5520
name: smbshare1-pvc-smb
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /var/lib/samba
name: samba-state-dir
- mountPath: /etc/ctdb
name: ctdb-config
- mountPath: /var/lib/ctdb/persistent
name: ctdb-persistent
- mountPath: /var/lib/ctdb/volatile
name: ctdb-volatile
- mountPath: /var/run/ctdb
name: ctdb-sockets
- mountPath: /var/lib/ctdb/shared
name: smbshare1-state-ctdb
- mountPath: /etc/container-users
name: users-config
- args:
- update-config
- --watch
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: Always
name: watch-update-config
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /mnt/94a42780-a8f8-4ae1-b9b1-debac6bd5520
name: smbshare1-pvc-smb
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /var/lib/samba
name: samba-state-dir
- mountPath: /etc/ctdb
name: ctdb-config
- mountPath: /var/lib/ctdb/persistent
name: ctdb-persistent
- mountPath: /var/lib/ctdb/volatile
name: ctdb-volatile
- mountPath: /var/run/ctdb
name: ctdb-sockets
- mountPath: /var/lib/ctdb/shared
name: smbshare1-state-ctdb
- mountPath: /etc/container-users
name: users-config
dnsPolicy: ClusterFirst
initContainers:
- args:
- --skip-if-file=/var/lib/ctdb/shared/nodes
- init
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: IfNotPresent
name: init
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /etc/container-users
name: users-config
- mountPath: /var/lib/samba
name: samba-state-dir
- mountPath: /var/lib/ctdb/shared
name: smbshare1-state-ctdb
- args:
- ensure-share-paths
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: IfNotPresent
name: ensure-share-paths
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /etc/container-users
name: users-config
- mountPath: /var/lib/samba
name: samba-state-dir
- mountPath: /mnt/94a42780-a8f8-4ae1-b9b1-debac6bd5520
name: smbshare1-pvc-smb
- args:
- ctdb-migrate
- --dest-dir=/var/lib/ctdb/persistent
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBACC_CTDB
value: ctdb-is-experimental
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: IfNotPresent
name: ctdb-migrate
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /etc/container-users
name: users-config
- mountPath: /var/lib/samba
name: samba-state-dir
- mountPath: /var/lib/ctdb/shared
name: smbshare1-state-ctdb
- mountPath: /etc/ctdb
name: ctdb-config
- mountPath: /var/lib/ctdb/persistent
name: ctdb-persistent
- args:
- ctdb-set-node
- --hostname=$(HOSTNAME)
- --take-node-number-from-hostname=after-last-dash
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBACC_CTDB
value: ctdb-is-experimental
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: IfNotPresent
name: ctdb-set-node
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /etc/container-users
name: users-config
- mountPath: /var/lib/samba
name: samba-state-dir
- mountPath: /var/lib/ctdb/shared
name: smbshare1-state-ctdb
- mountPath: /etc/ctdb
name: ctdb-config
- args:
- ctdb-must-have-node
- --hostname=$(HOSTNAME)
- --take-node-number-from-hostname=after-last-dash
env:
- name: SAMBA_CONTAINER_ID
value: smbshare1
- name: SAMBACC_CONFIG
value: /etc/container-config/config.json:/etc/container-users/users.json
- name: SAMBA_POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBA_POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: HOSTNAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: SAMBACC_CTDB
value: ctdb-is-experimental
image: quay.io/samba.org/samba-server:latest
imagePullPolicy: IfNotPresent
name: ctdb-must-have-node
resources: {}
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /etc/container-config
name: samba-container-config
- mountPath: /etc/container-users
name: users-config
- mountPath: /var/lib/samba
name: samba-state-dir
- mountPath: /var/lib/ctdb/shared
name: smbshare1-state-ctdb
- mountPath: /etc/ctdb
name: ctdb-config
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
shareProcessNamespace: true
terminationGracePeriodSeconds: 30
volumes:
- name: smbshare1-pvc-smb
persistentVolumeClaim:
claimName: smbshare1-pvc
- configMap:
defaultMode: 420
name: smbshare1
name: samba-container-config
- emptyDir: {}
name: samba-state-dir
- emptyDir:
medium: Memory
name: ctdb-config
- emptyDir:
medium: Memory
name: ctdb-persistent
- emptyDir:
medium: Memory
name: ctdb-volatile
- emptyDir:
medium: Memory
name: ctdb-sockets
- name: smbshare1-state-ctdb
persistentVolumeClaim:
claimName: smbshare1-state
- name: users-config
secret:
defaultMode: 420
items:
- key: demousers
path: users.json
secretName: users1
updateStrategy:
rollingUpdate:
partition: 0
type: RollingUpdate
status:
availableReplicas: 3
collisionCount: 0
currentReplicas: 3
currentRevision: smbshare1-689c69d6f4
observedGeneration: 1
readyReplicas: 3
replicas: 3
updateRevision: smbshare1-689c69d6f4
updatedReplicas: 3
Logs of init containers before kubectl rollout restart
:
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c init -n samba-operator-system
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c ctdb-migrate -n samba-operator-system
2022-10-13 03:42:48,590: INFO: Checking for /var/lib/samba/account_policy.tdb
2022-10-13 03:42:48,590: INFO: Converting /var/lib/samba/account_policy.tdb to /var/lib/ctdb/persistent/account_policy.tdb.0 ...
2022-10-13 03:42:48,592: INFO: Checking for /var/lib/samba/private/account_policy.tdb
2022-10-13 03:42:48,592: INFO: Checking for /var/lib/samba/group_mapping.tdb
2022-10-13 03:42:48,592: INFO: Converting /var/lib/samba/group_mapping.tdb to /var/lib/ctdb/persistent/group_mapping.tdb.0 ...
2022-10-13 03:42:48,593: INFO: Checking for /var/lib/samba/private/group_mapping.tdb
2022-10-13 03:42:48,593: INFO: Checking for /var/lib/samba/passdb.tdb
2022-10-13 03:42:48,593: INFO: Checking for /var/lib/samba/private/passdb.tdb
2022-10-13 03:42:48,593: INFO: Converting /var/lib/samba/private/passdb.tdb to /var/lib/ctdb/persistent/passdb.tdb.0 ...
2022-10-13 03:42:48,595: INFO: Checking for /var/lib/samba/registry.tdb
2022-10-13 03:42:48,595: INFO: Converting /var/lib/samba/registry.tdb to /var/lib/ctdb/persistent/registry.tdb.0 ...
2022-10-13 03:42:48,596: INFO: Checking for /var/lib/samba/private/registry.tdb
2022-10-13 03:42:48,597: INFO: Checking for /var/lib/samba/secrets.tdb
2022-10-13 03:42:48,597: INFO: Checking for /var/lib/samba/private/secrets.tdb
2022-10-13 03:42:48,597: INFO: Converting /var/lib/samba/private/secrets.tdb to /var/lib/ctdb/persistent/secrets.tdb.0 ...
2022-10-13 03:42:48,598: INFO: Checking for /var/lib/samba/share_info.td
2022-10-13 03:42:48,598: INFO: Checking for /var/lib/samba/private/share_info.td
2022-10-13 03:42:48,598: INFO: Checking for /var/lib/samba/winbindd_idmap.tdb
2022-10-13 03:42:48,598: INFO: Checking for /var/lib/samba/private/winbindd_idmap.tdb
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c ctdb-set-node -n samba-operator-system
2022-10-13 03:42:49,644: INFO: Determined address for smbshare1-0: 10.42.1.173
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c ctdb-must-have-node -n samba-operator-system
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c ensure-share-paths -n samba-operator-system
2022-10-13 03:42:47,578: INFO: Ensuring share path: /mnt/cfa64779-58dc-4278-9883-2b6477a3f56a
2022-10-13 03:42:47,578: INFO: Updating permissions if needed: /mnt/cfa64779-58dc-4278-9883-2b6477a3f56a
2022-10-13 03:42:47,578: INFO: Using initializing posix permissions handler
Logs of init containers after kubectl rollout restart
:
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c init -n samba-operator-system
Action skipped: skip-if-file: /var/lib/ctdb/shared/nodes exists
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c ctdb-migrate -n samba-operator-system
2022-10-13 03:57:42,866: INFO: Checking for /var/lib/samba/account_policy.tdb
2022-10-13 03:57:42,866: INFO: Checking for /var/lib/samba/private/account_policy.tdb
2022-10-13 03:57:42,866: INFO: Checking for /var/lib/samba/group_mapping.tdb
2022-10-13 03:57:42,866: INFO: Checking for /var/lib/samba/private/group_mapping.tdb
2022-10-13 03:57:42,866: INFO: Checking for /var/lib/samba/passdb.tdb
2022-10-13 03:57:42,866: INFO: Checking for /var/lib/samba/private/passdb.tdb
2022-10-13 03:57:42,866: INFO: Checking for /var/lib/samba/registry.tdb
2022-10-13 03:57:42,867: INFO: Checking for /var/lib/samba/private/registry.tdb
2022-10-13 03:57:42,867: INFO: Checking for /var/lib/samba/secrets.tdb
2022-10-13 03:57:42,867: INFO: Checking for /var/lib/samba/private/secrets.tdb
2022-10-13 03:57:42,867: INFO: Checking for /var/lib/samba/share_info.td
2022-10-13 03:57:42,867: INFO: Checking for /var/lib/samba/private/share_info.td
2022-10-13 03:57:42,867: INFO: Checking for /var/lib/samba/winbindd_idmap.tdb
2022-10-13 03:57:42,867: INFO: Checking for /var/lib/samba/private/winbindd_idmap.tdb
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c ctdb-set-node -n samba-operator-system
2022-10-13 03:57:43,809: INFO: Determined address for smbshare1-0: 10.42.1.174
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c ctdb-must-have-node -n samba-operator-system
2022-10-13 03:57:49,162: INFO: node not yet ready
root@k8s-test4:~/samba-service# kubectl logs smbshare1-0 -c ensure-share-paths -n samba-operator-system
2022-10-13 03:57:41,871: INFO: Ensuring share path: /mnt/cfa64779-58dc-4278-9883-2b6477a3f56a
2022-10-13 03:57:41,871: INFO: Updating permissions if needed: /mnt/cfa64779-58dc-4278-9883-2b6477a3f56a
2022-10-13 03:57:41,871: INFO: Using initializing posix permissions handler
UPDATES: Looks like the user information is mounted successfully in the pod:
[root@smbshare1-1 /]# cat /etc/container-users/users.json
{
"samba-container-config": "v0",
"users": {
"all_entries": [
{
"name": "smbuser1",
"password": "samba"
},
{
"name": "smbuser2",
"password": "samba"
}
]
}
}
And when I execute samba-container import
and samba-container import-users
in every pods after rollout restart, I can login to Samba server normally. Is this expected behavior or I just misunderstood something?
And when I execute
samba-container import
andsamba-container import-users
in every pods after rollout restart, I can login to Samba server normally. Is this expected behavior or I just misunderstood something?
Very interesting! No, this is not intended behavior. I would consider this a bug.
I think what might be happening is that when all the pods restart they are not importing all the configuration data - because some data is expected to persist in the samba db files, but other data like the users/groups stored in /etc are kept in system files and these files are "reset" every time the container is started. The code may be failing to distinguish between configuration data that needs to be recreated on every container vs. configuration data that persists and then simply doesn't bother trying to recreate the former type of configuration data.
In addition, when we test with AD enabled we might miss this because in the AD case the users/groups data is not stored in the containers. One of my teammates @anoopcs9 suggested that we should incorporate the rolling restart into one of test cases. I agree. We should try to fix this issue and create an integration test to verify the fix!
@anoopcs9 this issue may also be more in sambacc if we tightly couple the creation of /etc/passwd and /etc/group with the population of the samba passdb. We may need to make sure that the passdb can be populated independently of the /etc/{passwd,group} files and maybe have an init container that creates the latter every time the pod is started? At the very least it's a good first place to look if we can reproduce the error.
Hi, is there any news regarding this issue?
Unfortunately, no. I don't think anyone on the team has had the time to work on the CTDB support in a while. It's still on our radar though!
@phlogistonjohn I think I figured out where the problem is. In Samba Statefulset, there is a init container to do some initialization works:
initContainers:
- args:
- init
- --skip-if-file=/var/lib/ctdb/shared/nodes
This command includes importing samba config and user data to registry. When performing a rollout restart, the node file is still in state pvc and init container "wrongly" thinks that the initialization process has been done.
Ohh, interesting. That could be it. Can you see if there are other unique states we could key off of to tell when the rollout is occuring vs. a typical bringup?
@phlogistonjohn To my understanding, there isn't a specific field or status condition in k8s api that indicates whether a resource is undergoing a rollout restart. If this method is not feasible, an alternative (but not convenient) way may be manually adding custom labels/annotations to the Statefulset before rollout.
Hi, I am trying to deploy a Samba service with CTDB support on a bare-metal k3s cluster, here is my environment:
I use rook-ceph as my backend storage and metalLB as my load balancer:
I deploy a Samba statefulset with
minClusterSize=3
and so far so good:However when I restart samba by
kubectl rollout restart
without any changes, I cannot login to samba server anymore after rolling update is done:What I figured out are that
account_policy.tdb
,group_mapping.tdb
andregistry.tdb
disappeared in/var/lib/samba
in the pod, and some error messages in log file of samba container:Also samba share information is lost in registry in the pod. Any ideas? Thanks.