jobservice CrashLoopBackOff -- redis volume permission issue.

[ruanshudong]

tars-harbor-redis is running, but jobservice is CrashLoopBackOff.

kubectl logs tars-harbor-jobservice-578f7b68bd-8nv5f

2019-12-31T01:55:00Z [DEBUG] [/common/rbac/casbin.go:97]: Starting regexp store purge in 1m0s 2019-12-31T01:55:00Z [INFO] [/replication/adapter/native/adapter.go:42]: the factory for adapter docker-registry registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/harbor/adapter.go:40]: the factory for adapter harbor registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/dockerhub/adapter.go:25]: Factory for adapter docker-hub registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/huawei/huawei_adapter.go:27]: the factory of Huawei adapter was registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/googlegcr/adapter.go:29]: the factory for adapter google-gcr registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/awsecr/adapter.go:47]: the factory for adapter aws-ecr registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/azurecr/adapter.go:15]: Factory for adapter azure-acr registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/aliacr/adapter.go:26]: the factory for adapter ali-acr registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/jfrog/adapter.go:30]: the factory of jfrog artifactory adapter was registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/quayio/adapter.go:34]: the factory of Quay.io adapter was registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/helmhub/adapter.go:30]: the factory for adapter helm-hub registered 2019-12-31T01:55:00Z [INFO] [/replication/adapter/gitlab/adapter.go:19]: the factory for adapter gitlab registered 2019-12-31T01:55:00Z [INFO] [/common/config/store/driver/rest.go:24]: get configuration from url: http://tars-harbor-core/api/internal/configurations 2019-12-31T01:55:00Z [INFO] [/jobservice/logger/sweeper_controller.go:97]: 0 outdated log entries are sweepped by sweeper sweeper.FileSweeper 2019-12-31T01:55:00Z [INFO] [/common/dao/base.go:84]: Registering database: type-PostgreSQL host-tars-harbor-database port-5432 databse-registry sslmode-"disable" 2019-12-31T01:55:00Z [INFO] [/common/dao/base.go:89]: Register database completed 2019-12-31T01:55:00Z [INFO] [/jobservice/migration/manager.go:111]: No migration needed 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job replication.Scheduler with name IMAGE_REPLICATE 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job retention.Job with name RETENTION 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job notification.WebhookJob with name WEBHOOK 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job sample.Job with name DEMO 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job scan.Job with name IMAGE_SCAN 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job all.Job with name IMAGE_SCAN_ALL 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job gc.GarbageCollector with name IMAGE_GC 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job replication.Replication with name REPLICATION 2019-12-31T01:55:00Z [INFO] [/jobservice/worker/cworker/c_worker.go:419]: Register job scheduler.PeriodicJob with name SCHEDULER 2019-12-31T01:56:00Z [DEBUG] [/common/rbac/casbin.go:102]: Starting regexp store purge in 24h0m0s 2019-12-31T01:56:35Z [FATAL] [/jobservice/main.go:80]: load and run worker error: connect to redis server timeout: MISCONF Redis is configured to save RDB snapshots, but it is currently not able to persist on disk. Commands that may modify the data set are disabled, because this instance is configured to report errors during writes if RDB snapshotting fails (stop-writes-on-bgsave-error option). Please check the Redis logs for details about the RDB error.

[ruanshudong]

after all pod run: kubectl logs tars-harbor-redis-0 -n harbor

39:C 31 Dec 02:23:48.030 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis) for saving: Permission denied

and then job service crash

[ruanshudong]

i enter redis pod, i find the user of the redis pod is the 'redis', but the user of /var/lib/redis is root??

[secret104278]

Same issue here. I think there should be init container to make sure the permission of volume is correct like https://github.com/goharbor/harbor-helm/pull/412

[reasonerjt]

@secret104278 @ruanshudong is it because the fsGroup is not supported by your fs?

Could you let me know what fs you are using?

[secret104278]

@secret104278 @ruanshudong is it because the fsGroup is not supported by your fs?

Could you let me know what fs you are using?

Yes, my fs is cephfs and the fsGroup is not supported.

[davidhoelzel]

Same Problem here with longhorn (ext4 volume), but permissions are a bit different in the container: drwxrwsr-x 2 root 10000 4096 2020-10-15 06:30 redis Error is the same: Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis) for saving: Permission denied

[davidhoelzel]

Issue seems to occur after some time: 1:M 12 Nov 08:15:48.892 DB loaded from disk: 0.012 seconds 1:M 12 Nov 08:15:48.892 Ready to accept connections 1:M 12 Nov 08:20:49.092 10 changes in 300 seconds. Saving... 1:M 12 Nov 08:20:49.093 Background saving started by pid 9 9:C 12 Nov 08:20:49.150 DB saved on disk 9:C 12 Nov 08:20:49.151 RDB: 8 MB of memory used by copy-on-write 1:M 12 Nov 08:20:49.194 Background saving terminated with success 1:M 12 Nov 08:25:50.072 10 changes in 300 seconds. Saving... 1:M 12 Nov 08:25:50.073 Background saving started by pid 10 10:C 12 Nov 08:25:50.141 DB saved on disk 10:C 12 Nov 08:25:50.142 RDB: 8 MB of memory used by copy-on-write 1:M 12 Nov 08:25:50.175 Background saving terminated with success 1:M 12 Nov 08:30:51.009 10 changes in 300 seconds. Saving... 1:M 12 Nov 08:30:51.010 Background saving started by pid 11 11:C 12 Nov 08:30:51.046 DB saved on disk 11:C 12 Nov 08:30:51.047 RDB: 8 MB of memory used by copy-on-write 1:M 12 Nov 08:30:51.111 Background saving terminated with success 1:M 12 Nov 08:35:52.034 10 changes in 300 seconds. Saving... 1:M 12 Nov 08:35:52.035 Background saving started by pid 12 12:C 12 Nov 08:35:52.036 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis) for saving: Permission denied 1:M 12 Nov 08:35:52.135 # Background saving error 1:M 12 Nov 08:35:58.061 10 changes in 300 seconds. Saving... 1:M 12 Nov 08:35:58.062 Background saving started by pid 13 13:C 12 Nov 08:35:58.062 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis) for saving: Permission denied 1:M 12 Nov 08:35:58.163 # Background saving error 1:M 12 Nov 08:36:04.082 10 changes in 300 seconds. Saving... 1:M 12 Nov 08:36:04.083 * Background saving started by pid 14 14:C 12 Nov 08:36:04.084 # Failed opening the RDB file dump.rdb (in server root dir /var/lib/redis) for saving: Permission denied 1:M 12 Nov 08:36:04.183 # Background saving error

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.