search

RedisLabs / redis-enterprise-k8s-docs

151 stars 89 forks source link

Cannot be started “statefulset.apps/rec. ” #249

Open zhenyu-lan opened 1 year ago

zhenyu-lan commented 1 year ago 
[zhenyu.lan-a@----s` ~]$ kubectl  get all
NAME                                             READY   STATUS    RESTARTS   AGE
pod/rec-0                                        1/2     Running   0          2m37s
pod/rec-services-rigger-6ddcd5796f-5tcjc         1/1     Running   0          4d18h
pod/redis-enterprise-operator-69d566468c-m2p6k   3/3     Running   4          4d18h

NAME                TYPE           CLUSTER-IP                         EXTERNAL-IP   PORT(S)             AGE
service/admission   ClusterIP      240b:c0e0:202:554f:b414:2:0:cd4e   <none>        443/TCP             4d18h
service/rec         ClusterIP      None                               <none>        9443/TCP,8001/TCP   4d18h
service/rec-prom    ClusterIP      None                               <none>        8070/TCP            4d18h
service/rec-ui      LoadBalancer   240b:c0e0:202:554f:b414:2:0:dfc2   <pending>     8443:31981/TCP      4d18h

NAME                                        READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/rec-services-rigger         1/1     1            1           4d18h
deployment.apps/redis-enterprise-operator   1/1     1            1           4d18h

NAME                                                   DESIRED   CURRENT   READY   AGE
replicaset.apps/rec-services-rigger-6ddcd5796f         1         1         1       4d18h
replicaset.apps/redis-enterprise-operator-69d566468c   1         1         1       4d18h

NAME                   READY   AGE
statefulset.apps/rec   0/3     4d18h

[zhenyu.lan-a@uhn5---t-s ~]$ kubectl  logs rec-0 redis-enterprise-node -f
Generating RSA private key, 4096 bit long modulus (2 primes)
.....++++
................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
140321274876352:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = Admin UI, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
..................................................................................++++
.......................................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
140394671956416:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = Cluster API, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
........................................................................................................++++
........................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139849269596608:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = redisdb, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
.................................................................................................................................................................................................................................................................................++++
........................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139763628327360:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = metrics exporter, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
.................................................++++
...................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139991659815360:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = rec-0.rec.agile.svc.cluster.local, OU = syncer, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
...++++
....................................................................................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
140530423501248:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = node.mesh_ca_signed, OU = mesh, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
........++++
..++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139994297110976:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = node.gossip_ca_signed, OU = gossip, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
...................................................................++++
........................................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
139931683258816:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = node.ccs_internode_encryption, OU = ccs_internode_encryption, O = RedisLabs Enterprise Cluster
Getting Private key
Generating RSA private key, 4096 bit long modulus (2 primes)
...........................................................................................................................................................................................................................++++
...................................................++++
e is 65537 (0x010001)
Can't load /opt/redislabs/.rnd into RNG
140524452495808:error:2406F079:random number generator:RAND_load_file:Cannot open file:../crypto/rand/randfile.c:88:Filename=/opt/redislabs/.rnd
Signature ok
subject=CN = node.data_internode_encryption, OU = Data plane internode encryption, O = RedisLabs Enterprise Cluster
Getting Private key
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'cm', '--org', 'Admin UI']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'api', '--org', 'Cluster API']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'proxy', '--org', 'redisdb']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'metrics_exporter', '--org', 'metrics exporter']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'syncer', '--org', 'syncer']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'mesh_ca_signed', '--org', 'mesh', '--CN', 'node.mesh_ca_signed', '--days', '90']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'gossip_ca_signed', '--org', 'gossip', '--CN', 'node.gossip_ca_signed', '--days', '90']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'ccs_internode_encryption', '--org', 'ccs_internode_encryption', '--CN', 'node.ccs_internode_encryption', '--days', '90']
Run command: ['/opt/redislabs/sbin/generate_cert.sh', '--type', 'data_internode_encryption', '--org', 'Data plane internode encryption', '--CN', 'node.data_internode_encryption', '--days', '90']
Done.
/opt/redislabs/sbin/supervisord_prestart_script.sh: line 42: type: getenforce: not found
2022-10-17 02:15:41,631 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_cm_server.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_crdb_coordinator.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_crdb_worker.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_mdns_server.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_pdns_server.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_saslauthd.conf" during parsing
2022-10-17 02:15:41,633 INFO Included extra file "/etc/opt/redislabs/supervisord.conf.d/supervisord_stats_archiver.conf" during parsing
2022-10-17 02:15:41,698 INFO __main__ MainThread: Waiting for REST API to be accessible
2022-10-17 02:15:41,698 INFO RPC interface 'supervisor' initialized
2022-10-17 02:15:41,699 CRIT Server 'unix_http_server' running without any HTTP authentication checking
2022-10-17 02:15:41,699 INFO supervisord started with pid 173
2022-10-17 02:15:42,703 INFO spawned: 'ccs' with pid 182
2022-10-17 02:15:42,707 INFO spawned: 'cluster_wd' with pid 183
2022-10-17 02:15:42,712 INFO spawned: 'envoy' with pid 184
2022-10-17 02:15:42,716 INFO spawned: 'envoy_control_plane' with pid 185
2022-10-17 02:15:42,720 INFO spawned: 'gossip_envoy' with pid 186
2022-10-17 02:15:42,725 INFO spawned: 'heartbeatd' with pid 187
2022-10-17 02:15:42,728 INFO spawned: 'ldap_agent_mgr' with pid 188
2022-10-17 02:15:42,730 INFO spawned: 'alert_mgr' with pid 189
2022-10-17 02:15:42,733 INFO spawned: 'cnm_exec' with pid 190
2022-10-17 02:15:42,776 INFO spawned: 'cnm_http' with pid 197
2022-10-17 02:15:42,780 INFO spawned: 'crdb_coordinator' with pid 198
2022-10-17 02:15:42,872 INFO spawned: 'crdb_worker_0' with pid 199
2022-10-17 02:15:42,876 INFO spawned: 'crdb_worker_1' with pid 206
2022-10-17 02:15:42,880 INFO spawned: 'pdns_ccs_backend' with pid 214
2022-10-17 02:15:42,973 INFO spawned: 'redis_mgr' with pid 215
2022-10-17 02:15:42,978 INFO spawned: 'resource_mgr' with pid 216
2022-10-17 02:15:43,071 INFO spawned: 'rl_info_provider' with pid 217
2022-10-17 02:15:43,118 INFO spawned: 'stats_archiver' with pid 219
2022-10-17 02:15:43,174 INFO spawned: 'cm_server' with pid 220
2022-10-17 02:15:43,178 INFO spawned: 'dmcproxy' with pid 222
2022-10-17 02:15:43,271 INFO spawned: 'mdns_server' with pid 223
2022-10-17 02:15:43,274 INFO spawned: 'pdns_server' with pid 224
2022-10-17 02:15:43,371 INFO spawned: 'saslauthd' with pid 225
2022-10-17 02:15:43,400 INFO spawned: 'sentinel_service' with pid 226
2022-10-17 02:15:43,474 INFO spawned: 'job_scheduler' with pid 227
2022-10-17 02:15:43,573 INFO spawned: 'metrics_exporter' with pid 228
2022-10-17 02:15:43,577 INFO spawned: 'node_wd' with pid 229
2022-10-17 02:15:43,677 INFO spawned: 'start_redis_servers' with pid 232
2022-10-17 02:15:43,782 INFO success: ccs entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: cluster_wd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: envoy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: envoy_control_plane entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: gossip_envoy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: heartbeatd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: ldap_agent_mgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: alert_mgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: cnm_exec entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: cnm_http entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,783 INFO success: crdb_coordinator entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,871 INFO success: crdb_worker_0 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:43,872 INFO success: start_redis_servers entered RUNNING state, process has stayed up for > than 0 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: crdb_worker_1 entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: pdns_ccs_backend entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: redis_mgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: resource_mgr entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,071 INFO success: rl_info_provider entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,072 INFO success: stats_archiver entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,175 INFO success: cm_server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,175 INFO success: dmcproxy entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,272 INFO success: mdns_server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,273 INFO success: pdns_server entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,680 INFO success: saslauthd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,680 INFO success: sentinel_service entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,680 INFO success: metrics_exporter entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:44,680 INFO success: node_wd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
2022-10-17 02:15:53,972 INFO success: job_scheduler entered RUNNING state, process has stayed up for > than 10 seconds (startsecs)
2022-10-17 02:18:03,445 INFO __main__ MainThread: Done, moving to bootstrapping
2022-10-17 02:18:03,447 INFO __main__ MainThread: Bootstrapping node with action 'None'
2022-10-17 02:18:03,448 INFO __main__ MainThread: Done
laurentdroin commented 1 year ago

There could be many reasons for this. Can you generate a diagnostic package using the log collector script (https://docs.redis.com/latest/kubernetes/logs/collect-logs/) and open a Support ticket with Redis?