search

eliu / openshift-redis-sentinel

The Redis Sentinel cluster template of OpenShift Origin
MIT License
6 stars 11 forks source link

The pod keeps failing to start saying crash loop back off #1

Open Natarajan77 opened 6 years ago

Natarajan77 commented 6 years ago

The pod keeps failing to start saying crash loop back off. Do you know anything about it?

  1. Could not connect to Redis at 172.30.193.44:26379: Host is unreachable Failed to find master.

    1. Fatal error, can't open config file 'sentinel.conf'

eliu commented 6 years ago

Hi dude, Thanks for the feedback.

I have followed the steps in README and re-deploy this sentinel on openshift 3.7.0 cluster again with no errors. See screenshots above: image image

For the question 1, This redis sentinel was meant to communicate within the cluster. I might fail if you try to connect this sentinel cluster outside openshift, like a desktop client for example. If you insist doing this, you can expose the ports by creating a nodePort service, see https://docs.openshift.org/latest/minishift/openshift/exposing-services.html#nodeport-services for more details.

For question 2, I didn't any similar errors in container logs, everything works as expected.

Natarajan77 commented 6 years ago

What i ran into issue is it is not creating sentinel.conf file..can u pls share the yaml file from ur deployment and the order in which u ran stuff

Sent from Yahoo Mail for iPhone

On Sunday, March 18, 2018, 10:16 PM, Hong-Yu Liu notifications@github.com wrote:

Hi dude, Thanks for the feedback.

I have followed the steps in README and re-deploy this sentinel on openshift 3.7.0 cluster again with no errors. See screenshots above:

For the question 1, This redis sentinel was meant to communicate within the cluster. I might fail if you try to connect this sentinel cluster outside openshift, like a desktop client for example. If you insist doing this, you can expose the ports by creating a nodePort service, see https://docs.openshift.org/latest/minishift/openshift/exposing-services.html#nodeport-services for more details.

For question 2, I didn't any similar errors in container logs, everything works as expected.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

eliu commented 6 years ago

Please check out YAML content grabbed from my deployment after oc process ... | oc create -f -

apiVersion: apps.openshift.io/v1
kind: DeploymentConfig
metadata:
  annotations:
    template.alpha.openshift.io/wait-for-ready: 'true'
  creationTimestamp: '2018-03-19T02:26:09Z'
  generation: 2
  name: redis
  namespace: redis-test
  resourceVersion: '4394974'
  selfLink: /apis/apps.openshift.io/v1/namespaces/redis-test/deploymentconfigs/redis
  uid: e2c13a85-2b1c-11e8-8f54-52540035034f
spec:
  replicas: 2
  selector:
    name: redis
  strategy:
    activeDeadlineSeconds: 21600
    recreateParams:
      timeoutSeconds: 600
    resources: {}
    type: Recreate
  template:
    metadata:
      creationTimestamp: null
      labels:
        name: redis
        redis-sentinel: 'true'
    spec:
      containers:
        - image: >-
            eliu/redis-sentinel@sha256:55bcd9ab958b49eb35c4e4589ca042a56f2ce607aacded7f9dbc55bf116c7a1e
          imagePullPolicy: IfNotPresent
          name: redis
          ports:
            - containerPort: 6379
              protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /redis-master-data
              name: redis-data
        - env:
            - name: SENTINEL
              value: 'true'
          image: >-
            eliu/redis-sentinel@sha256:55bcd9ab958b49eb35c4e4589ca042a56f2ce607aacded7f9dbc55bf116c7a1e
          imagePullPolicy: IfNotPresent
          name: redis-sentinel
          ports:
            - containerPort: 26379
              protocol: TCP
          resources: {}
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
          volumeMounts:
            - mountPath: /redis-sentinel-data
              name: redis-sentinel-data
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
        - emptyDir: {}
          name: redis-data
        - emptyDir: {}
          name: redis-sentinel-data
  test: false
  triggers:
    - type: ConfigChange
    - imageChangeParams:
        automatic: true
        containerNames:
          - redis
          - redis-sentinel
        from:
          kind: ImageStreamTag
          name: 'redis-sentinel:latest'
          namespace: redis-test
        lastTriggeredImage: >-
          eliu/redis-sentinel@sha256:55bcd9ab958b49eb35c4e4589ca042a56f2ce607aacded7f9dbc55bf116c7a1e
      type: ImageChange
status:
  availableReplicas: 2
  conditions:
    - lastTransitionTime: '2018-03-19T02:28:28Z'
      lastUpdateTime: '2018-03-19T02:28:28Z'
      message: replication controller "redis-1" successfully rolled out
      reason: NewReplicationControllerAvailable
      status: 'True'
      type: Progressing
    - lastTransitionTime: '2018-03-19T02:29:47Z'
      lastUpdateTime: '2018-03-19T02:29:47Z'
      message: Deployment config has minimum availability.
      status: 'True'
      type: Available
  details:
    causes:
      - type: ConfigChange
    message: config change
  latestVersion: 1
  observedGeneration: 2
  readyReplicas: 2
  replicas: 2
  unavailableReplicas: 0
  updatedReplicas: 2

The stuff i ran list below in order: 

git clone https://github.com/eliu/openshift-redis-sentinel.git
cd openshift-redis-sentinel
oc login -u system:admin
oc project openshift
oc create -f templates/redis-sentinel-template.json

oc login -u developer -p xxx
oc new-project redis-test --display-name="Redis Test"
oc process openshift//redis-sentinel | oc create -f -

# then you just wait for secs to complete initializations...
Natarajan77 commented 6 years ago

Sir my project name is not openshift will it make any difference 

Sent from Yahoo Mail for iPhone

On Sunday, March 18, 2018, 10:37 PM, Hong-Yu Liu notifications@github.com wrote:

Please check out YAML content grabbed from my deployment after oc process ... | oc create -f - apiVersion: apps.openshift.io/v1 kind: DeploymentConfig metadata: annotations: template.alpha.openshift.io/wait-for-ready: 'true' creationTimestamp: '2018-03-19T02:26:09Z' generation: 2 name: redis namespace: redis-test resourceVersion: '4394974' selfLink: /apis/apps.openshift.io/v1/namespaces/redis-test/deploymentconfigs/redis uid: e2c13a85-2b1c-11e8-8f54-52540035034f spec: replicas: 2 selector: name: redis strategy: activeDeadlineSeconds: 21600 recreateParams: timeoutSeconds: 600 resources: {} type: Recreate template: metadata: creationTimestamp: null labels: name: redis redis-sentinel: 'true' spec: containers:

oc login -u developer -p xxx oc new-project redis-test --display-name="Redis Test" oc process openshift//redis-sentinel | oc create -f -

then you just wait for secs to complete initializations...

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

eliu commented 6 years ago

openshift is an existing project pre-created by OpenShift Origin and we just put the deployment template resource in this project so that it can be shared through all your projects. If you create your template in you own personal project, that will not make any difference about the usage, except it can only be accessed by your own project. that's it.

Natarajan77 commented 6 years ago

This is the message I m getting

From slave

 

Could not connect to Redis at 172.30.2.131:26379: Host is unreachable

2 Failed to find master.

 

From master

 

Could not connect to Redis at 172.30.2.131:26379: Host is unreachable

2 # Server

3 redis_version:3.2.0

4 redis_git_sha1:45b825e3

5 redis_git_dirty:0

6 redis_build_id:135c17da770e54e

7 redis_mode:standalone

8 os:Linux 3.10.0-693.17.1.el7.x86_64 x86_64

9 arch_bits:64

10 multiplexing_api:epoll

11 gcc_version:5.3.0

12 process_id:7

13 run_id:24ff4c89bb911ba2923c21dbc8756fbc9f07cd8d

14 tcp_port:6379

15 uptime_in_seconds:5004

16 uptime_in_days:0

17 hz:10

18 lru_clock:11709409

19 executable:/redis-server

20 config_file:/redis-master/redis.conf

21

 

22 # Clients

23 connected_clients:5

24 client_longest_output_list:0

25 client_biggest_input_buf:0

26 blocked_clients:0

27

 

28 # Memory

29 used_memory:950520

30 used_memory_human:928.24K

31 used_memory_rss:2117632

32 used_memory_rss_human:2.02M

33 used_memory_peak:950520

34 used_memory_peak_human:928.24K

35 total_system_memory:50470563840

36 total_system_memory_human:47.00G

37 used_memory_lua:37888

38 used_memory_lua_human:37.00K

39 maxmemory:0

40 maxmemory_human:0B

41 maxmemory_policy:noeviction

42 mem_fragmentation_ratio:2.23

43 mem_allocator:libc

44

 

45 # Persistence

46 loading:0

47 rdb_changes_since_last_save:0

48 rdb_bgsave_in_progress:0

49 rdb_last_save_time:1521657336

50 rdb_last_bgsave_status:ok

51 rdb_last_bgsave_time_sec:0

52 rdb_current_bgsave_time_sec:-1

53 aof_enabled:1

54 aof_rewrite_in_progress:0

55 aof_rewrite_scheduled:0

56 aof_last_rewrite_time_sec:-1

57 aof_current_rewrite_time_sec:-1

58 aof_last_bgrewrite_status:ok

59 aof_last_write_status:ok

60 aof_current_size:166

61 aof_base_size:0

62 aof_pending_rewrite:0

63 aof_buffer_length:0

64 aof_rewrite_buffer_length:0

65 aof_pending_bio_fsync:0

66 aof_delayed_fsync:0

67

 

68 # Stats

69 total_connections_received:26

70 total_commands_processed:30

71 instantaneous_ops_per_sec:0

72 total_net_input_bytes:598

73 total_net_output_bytes:48759

74 instantaneous_input_kbps:0.00

75 instantaneous_output_kbps:0.00

76 rejected_connections:0

77 sync_full:0

78 sync_partial_ok:0

79 sync_partial_err:0

80 expired_keys:0

81 evicted_keys:0

82 keyspace_hits:5

83 /run.sh: line 44: sentinel.conf: Permission denied

84 /run.sh: line 45: sentinel.conf: Permission denied

85 /run.sh: line 46: sentinel.conf: Permission denied

86 /run.sh: line 47: sentinel.conf: Permission denied

87 /run.sh: line 48: sentinel.conf: Permission denied

88 keyspace_misses:0

89 pubsub_channels:0

90 pubsub_patterns:0

91 latest_fork_usec:142

92 migrate_cached_sockets:0

93

 

94 # Replication

95 role:master

96 connected_slaves:0

97 master_repl_offset:0

98 repl_backlog_active:0

99 repl_backlog_size:1048576

100 repl_backlog_first_byte_offset:0

101 repl_backlog_histlen:0

102

 

103 # CPU

104 used_cpu_sys:2.38

105 used_cpu_user:1.58

106 used_cpu_sys_children:2.38

107 used_cpu_user_children:1.58

108

 

109 # Cluster

110 cluster_enabled:0

111

 

112 # Keyspace

113 db0:keys=3,expires=0,avg_ttl=0

114 13:X 22 Mar 03:00:49.721 # Fatal error, can't open config file 'sentinel.conf'

Sent from Yahoo Mail for iPhone

On Monday, March 19, 2018, 10:39 PM, Hong-Yu Liu notifications@github.com wrote:

openshift is an existing project pre-created by OpenShift Origin and we just put the deployment template resource in this project so that it can be shared through all your projects. If you create your template in you own personal project, that will not make any difference about the usage, except it can only be accessed by your own project. that's it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

Natarajan77 commented 6 years ago

Looks like not able to execute run.sh inside the container. Do u know how to fix this?

Sent from Yahoo Mail for iPhone

On Wednesday, March 21, 2018, 2:09 PM, Natarajan Sundaramoorthy natarajan77@yahoo.com wrote:

This is the message I m getting

From slave

 

Could not connect to Redis at 172.30.2.131:26379: Host is unreachable

2 Failed to find master.

 

From master

 

Could not connect to Redis at 172.30.2.131:26379: Host is unreachable

2 # Server

3 redis_version:3.2.0

4 redis_git_sha1:45b825e3

5 redis_git_dirty:0

6 redis_build_id:135c17da770e54e

7 redis_mode:standalone

8 os:Linux 3.10.0-693.17.1.el7.x86_64 x86_64

9 arch_bits:64

10 multiplexing_api:epoll

11 gcc_version:5.3.0

12 process_id:7

13 run_id:24ff4c89bb911ba2923c21dbc8756fbc9f07cd8d

14 tcp_port:6379

15 uptime_in_seconds:5004

16 uptime_in_days:0

17 hz:10

18 lru_clock:11709409

19 executable:/redis-server

20 config_file:/redis-master/redis.conf

21

 

22 # Clients

23 connected_clients:5

24 client_longest_output_list:0

25 client_biggest_input_buf:0

26 blocked_clients:0

27

 

28 # Memory

29 used_memory:950520

30 used_memory_human:928.24K

31 used_memory_rss:2117632

32 used_memory_rss_human:2.02M

33 used_memory_peak:950520

34 used_memory_peak_human:928.24K

35 total_system_memory:50470563840

36 total_system_memory_human:47.00G

37 used_memory_lua:37888

38 used_memory_lua_human:37.00K

39 maxmemory:0

40 maxmemory_human:0B

41 maxmemory_policy:noeviction

42 mem_fragmentation_ratio:2.23

43 mem_allocator:libc

44

 

45 # Persistence

46 loading:0

47 rdb_changes_since_last_save:0

48 rdb_bgsave_in_progress:0

49 rdb_last_save_time:1521657336

50 rdb_last_bgsave_status:ok

51 rdb_last_bgsave_time_sec:0

52 rdb_current_bgsave_time_sec:-1

53 aof_enabled:1

54 aof_rewrite_in_progress:0

55 aof_rewrite_scheduled:0

56 aof_last_rewrite_time_sec:-1

57 aof_current_rewrite_time_sec:-1

58 aof_last_bgrewrite_status:ok

59 aof_last_write_status:ok

60 aof_current_size:166

61 aof_base_size:0

62 aof_pending_rewrite:0

63 aof_buffer_length:0

64 aof_rewrite_buffer_length:0

65 aof_pending_bio_fsync:0

66 aof_delayed_fsync:0

67

 

68 # Stats

69 total_connections_received:26

70 total_commands_processed:30

71 instantaneous_ops_per_sec:0

72 total_net_input_bytes:598

73 total_net_output_bytes:48759

74 instantaneous_input_kbps:0.00

75 instantaneous_output_kbps:0.00

76 rejected_connections:0

77 sync_full:0

78 sync_partial_ok:0

79 sync_partial_err:0

80 expired_keys:0

81 evicted_keys:0

82 keyspace_hits:5

83 /run.sh: line 44: sentinel.conf: Permission denied

84 /run.sh: line 45: sentinel.conf: Permission denied

85 /run.sh: line 46: sentinel.conf: Permission denied

86 /run.sh: line 47: sentinel.conf: Permission denied

87 /run.sh: line 48: sentinel.conf: Permission denied

88 keyspace_misses:0

89 pubsub_channels:0

90 pubsub_patterns:0

91 latest_fork_usec:142

92 migrate_cached_sockets:0

93

 

94 # Replication

95 role:master

96 connected_slaves:0

97 master_repl_offset:0

98 repl_backlog_active:0

99 repl_backlog_size:1048576

100 repl_backlog_first_byte_offset:0

101 repl_backlog_histlen:0

102

 

103 # CPU

104 used_cpu_sys:2.38

105 used_cpu_user:1.58

106 used_cpu_sys_children:2.38

107 used_cpu_user_children:1.58

108

 

109 # Cluster

110 cluster_enabled:0

111

 

112 # Keyspace

113 db0:keys=3,expires=0,avg_ttl=0

114 13:X 22 Mar 03:00:49.721 # Fatal error, can't open config file 'sentinel.conf'

Sent from Yahoo Mail for iPhone

On Monday, March 19, 2018, 10:39 PM, Hong-Yu Liu notifications@github.com wrote:

openshift is an existing project pre-created by OpenShift Origin and we just put the deployment template resource in this project so that it can be shared through all your projects. If you create your template in you own personal project, that will not make any difference about the usage, except it can only be accessed by your own project. that's it.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

eliu commented 6 years ago

Hmmm, the redis containers needs to run as root user by default, which is not allowed by openshift when you firstly deploy it. Since your case rarely happens in my cluster, i guess your need to allow the default Security Context Constraint (SCC) restricted to run as any user.

Try run the following command to allow this:

oc patch scc restricted -p '{"runAsUser": {"type": "RunAsAny"},"fsGroup": {"type": "RunAsAny"}}'

And please try deploy your redis cluster again when you finishing running the command above in order to see if error still there. thanks.