search

rabbitmq / rabbitmq-peer-discovery-k8s

Kubernetes-based peer discovery mechanism for RabbitMQ
Other
295 stars 94 forks source link

pods crashed with 404 error #26

Closed autumnw closed 6 years ago

autumnw commented 6 years ago

Hi,

I run the rabbitmq V3.7.3 over k8s 1.10, and pods keep crashing with the following error message in log. I am sure I applied the rbac rule, there is no 403 error.

2018-04-22 17:37:16.745 [info] <0.195.0> Memory high watermark set to 77308 MiB (81064201420 bytes) of 193272 MiB (202660503552 bytes) total 2018-04-22 17:37:16.753 [info] <0.197.0> Enabling free disk space monitoring 2018-04-22 17:37:16.753 [info] <0.197.0> Disk free limit set to 50MB 2018-04-22 17:37:16.761 [info] <0.199.0> Limiting to approx 65436 file handles (58890 sockets) 2018-04-22 17:37:16.762 [info] <0.200.0> FHC read buffering: OFF 2018-04-22 17:37:16.762 [info] <0.200.0> FHC write buffering: ON 2018-04-22 17:37:16.765 [info] <0.187.0> Node database directory at /mnt/data/rabbitmq/wf2-rabbitmq-2/rabbit@10.32.0.29 is empty. Assuming we need to join an existing cluster or initialise from scratch... 2018-04-22 17:37:16.765 [info] <0.187.0> Configured peer discovery backend: rabbit_peer_discovery_k8s 2018-04-22 17:37:16.765 [info] <0.187.0> Will try to lock with peer discovery backend rabbit_peer_discovery_k8s 2018-04-22 17:37:16.765 [info] <0.187.0> Peer discovery backend rabbit_peer_discovery_k8s does not support registration, skipping randomized startup delay. 2018-04-22 17:37:16.803 [info] <0.187.0> Failed to get nodes from k8s - 404 2018-04-22 17:37:16.803 [error] <0.186.0> CRASH REPORT Process <0.186.0> with 0 neighbours exited with reason: no case clause matching {error,"404"} in rabbit_mnesia:init_from_config/0 line 163 in application_master:init/4 line 134 2018-04-22 17:37:16.804 [info] <0.31.0> Application rabbit exited with reason: no case clause matching {error,"404"} in rabbit_mnesia:init_from_config/0 line 163 {"Kernel pid terminated",application_controller,"{application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,\"404\"}},[{rabbit_mnesia,init_from_config,0,[{file,\"src/rabbit_mnesia.erl\"},{line,163}]},{rabbit_mnesia,init_with_lock,3,[{file,\"src/rabbit_mnesia.erl\"},{line,143}]},{rabbit_mnesia,init,0,[{file,\"src/rabbit_mnesia.erl\"},{line,111}]},{rabbit_boot_steps,'-run_step/2-lc$^1/1-1-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,run_step,2,[{file,\"src/rabbit_boot_steps.erl\"},{line,49}]},{rabbit_boot_steps,'-run_boot_steps/1-lc$^0/1-0-',1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit_boot_steps,run_boot_steps,1,[{file,\"src/rabbit_boot_steps.erl\"},{line,26}]},{rabbit,start,2,[{file,\"src/rabbit.erl\"},{line,792}]}]}}}}}"} Kernel pid terminated (application_controller) ({application_start_failure,rabbit,{bad_return,{{rabbit,start,[normal,[]]},{'EXIT',{{case_clause,{error,"404"}},[{rabbit_mnesia,init_from_config,0,[{file

michaelklishin commented 6 years ago

Thank you for your time.

Team RabbitMQ uses GitHub issues for specific actionable items engineers can work on. GitHub issues are not used for questions, investigations, root cause analysis, discussions of potential issues, etc (as defined by this team).

We get at least a dozen of questions through various venues every single day, often light on details. At that rate GitHub issues can very quickly turn into a something impossible to navigate and make sense of even for our team. Because GitHub is a tool our team uses heavily nearly every day, the signal/noise ratio of issues is something we care about a lot.

Please post this to rabbitmq-users.

Thank you.

michaelklishin commented 6 years ago

A peer discovery attempt returned a 404. API endpoint hostname, schema and port are 3 values that are used to construct the effective URL, which will be seen at debug log level.