search

RocketChat / helm-charts

Repository for RocketChat helm charts
36 stars 67 forks source link

EADDRINUSE: address already in use 0.0.0.0:9458 #56

Closed matthias4217 closed 2 years ago

matthias4217 commented 2 years ago

I have tried to migrate from a monolithic deployment of Rocketchat to a multiservice one, but I've encountered some issues.

What I've tried

:heavy_check_mark: Old installation :

This is the stable setup that we had previously. We've been using the prometheus scraping for a while and things are fine.

:x: New installation A :

This is where I have a EADDRINUSE: address already in use 0.0.0.0:9458 error. It seems that the Prometheus port in the Rocketchat deployment has two application trying to bind to it.

:heavy_check_mark: New installation B :

This works fine. There is no binding issue.

:heavy_check_mark: New installation C :

This works fine. There is no binding issue either.

Conclusion

I am not sure if this issue would appear on previous RocketChat version, and if it needs to be fixed on the Helm Chart or Rocketchat itself. However, it's clear that there is an issue when using the metrics scraping with microservices on the default port.

prppedro commented 2 years ago

Thanks for the report, @matthias4217. I've encountered the very same issue while using Helm alongside one of our enterprise deployment methods, last month. My workaround was the same as you B installation. Not a very good solution, but I'm yet to understand why it happens. For now, it is still under investigation.

prppedro commented 2 years ago

There seems to be a conflict — inside the main Rocket.Chat instance — between RC's exporter and the Moleculer exporter. Using that workaround prevents such conflict, but, from what I've been observing, keeps RC's from exporting any metrics (RC's metrics exporter don't even listens on the alternate port). I've contacted the dev team and we're now debugging this very issue. Hopefully we are going to fix it, very soon.

prppedro commented 2 years ago

@matthias4217, I think I found a workaround for that. I've set the default metrics port as 9100 and defined 9458 for Moleculer exporter on all services. Also, I made sure RC's main chat service creates an entry for 9100. Now both exporters work and Prometheus are able to find them. The conflict itself isn't easy to solve, I guess, so it's better just to leave all Moleculer exporters alone and just using a different port.

I am, therefore, closing this issue. If you find other issues, please feel free to reopen it.