Hi,
I was currently testing this project on our cluster but we are facing an issue with the current version.
Our cluster currently has more than 2k topics and even more consumers groups. Apparently this exporter launch a go-routine by cluster, then by topic & then by consumer group to scrape the REST api of Burrow in parallel. The problem that was that too many simultaneous call to the rest api were done causing this exporter to crash with a unable to open socket : too many open files.
For now we just disabled all the go routine call (exporter.go => line 139, 148, 191) and everything works fine, the exporter is able to scrape the whole api in less than 10 seconds
Hi, I was currently testing this project on our cluster but we are facing an issue with the current version. Our cluster currently has more than 2k topics and even more consumers groups. Apparently this exporter launch a go-routine by cluster, then by topic & then by consumer group to scrape the REST api of Burrow in parallel. The problem that was that too many simultaneous call to the rest api were done causing this exporter to crash with a
unable to open socket : too many open files.For now we just disabled all the go routine call (exporter.go => line 139, 148, 191) and everything works fine, the exporter is able to scrape the whole api in less than 10 seconds