search

christiangalsterer / httpbeat

Elastic Beat to call HTTP endpoints
Apache License 2.0
107 stars 40 forks source link

Connection error when sampling many URLs #20

Open eyala01 opened 7 years ago

eyala01 commented 7 years ago

Hi,

We're using httpbeat to sample every minute (cron: "@every 1m") multiple URLs from http://127.0.0.1:8443 on a Windows 2012 R2 machine. It works great for a list of a few URLs, but when we increase the list to around 30 URLs or more, we sometimes get this error in the LOG file:

ERR An error occurred while executing HTTP request: [dial tcp 127.0.0.1:8443: connectex: No connection could be made because the target machine actively refused it. Get http://127.0.0.1:8443/: dial tcp 127.0.0.1:8443: connectex: No connection could be made because the target machine actively refused it.]

It isn't a problem on the web server side, because the same sample of URLs with the same frequency works just fine using a different sampling tool.

Is there a limit to the number of URLs we can sample with one instance of httpbeat, or did we find a performance problem?

Please advise, Eyal

christiangalsterer commented 7 years ago

In general there aren't any limitations on the number of connections. The error message indicates that the server refuses additional connections, although you mention that a different tool doesn't show this issues. Maybe there is a difference in connection reuse.

I don't have a Window machine available for testing but will try to reproduce the issue with Linux. Q: Which version of httpbeat do you use?

eyala01 commented 7 years ago

Thanks for the quick reply. We're using httpbeat 3.2.0.

eyala01 commented 7 years ago

Quick update: We upgraded our httpbeat to the latest version (4.0.0), and the problem persists.

christiangalsterer commented 7 years ago

I just tried it with 100 connections and it runs without problems. As the error message "No connection could be made because the target machine actively refused it.]" indicated there seems to be a setting on the server side which prevents further connections after a threshold. Can you check your server again?

christiangalsterer commented 6 years ago

Did you had a change to check it again?