Adding ingestor inputs routing

[hjaffan]

Currently release supports adding plugins to the syslog_ingestor, but the problem there is that we are not able to route them from the HAProxy.

This PR allows for an opinionated but also dynamic addition of inputs and HA Proxy configuration.

[CLAassistant]

Thank you for your submission, we really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.

---

Hilal Jaffan seems not to be a GitHub user. You need a GitHub account to be able to sign the CLA. If you have already a GitHub account, please add the email address used for this commit to your account.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

[Infra-Red]

Hi there! Thank you for submitting the PR. I will take a look at the changes and provide a feedback in coming days.

[Infra-Red]

@hjaffan One note, for any agents that are deployed as a bosh-release you can do a simple trick with bosh-dns an avoid using ls-router job. You can configure your job (for example syslog-release) to use q-s0.ingestor.<NETWORK-NAME>.<DEPLOYMENT-NAME>.bosh bosh-dns name as a remote address and it's should be enough to spread the load between ingestor instances.

[hjaffan]

HI @Infra-Red, So regarding the DNS name, I feel yes that would work, but how would I handled the port routing?

I feel like I had to modify the syslog section here, because the listening ports on the agents / inputs changes

[Infra-Red]

Hi @hjaffan. The input port is static across all Logstash instances, for example default port for tcp-input-plugin is 5514. Can you describe your usecase and why port routing required in your setup.

[hjaffan]

Hi @Infra-Red , so if you notice, I was exporting the inputs section, now those inputs listen on different ports. For example, the main reason for starting this PR was to actually ingest messages coming from beat agents. and those you have to set them up as a different input in the logstash.conf where you put the inputs {} filters {} outputs {}

[hjaffan]

@Infra-Red did you have more questions regarding this?

[hjaffan]

Thanks @Infra-Red I have updated the name of the file.

[Infra-Red]

Thanks for the contribution @hjaffan !