Metric gathering make Logstash 5 crash at startup when procfs is not mounted

[L4rS6]

I just installed logstash5[1] on a FreeBSD 10[2] system. Logstash startup fails due to unmounted procfs:

OperatingSystemImpl.java:-2:in `getCommittedVirtualMemorySize': java.lang.InternalError: errno: 2 error: Unable to open /proc/curproc/map

For security reasons we can't mount the procfs. Is there any solution for this? procfs wasn't required in logstash version 2.3.2.

[1] Logstash Version: logstash5-5.0.0

[2] FreeBSD Version: FreeBSD 10.3-RELEASE-p12

[untergeek]

This is because the new monitoring feature uses procfs to read specific performance metrics. I don't recall of the top of my head (I'm on my iPhone) if it can be disabled. Look inside $LS_HOME/config/logstash.yml to see if metric gathering can be disabled.

If not, this may be a feature request.

[ph]

@untergeek @L4rS6 Metric gathering cannot at the moment be disabled, I am open to discuss it if you open a feature request.

But metrics shouldn't negatively impact a running system, so I think we should revise our stats calls to wrap them with rescue block and gracefully fails instead of making the whole system crash.

So for me its a bug, since we don't recover from that case.

[L4rS6]

@ph Thanks for your explanation. Would be great if we could disable metrics gathering. Do I have to open a new issue as a feature request or can I change the topic of that one?

[ph]

@L4rS6 let keep this bug open, I will create a new one for discussing the disabling of metric

[jordansissel]

Instead of disabling metrics, let's figure out how to fix the specific problem causing this failure (something in Logstash wanting procfs and crashing if it's not available? Iet's not crash)

[ph]

I've made some refactoring on a jruby helper gem that we use to get metrics to catch theses exception. I've tested the changes on freebsd 11 with openjdk8 and it sees to be fine.

by default the /procfs is not mount on freebsd, but when you install the openjdk8 they tell you that openjdk8 requires theses mount. I am not fully aware of the implication of not doing it (or doing it)

On a side note, @L4rS6 are you running ES on the same FreeBSD configuration?

[jordansissel]

@ph for freebsd, I don't even know if /proc is the correct thing. FreeBSD has its own procfs (not mounted by default because most stuff is handled by sysctl); it also has linprocfs which emulates Linux's procfs.

Neither are probably reliable ways to gather data on FreeBSD, since there's probably APIs available by default without /proc.

Anyway, if this error is coming from the JVM when we ask the JVM for memory detaisl, then I suppose users will need to mount procfs to /proc on FreeBSD for Logstash?

[ph]

@jordansissel The error is coming from JVM, on the OperatingSystemBean they don't have any checked exception, it just crash with an internal error on that case. I can reproduce it easy with a plain freebsd install, Also note that in the screenshot after installing openjdk8 they explicitly ask to mount theses, because it requires it.

So if we want to fix that problem we either have to wrap our calls with either Throwable or InternalError, Not sure we should go that route.

[L4rS6]

@ph Sorry, I overlooked your question. Yes, ElasticSearch is running with the same configuration. But I didn't test it yet for ES version 5 (currently running elasticsearch2-2.4.1_1).