Implement sentry with elasticsearch 6.2

[Marc3001]

Hi,

Context

We are using elasticsearch 6.2 since a while now and as we got some incident recently, we would like to get elasticsearch errors in sentry. As you surely know Elasticsearch is developped using java language and is using log4j2 as logging manager. So we were happy to see sentry-log4j2 exists and wanted to use it.

Issue

We can see only booting early stage logs in sentry with no possibility to filter via log level using Sentry appender.

Question

As sentry-java is a dependency of sentry-log4j2 and sentry-java has slf4j as a dependency, does that mean the app we want to integrate sentry in has to use slf4j with log4j2 ? As Elasticsearch is not, I was wondering if our issue was not there.

I added slf4j-api, sentry-java and sentry-log4j2 jars in lib folder and added sentry appender definition like so

appender.sentry.type = Sentry
appender.sentry.name = sentry

rootLogger.level = info
rootLogger.appenderRef.sentry.ref = sentry

With info level on rootLogger, nothing can be seen on sentry. With all level on rootLogger, I can see some debug and info logs of elasticsearch early stage (but nothing after some seconds of start). No idea if issue is on my configuration or the way sentry or Elasticsearch are using log4j2.

[bruno-garcia]

I haven't tried to add Sentry to elasticsearch but as you said, that being written in Java, it should be possible. sentry-java using slf4j shouldn't have any effect. It just means you wouldn't get the logs written by Sentry itself.

It should be possible to simply use sentry-log4j2 to capture the log messages as events.

Sounds like you're having difficulty with the log level as a filter though. Could you please share a repro? A small repository we can run the code to see the issue?

[Marc3001]

Thx @bruno-garcia for replying ;)

See below my repro steps

Install elasticsearch 6.2.4 (the one we are actually using) on a fresh debian box

cat << EOF > /etc/apt/sources.list.d/elasticsearch.list 
deb https://artifacts.elastic.co/packages/6.x/apt stable main
EOF
apt-get update
apt-get install elasticsearch=6.2.4

Get sentry/log4j libs from maven central

wget -O /usr/share/elasticsearch/lib/slf4j-api-1.7.30.jar 'https://repo1.maven.org/maven2/org/slf4j/slf4j-api/1.7.30/slf4j-api-1.7.30.jar'
wget -O /usr/share/elasticsearch/lib/sentry-log4j2-1.7.30.jar 'https://repo1.maven.org/maven2/io/sentry/sentry-log4j2/1.7.30/sentry-log4j2-1.7.30.jar'
wget -O /usr/share/elasticsearch/lib/sentry-1.7.30.jar 'https://repo1.maven.org/maven2/io/sentry/sentry/1.7.30/sentry-1.7.30.jar'

Set .java.policy to allow jar to send event to sentry.io This is quite dirty to set grant globally but I want to be sure I´m able to make it work before setting security in a cleaner way

cat << EOF > /home/elasticsearch/.java.policy
grant {
  permission java.net.SocketPermission "sentry.io:443", "connect";
  permission java.net.SocketPermission "sentry.io:443", "resolve";
};
EOF

Set sentry conf in jvm.options

cat << EOF >> /etc/elasticsearch/jvm.options
-Dsentry.dsn=https://<credentials>@sentry.io/<project>
-Dsentry.environment=production
EOF

Set log4j.properties as below

appender.sentry.type = Sentry
appender.sentry.name = sentry
[...]
rootLogger.level = info
rootLogger.appenderRef.sentry.ref = sentry

Restart elasticsearch using systemctl restart elasticsearch Then you should see nothing in sentry notifications

In an other way, if you set rootLogger.level = all in the log4j.properties sentry will get flooded by logs with info,debug, trace and warn levels

[bruno-garcia]

Thanks for the repro steps. It's rather hard to 'debug' this since Sentry is being added via configuration online to an already built and packaged app and this is not really something we support.

[Marc3001]

OK. So I'm eventually waiting for somebody who maybe did it successfully and will share his knowlegde :pray:

[bruno-garcia]

I'll leave this issue open then for a while if someone wants to collaborate.

I'd even suggest raising a post on Sentry's forum: forum.sentry.io, feel free to point to this issue too.