log4j vulnerability CVE-2021-44228.

[daenney]

There's a high severity vulnerability out for log4j, CVE-2021-44228. The videobridge appears to use log4j.

I'd like to understand if:

• The bridge is vulnerable?
• If so, what's the timeline for a fix?

[jbg]

A fast mitigation in advance of updates being available is to add:

-Dlog4j2.formatMsgNoLookups=true

to the JVM command line. This is also relevant to other Jitsi components.

[dkasak]

According to https://news.ycombinator.com/item?id=29507263, that mitigation is only available in versions >= 2.10.0, but Jitsi appears to be using 2.3.

[jbg]

Sigh!

The comment also mentions this:

• Modify every logging pattern layout to say %m{nolookups} instead of %m in your logging config files, see details at https://issues.apache.org/jira/browse/LOG4J2-2109

... which is interesting because our logging patterns don't have %m in them anyway.

[jbg]

So far as I can see, log4j is not being used directly, only as a transitive dependency from callstats:

jbg@Jaspers-MacBook-Air:[~/dev/avstack/jitsi-videobridge]: mvn -pl org.jitsi:jitsi-videobridge dependency:tree -Dincludes=org.apache.logging.log4j:log4j-api
[INFO] org.jitsi:jitsi-videobridge:jar:2.1-SNAPSHOT
[INFO] \- org.jitsi:jitsi-stats:jar:1.0-7-g2a9b765:compile
[INFO]    \- io.callstats:callstats-java-sdk:jar:5.2.1:compile
[INFO]       \- org.apache.logging.log4j:log4j-api:jar:2.3:compile

[saghul]

Hey folks. The team has been looking into it since this morning. Thanks for being vigilant!

[daenney]

Sigh!

The comment also mentions this:

* Modify every logging pattern layout to say %m{nolookups} instead of %m in your logging config files, see details at https://issues.apache.org/jira/browse/LOG4J2-2109

... which is interesting because our logging patterns don't have %m in them anyway.

I'm not sure who "our" is referring too, but at least for the video bridge they are in use: https://github.com/jitsi/jitsi-videobridge/blob/56e0dae00c503470cf38d44650ddf20cdf228529/config/log4j2.xml#L9

[jbg]

Ah my mistake, I assumed $work was using the same pattern as upstream but it turns out we've customised it.

[daenney]

With #1776 merged all we now need is a release.

[hardfalcon]

Until there's a patched release, the following hack should do the trick:

zip -q -d /usr/share/jitsi-videobridge/lib/log4j-core-2.*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class
service jitsi-videobridge2 restart

[bgrozev]

We've determined that recent versions of jitsi-videobridge (since 2.1-504-g2f7fcb978 from May) are not affected by this CVE, thus it is not necessary to patch these systems. This is because the mistakenly mismatched versions of log4j-api and log4j-core lead to a failure to use the vulnerable log4j logger. See more details here: https://community.jitsi.org/t/cve-2021-44228-and-jitsi-components/108844

[damencho]

The new stable is out, with the updated version of jvb.

[krauthosting]

A fast mitigation in advance of updates being available is to add:

-Dlog4j2.formatMsgNoLookups=true

to the JVM command line. This is also relevant to other Jitsi components.

@saghul @damencho @bgrozev Can you confirm this works as hotfix to potential vulnerable versions as log4j is only used in JVB?

--- before: /etc/jitsi/videobridge/config (content)                                                                                  
+++ after: /etc/jitsi/videobridge/config (content)                                                                                   
@@ -17,4 +17,4 @@                                                                                                                    

 # adds java system props that are passed to jvb (default are for home and logging config file)
-JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java
.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.con
fig.file=/etc/jitsi/videobridge/logging.properties"
+JAVA_SYS_PROPS="-Dconfig.file=/etc/jitsi/videobridge/jvb.conf -Dnet.java.sip.communicator.SC_HOME_DIR_LOCATION=/etc/jitsi -Dnet.java
.sip.communicator.SC_HOME_DIR_NAME=videobridge -Dnet.java.sip.communicator.SC_LOG_DIR_LOCATION=/var/log/jitsi -Djava.util.logging.con
fig.file=/etc/jitsi/videobridge/logging.properties -Dlog4j2.formatMsgNoLookups=true"

[saghul]

Yes that should do it. Norge we already published new patched releases.

[twmobius]

It seems that a new CVE was issued yesterday (https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2021-45046) which states that the patch released in 2.15.0 was incomplete in certain non-default configurations. 2.16.0 was released yesterday to mitigate this instead.

I'm not sure how it affects jvb but I thought of mentioning

[bgrozev]

CVE-2021-45046 does not affect jitsi-videobridge, because it doesn't use any of the related features in PatternLayout. We're in the process of updating to 2.16.0 in any case.

[RogerWeihrauch]

Hello all hmm, shouldn't than this one not already be in state 'Closed'? Since latest log4j-version is goint to be integrated. Regards, Roger

[daenney]

It's in a closed state, because this issue was about the original CVE which is fixed and updates for Jitsi have been pushed. As noted, the second CVE doesn't affect JVB.

[RogerWeihrauch]

Ah, Ok, thanx daenney So, regardlessly, we can update JVB to latest version, itself only, or do you suggest to update the whole jitsi release? Regards, Roger

[basildane]

I have the latest versions installed (according to apt): jicofo/stable,now 1.0-832-1 all [installed,automatic]
jigasi/stable,now 1.1-216-ga2399b9-1 amd64 [installed] jitsi-meet-prosody/stable,now 1.0.5675-1 all [installed,automatic]
jitsi-meet-turnserver/stable,now 1.0.5675-1 all [installed,automatic] jitsi-meet-web-config/stable,now 1.0.5675-1 all [installed,automatic]
jitsi-meet-web/stable,now 1.0.5675-1 all [installed,automatic] jitsi-meet/stable,now 2.0.6726-1 all [installed]
jitsi-videobridge2/stable,now 2.1-595-g3637fda4-1 all [installed,automatic]

Yet I am getting this: WARNING: /usr/share/jitsi-videobridge/lib/log4j-core-2.15.0.jar contains "JndiLookup.class" WARNING: /usr/share/jigasi/lib/sentry-1.7.30.jar contains "JndiLookup.class" WARNING: /usr/share/jigasi/lib/log4j-core-2.15.0.jar contains "JndiLookup.class"

Just trying to determine if we are safe?

[bgrozev]

The presence of the jar files alone does not mean that the system is vulnerable. See the security advisories on how to determine whether you are affected or not. To reiterate -- the current stable release and master branches, without custom changes to log4j configuration, are not affected by either of the CVEs.

We are currently working on removing the log4j dependency.

[hardfalcon]

@basildane: It if makes you sleep better, the hack I mentioned above still works, and TBH I will keep applying that hack even to supposedly fixed log4j releases like 2.16.0 as long as the log4j devs fail to completely remove any JNDI support whatsoever instead of just disabling it by default, but leaving that fundamentally vulnerable and unfixable implementation of a fundamentally flawed idea in there. And I'd probably do the same to the sentry JAR as well if my Jitsi server had Jigasi installed.

The fact that the supposedly "only DoS" CVE in log4j 2.15.0 had to be upgraded to an RCE doesn't exactly inspire confidence in the reliability of any other, less radical measures.

Of course this doesn't mean that Jitsi is actually attackable through this, but if it doesn't use the vulnerable part of the code at all, then it shouldn't be a problem to rip that vulnerable part of the code out before somebody figures out a way to nudge Jitsi into using/executing that vulnerable code.

[bgrozev]

it shouldn't be a problem to rip that vulnerable part of the code out before somebody figures out a way to nudge Jitsi into using/executing that vulnerable code.

And we are working on it.