[CVE-2021-44228] Log4j vulnerability

[mbrunger]

Hi, what is the impact of the log4j vulnerability on GeoNetwork. Are you planning to release a patch?

[jahow]

Hi, this is a good question. Our first analysis showed that log4j2 is present in the GeoNetwork classpath, both in versions 3 and 4, but it is apparently not used anywhere.

At Camptocamp we are still hard at work investigating if parts of the application are exposed to this vulnerability. Any effort in that direction is welcome.

In case parts of the application are indeed exposed, hotfixes will have to be produced on all recent branches (4.0, 3.12, 3.10 and maybe earlier versions as well).

[jahow]

Actually, thanks to @josegar74 the version of log4j2 was upgraded and the backports done on friday, see: https://github.com/geonetwork/core-geonetwork/pull/6075

@josegar74 could you please confirm that everything was done to be safe from this vunlerability? Thanks!

[jahow]

As for users of GeoNetwork 4, some versions of ElasticSearch use a version of log4j2 that is exposed to the vulnerability; please refer to this announcement for more information. Note that ElasticSearch is on the safe side anyway, so what's described there is mainly for additional safety.

Also this states that Kibana is not affected by the vulnerability.

Last thing is to make sure that the recommandations cited in the announcement are applied to the embedded ElasticSearch instance in the es module of GeoNetwork (which is version 7.11.1).

[mbrunger]

I saw this page from the GN installation guide where it is written that Log4j is used for logging in GeoNetwork. The version is not mentioned, but Log4j v1.x might be just as vulnerable, it is only no longer supported.

[josegar74]

Hi

A patch for 3.10.x has been released (3.10.10 https://sourceforge.net/projects/geonetwork/files/GeoNetwork_opensource/v3.10.10/), tomorrow I'll check to make a release for 3.12.

[KoalaGeo]

Great work @josegar74!

I've made a PR to update the docker repo for 3.10.10 - https://github.com/geonetwork/docker-geonetwork/pull/75

What's the lead time between Sourceforge & dockerhub? I can't see a GitHub action setup - is it manual?

[jahow]

I saw this page from the GN installation guide where it is written that Log4j is used for logging in GeoNetwork. The version is not mentioned, but Log4j v1.x might be just as vulnerable, it is only no longer supported.

This is true but at least log4j v1 is not exposed to this specific vulnerability (which relies on a feature not present in versions before 2). Migrating to a more recent logger is indeed a task long overdue for GeoNetwork.

[juanluisrp]

I've created https://github.com/geonetwork/docker-geonetwork/pull/76 (I didn't see your PR @KoalaGeo).

What's the lead time between Sourceforge & dockerhub? I can't see a GitHub action setup - is it manual?

I've created https://github.com/docker-library/official-images/pull/11490. Now Docker official images team has to review it and merge. I think they work with US office time. When merged the image is automatically built and published in https://hub.docker.com/_/geonetwork

[fnicollet]

Hello,

We are using Geonetwork 3.10.4. It seems to use log4j 2.7:

Mitigating the issue using a java parameter (-Dlog4j.formatMsgNoLookups=true) is apparently only available in log4j 2.10+, so i guess it has no effect on this version Geonetwork.

I read online that:

For Log4j 2.7 or higher: specify %m{nolookups} in the PatternLayout configuration to prevent lookups in log event messages.

I tried changing WEB-INF/classes/log4j.xml and changing:

<param name="ConversionPattern" value="%d{ISO8601} %-5p [%c] - %m%n"/>

to

<param name="ConversionPattern" value="%d{ISO8601} %-5p [%c] - %m{nolookups}%n"/>

Which seem to change the output log (so log4j is used @jahow ?) but seems to be appended:

2021-12-13 13:47:40,939 WARN  [geonetwork.schemamanager] - WARNING: schema-substitutions.xml has wiped out XSD substitution list for gml:VerticalCS{nolookups}
2021-12-13 13:47:40,941 WARN  [geonetwork.schemamanager] - WARNING: schema-substitutions.xml has wiped out XSD substitution list for gml:CoordinateSystemAxis{nolookups}

So I am not sure if it really mitigates the issue. Anyone else is try to hotfix this ?

Thanks! Fabien

[juanluisrp]

Which seem to change the output log (so log4j is used @jahow ?) but seems to be appended

That looks like an output produced by log4j-1.2.17

[fnicollet]

@juanluisrp that makes sense! So both log4j 1.x and log4j 2.x are loaded in the classpath but only v1.x is used. Then this version is not subject to vulnerability, that's good news :)

[juanluisrp]

We aren't sure if any dependency is using it internally, so I'd recommend to be cautious with such statement.

In the meantime you can remove the JndiLookup.class from the 2.7 jar running

zip -d ${TOMCAT_HOME}/webapps/geonetwork/WEB-INF/lib/log4j-core-2.* org/apache/logging/log4j/core/lookup/JndiLookup.class

and restarting your server. If you have deployed the app using the WAR you will also need to replace the WEB-INF/lib/log4j-core-2.* file with the patched one and repackage the WAR.

[pmauduit]

The Log4j configurations provided in the webapp are targetting log4j1, which is also still present in the classpath. I tried removing the log4j2 dependency and GN still compiles fine, my assumption is that even if log4j2 is present, the code is never called nor used.

Checking dependencies using mvn dependency:tree, it looks like it is needed by the elasticsearch client though:

[INFO] +- org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:7.11.1:compile
[INFO] |  +- org.elasticsearch:elasticsearch:jar:7.11.1:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-core:jar:7.11.1:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-secure-sm:jar:7.11.1:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-x-content:jar:7.11.1:compile
[INFO] |  |  |  +- org.yaml:snakeyaml:jar:1.26:compile
[INFO] |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.10.4:compile
[INFO] |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.10.4:compile
[INFO] |  |  |  \- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.10.4:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-geo:jar:7.11.1:compile
[INFO] |  |  +- org.apache.lucene:lucene-core:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-analyzers-common:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-backward-codecs:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-grouping:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-highlighter:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-join:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-memory:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-misc:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-queries:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-queryparser:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-sandbox:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-spatial-extras:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-spatial3d:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-suggest:jar:8.7.0:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-cli:jar:7.11.1:compile
[INFO] |  |  |  \- net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile
[INFO] |  |  +- com.carrotsearch:hppc:jar:0.8.1:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.10.4:compile
[INFO] |  |  +- com.tdunning:t-digest:jar:3.2:compile
[INFO] |  |  +- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile
[INFO] |  |  +- org.apache.logging.log4j:log4j-api:jar:2.11.1:compile
[...]

Main question is: is it possible to trick the ES rest client so that something is logged ?

[archaeogeek]

Can anyone comment on whether the patch for 3.10.x and 3.12.x (eg https://github.com/geonetwork/core-geonetwork/pull/6075) will also be applicable for earlier versions? I have a site running 3.4.x that I am currently trying to upgrade but I may need to patch it until the upgrade is complete.

[jahow]

Can anyone comment on whether the patch for 3.10.x and 3.12.x (eg #6075) will also be applicable for earlier versions? I have a site running 3.4.x that I am currently trying to upgrade but I may need to patch it until the upgrade is complete.

If there is an ongoing upgrade then instead of relying on an official release you can maybe try to remove the log4j2 jars from the war file.

For those wondering, log4j2 was introduced in v3.4, see commit: https://github.com/geonetwork/core-geonetwork/commit/69be7afc55e1cc1e43cf73f7f45a429d55dba0ce

Any version before that is supposedly safe

[jahow]

@josegar74 I took the liberty of backporting your commit on all the other affected branches: 3.4.x, 3.6.x and 3.8.x

Still no new releases done though, not sure if we should go that far, maybe the PSC can give a clear orientation on this?

[geonetworkbuild]

Hi Olivier

Thanks for backporting it. At least for now, we don't have the capacity to make new releases for old versions.

Regards, Jose García

On Mon, Dec 13, 2021 at 3:42 PM Olivier Guyot @.***> wrote:

@josegar74 https://github.com/josegar74 I took the liberty of backporting your commit on all the other affected branches: 3.4.x, 3.6.x and 3.8.x

Still no new releases done though, not sure if we should go that far, maybe the PSC can give a clear orientation on this?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/geonetwork/core-geonetwork/issues/6076#issuecomment-992548420, or unsubscribe https://github.com/notifications/unsubscribe-auth/AOCJAMP2JYYV4VFKKFSMFWTUQYA6NANCNFSM5J52ZXVA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

--

Vriendelijke groeten / Kind regards,Jose García http://www.geocat.net/Veenderweg 136721 WD BennekomThe NetherlandsT: +31 (0)318 416664 <+31318416664>Please consider the environment before printing this email.

[etj]

About the older log4j 1.2.17, GeoTools is using a brand new log4j build (see https://osgeo-org.atlassian.net/browse/GEOT-7038) that removes the classes that can be exploited.

The patched fork is https://github.com/aaime/log4j/tree/1.2.17.x

The related jar has been deployed at https://repo.osgeo.org/repository/geotools-releases/log4j/log4j/1.2.17.norce/log4j-1.2.17.norce.jar

Installations that are not using the JMSAppender can safely use the patched version.

Since the osgeo repository is already included in GN pom file, we may think about upgrading the 1.2.17 as well.

[juanluisrp]

GN 3.12.2 including updated log4j-core-2.15.0.jar has been released and it's available at Sourceforge and a PR (https://github.com/docker-library/official-images/pull/11503) to the Docker official images repo has been created.

[juanluisrp]

About the new CVE-2021-45046 I've created https://github.com/geonetwork/core-geonetwork/pull/6079 to update log4j to 2.16.0 although I think the default config of GN loggers isn't affected by this.

[culkeri]

Is there a timeline on when the sourceforge will be updated (GN 3.12.2) with og4j-core-2.17.0.jar?

[KoalaGeo]

@juanluisrp is #6079 going to be a 3.12.2.1 or 3.12.3 to look out for? Or thinking of giving it a couple of days in case something else pops up?

[terrycurran]

Is there a timeline on when the sourceforge will be updated (GN 3.12.2) with log4j-core-2.16.0.jar?

Now that the Elasticsearch potential vulnerability seems resolved, is there a tentative target date to resolve the Log4j issue in GN4.0.5?

[Naveenapr]

The Log4j configurations provided in the webapp are targetting log4j1, which is also still present in the classpath. I tried removing the log4j2 dependency and GN still compiles fine, my assumption is that even if log4j2 is present, the code is never called nor used.

Checking dependencies using mvn dependency:tree, it looks like it is needed by the elasticsearch client though:

[INFO] +- org.elasticsearch.client:elasticsearch-rest-high-level-client:jar:7.11.1:compile
[INFO] |  +- org.elasticsearch:elasticsearch:jar:7.11.1:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-core:jar:7.11.1:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-secure-sm:jar:7.11.1:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-x-content:jar:7.11.1:compile
[INFO] |  |  |  +- org.yaml:snakeyaml:jar:1.26:compile
[INFO] |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-smile:jar:2.10.4:compile
[INFO] |  |  |  +- com.fasterxml.jackson.dataformat:jackson-dataformat-yaml:jar:2.10.4:compile
[INFO] |  |  |  \- com.fasterxml.jackson.dataformat:jackson-dataformat-cbor:jar:2.10.4:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-geo:jar:7.11.1:compile
[INFO] |  |  +- org.apache.lucene:lucene-core:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-analyzers-common:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-backward-codecs:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-grouping:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-highlighter:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-join:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-memory:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-misc:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-queries:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-queryparser:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-sandbox:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-spatial-extras:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-spatial3d:jar:8.7.0:compile
[INFO] |  |  +- org.apache.lucene:lucene-suggest:jar:8.7.0:compile
[INFO] |  |  +- org.elasticsearch:elasticsearch-cli:jar:7.11.1:compile
[INFO] |  |  |  \- net.sf.jopt-simple:jopt-simple:jar:5.0.2:compile
[INFO] |  |  +- com.carrotsearch:hppc:jar:0.8.1:compile
[INFO] |  |  +- joda-time:joda-time:jar:2.10.4:compile
[INFO] |  |  +- com.tdunning:t-digest:jar:3.2:compile
[INFO] |  |  +- org.hdrhistogram:HdrHistogram:jar:2.1.9:compile
[INFO] |  |  +- org.apache.logging.log4j:log4j-api:jar:2.11.1:compile
[...]

Main question is: is it possible to trick the ES rest client so that something is logged ?

Is this SnakeYAML package impacted to log4j??

[KoalaGeo]

Is it possible to remove the log4jv2 from the class path in the docker image? it's still flagging in our security scans. Or could I remove and then 'COPY' in v2.17?Our security are being super cautious and have made us turn off all Geonetwork instances (6) we had running

[terrycurran]

We haven't been disabled yet with our GN4.0.5, but we are on "security watch".

On 2022-01-02 3:43 a.m., KoalaGeo wrote:

Is it possible to remove the log4jv2 from the class path in the docker image? Or remove and then 'COPY' in v2.17?Our security are being super cautious and have made is turn off all Geonetwork instances (6) we had running

— Reply to this email directly, view it on GitHub https://github.com/geonetwork/core-geonetwork/issues/6076#issuecomment-1003702573, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB2Q4XP56AK7HGRE22QUWXLUUA25HANCNFSM5J52ZXVA. Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you commented.Message ID: @.***>

[jahow]

Unfortunately there is currently no ongoing initiative (that I know of) for completely resolving this issue, which would mean getting rid of log4j2 in the classpath altogether, one way or another. Different workarounds have been suggested here for making IT security teams happy, so I'm afraid this will have to do for now.

I invite anyone with the intent to solve this issue (either by themselves or by sponsoring someone) to make themselves known here. Thanks in advance!

[KoalaGeo]

We (British Geological Survey) will be taking a proper look at workarounds as we can't leave them off as the CSW is a dependency for other services. Will certainly share if we come up with anything. Interim might be to implement pyCSW then we've got the CSW (and OGCAPI-Records now!) available even if no front end.

[juanluisrp]

Is it possible to remove the log4jv2 from the class path in the docker image? it's still flagging in our security scans. Or could I remove and then 'COPY' in v2.17?Our security are being super cautious and have made us turn off all Geonetwork instances (6) we had running

@KoalaGeo yes, you could do that, you can build your own patched Docker image using this for example:

FROM geonetwork:3.12.2

RUN rm "${CATALINA_HOME}"/webapps/geonetwork/WEB-INF/lib/log4j-core-* && \
    rm "${CATALINA_HOME}"/webapps/geonetwork/WEB-INF/lib/log4j-api-* && \
    curl https://dlcdn.apache.org/logging/log4j/2.17.1/apache-log4j-2.17.1-bin.tar.gz -o /tmp/log4j2.tgz && \
    mkdir /tmp/log4j2 && \
    tar -xvzf /tmp/log4j2.tgz -C /tmp/log4j2 && \
    cp /tmp/log4j2/apache-log4j-2.17.1-bin/log4j-core-2.17.1.jar "${CATALINA_HOME}"/webapps/geonetwork/WEB-INF/lib/ && \
    cp /tmp/log4j2/apache-log4j-2.17.1-bin/log4j-api-2.17.1.jar "${CATALINA_HOME}"/webapps/geonetwork/WEB-INF/lib/ && \
    rm -rf /tmp/log4j2*

docker build --tag geonetwork:3.12.2-log4jpatched .
docker run -p 8080:8080 geonetwork:3.12.2-log4jpatched

Or instead of downloading the JARs from Apache on build time download the JARs in advance and use

FROM geonetwork:3.12.2

RUN rm "${CATALINA_HOME}"/webapps/geonetwork/WEB-INF/lib/log4j-core-* && \
    rm "${CATALINA_HOME}"/webapps/geonetwork/WEB-INF/lib/log4j-api-*

COPY log4j-api-2.17.1.jar geonetwork/WEB-INF/lib/
COPY log4j-core-2.17.1.jar geonetwork/WEB-INF/lib/

[KoalaGeo]

@juanluisrp that's great thank you for that, security are now all happy so we can get our instances redeployed 🥳

[juanluisrp]

We have patched the GeoNetwork Docker official images to include Log4j 2.17.1 even when no new GN releases including it have been done yet. These Docker images have been updated:

• geonetwork:3.10.10, geonetwork:3.10.10-postgres
• geonetwork:3.12.2, geonetwork:3.12.2-postgres
• geonetwork:4.0.5

[nmtoken]

curl https://dlcdn.apache.org/logging/log4j/2.17.1/apache-log4j-2.17.1-bin.tar.gz

Note that this 2.17.1 version no longer exists at dlcdn.apache.org there is instead 2.19.0