search

IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
204 stars 80 forks source link

Elasticsearch and log4j vulnerability #357

Open Eneuman opened 2 years ago

Eneuman commented 2 years ago

Hi

Elasticsearch has released a updated version that fixes there internal log4j vulnerabilities. (https://www.elastic.co/blog/new-elasticsearch-and-logstash-releases-upgrade-apache-log4j2)

I can see that the docker files in this repo are still pointing to EL version 7.7.0 Can theese be updated to v 7.16.2 or will that break the frontend of Snowstorm?

//Per

rorydavidson commented 2 years ago

Hi @Eneuman, we have not yet tested Snowstorm with v7.16.2 of Elastic Search, and will do so before updating the pom in the repo. In the meantime, please feel free to try on your local copy. If you do encounter any issues, please do let us know.

kaicode commented 2 years ago

@Eneuman I've just run the full Snowstorm unit test suite against a Elasticsearch 7.16.2 instance and they all passed so we can have a good level of confidence that upgrading Elasticsearch should be fine.

[INFO] Tests run: 548, Failures: 0, Errors: 0, Skipped: 0

As Rory said please conduct your own testing because we have not taken this upgrade through our testing environments.

I did notice that Elasticsearch 7.16.2 returns the same one or two warnings for every call made from Snowstorm:

"Elasticsearch built-in security features are not enabled. Without authentication, your cluster could be accessible to anyone. See https://www.elastic.co/guide/en/elasticsearch/reference/7.16/security-minimal-setup.html to enable security.",
"[ignore_throttled] parameter is deprecated because frozen indices have been deprecated. Consider cold or frozen tiers in place of frozen indices."]

One is about enabling security and the other about a redundant parameter related to throttling. I think these can safely be ignored for now. SNOMED International currently deploy Elasticsearch behind a firewall rather than using the built in security options, I would recommend that.

Eneuman commented 2 years ago

@kaicode I did try ES 7.16.2 and I was getting errors when importing MAIN SNAPSHOT to a clean instance.

First I had to turn of xpack security since it's is now enabled by default and are causing alot of warnings on import. After that I keeped getting timeouts late in the process (from throtteling).

It also looks like SnowStorm client is using some depricated ElasticSearch API. Getting som warnings when it started up. Start a new instance in docker and check the logs. I dont have them any more otherwise I would have posted them here :(

Turning of security can be done by add theese lines to the elasticsearch yaml file (container env section): The changes ar backward compatible so they word for 7.7.0.

        - name: xpack.security.enabled
          value: "false"
kaicode commented 2 years ago

Thanks for the tip about disabling xpack 👍

Snowstorm 7.5.4 uses Elasticsearch Client 7.7.0 so that will produce a warning during startup when running against an Elasticsearch server with version 7.16.2 but they should still be compatible because they use semantic versioning. I was able to run all the automated integration tests using this combination, but to be fair that did not include whole RF2 imports.