IHTSDO / snowstorm

Scalable SNOMED CT Terminology Server using Elasticsearch
Other
209 stars 84 forks source link

How do I know where the actual SNOMED CT Release is stored on disk? #386

Closed anibal2j closed 2 years ago

anibal2j commented 2 years ago

Hi,

I need to manage the storage on the snowstorm server and I can't find where is the actual storage of the release I loaded.

I need to upload 2 more releases and I'd like to be confident that I'm placing them in the right partition that I have available.

I'm not really talking about the application per se, but the actual SNOMED release and/or different branches.

Thanks,

-Anibal

kaicode commented 2 years ago

Hi! Everything is stored in Elasticsearch. On a local deployment the data goes into a directory on disk like elasticsearch/data/. Data may not all be flushed to disk in this directory until Elasticsearch is shutdown. There are standard ways to backup and restore Elasticsearch data, in their documentation. I hope that helps.

anibal2j commented 2 years ago

Thanks for the quick response. I can't find the data directory. I don't have man pages installed (not sure why they were not installed).

Look:

[snowstorm@tcn00qubc03104 /usr/share/elasticsearch]$ ps -ef | fgrep elastic
root      206162       1  0 Mar24 ?        00:28:21 /usr/share/elasticsearch/jdk/bin/java -Xshare:auto -Des.networkaddress.cache.ttl=60 -Des.networkaddress.cache.negative.ttl=10 -XX:+AlwaysPreTouch -Xss1m -Djava.awt.headless=true -Dfile.encoding=UTF-8 -Djna.nosys=true -XX:-OmitStackTraceInFastThrow -XX:+ShowCodeDetailsInExceptionMessages -Dio.netty.noUnsafe=true -Dio.netty.noKeySetOptimization=true -Dio.netty.recycler.maxCapacityPerThread=0 -Dio.netty.allocator.numDirectArenas=0 -Dlog4j.shutdownHookEnabled=false -Dlog4j2.disable.jmx=true -Djava.locale.providers=SPI,COMPAT -Xms4g -Xmx4g -XX:+UseG1GC -XX:G1ReservePercent=25 -XX:InitiatingHeapOccupancyPercent=30 -Djava.io.tmpdir=/tmp/elasticsearch-7057980869020215243 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/var/lib/elasticsearch -XX:ErrorFile=/var/log/elasticsearch/hs_err_pid%p.log -Xlog:gc*,gc+age=trace,safepoint:file=/var/log/elasticsearch/gc.log:utctime,pid,tags:filecount=32,filesize=64m -XX:MaxDirectMemorySize=2147483648 -Des.path.home=/usr/share/elasticsearch -Des.path.conf=/etc/elasticsearch -Des.distribution.flavor=default -Des.distribution.type=rpm -Des.bundled_jdk=true -cp /usr/share/elasticsearch/lib/* org.elasticsearch.bootstrap.Elasticsearch -d -p /home/snowstorm/snowstorm/elasticsearch_pid.pid
root      206185  206162  0 Mar24 ?        00:00:00 /usr/share/elasticsearch/modules/x-pack-ml/platform/linux-x86_64/bin/controller
snowsto+  619168  618877  0 10:51 pts/2    00:00:00 grep -F --color=auto elastic

So, I don't see any data dir in there and nothing under /usr/share/elasticsearch either:

[snowstorm@tcn00qubc03104 /usr/share/elasticsearch]$ find /usr/share/elasticsearch/ -name data
[snowstorm@tcn00qubc03104 /usr/share/elasticsearch]$

I know this is not a forum for elasticsearch, but maybe you know. If not I'll have to contact somebody with elasticsearch knowledge. I just want to make sure I don't run out of disk space by using the wrong partitition.

anibal2j commented 2 years ago

Found it!

In my installation the configuration for elasticsearch is under /etc/elasticsearch, where I can find elasticsearch.yml which has the value of path.data: /var/lib/elasticsearch

And if I go there I have this:

[root@tcn00qubc03104 /var/lib/elasticsearch]# du -sh /var/lib/elasticsearch
2.9G    /var/lib/elasticsearch

Which sounds about right.

kaicode commented 2 years ago

@anibal2j Glad you found it! For the record which operating system are you using and which installation method did you use? Might help others later 😄

anibal2j commented 2 years ago
Red Hat Enterprise Linux release 8.5 (Ootpa)

And I used a combination of installation procedures - mostly helped by Rory :-) It was done several months ago and I forgot but tried to follow what's documented in the home page of this project here in github.

I'm about to do, for the first time, an update of our SNOMED CT release as well as adding the UK release on our own snowstorm installation, let's see how that goes...

kaicode commented 2 years ago

Thank you. Best of luck!