blazegraph truncate journal

earthcube / geocodes

This is the containers stacks to run geocodes

0 stars 0 forks source link

blazegraph truncate journal #33

Closed valentinedwv closed 10 months ago

valentinedwv commented 1 year ago

The famous, hey, can we document the exact steps for our systems, and not just the see here, so we can do this repeatedly, with a script request to truncate the blazegraph journal.

total 5823840
-rw-r--r--    1 root     root     6583746560 Jul 17 18:31 blazegraph.jnl
-rw-r--r--    1 root     root             0 May 15 18:52 rules.log

valentinedwv commented 1 year ago

MB managed this a one point.

Question is can this be documented on how to do it on our present stacks,

log into portainer,
start/stop service if needed,
log into graph container terminal
run command
command takes and estimate time of,
start server

so we can have a repeatable procedure, and maybe even a shell script.

ylyangtw commented 1 year ago

also check blazegraph truncate journal's usage

valentinedwv commented 1 year ago

Thought on possibly scripting this using the docker/portainer api... once we know if the start/stop technique works

using docker.py module

create a client like the scheduler generate code, so we can use the portainer as the proxy
find the service, get the running container.
execute the command to get rid of the journal cruft.
update the service so that it does not restart. config might be in this area
stop container
run command to copy truncated file to proper location
start container.
wait, query with simple call
if it works, then update the service to restart if failed.

valentinedwv commented 1 year ago

I think i was reading this wrong, and based on experience with Fuseki issues with journal filling disk.

Reading the doc says it will recover journal when it hits 200m statements.

I might be misunderstanding how the journaling works... might just be we have not hit the recylce limits. https://github.com/blazegraph/database/wiki/CommonProblems#problem-i-am-using-the-journal-and-the-file-size-grows-very-quickly 12:58 Looks like we are using the rw store this is from the docker file. https://github.com/nawerprod/blazegraph/blob/master/RWStore.properties

ylyangtw commented 1 year ago

putting this off. we probably don't need to worry about this.