Blazegraph fails to start, provides no log output at all

[lyrixderaven]

Having followed the instructions, the run.sh script originally pulled version tag v2022.08.0 for all containers without problems, and also brought them up, but fails at the Getting blazegraph status step:

Getting blazegraph status
Error contacting http://localhost:8889/bigdata/namespace/kb/ (000)
Exiting

I have confirmed that the container is properly publishing port 8889 on the host; the java process inside the container is up and running, but also seems to incur 100% CPU load. The docker compose logs blazegraph output is empty.

Checking the docker hub versions, I found that the latest version tag seems to be v2023.05.0, which i then tried for the whole stack via ./run.sh -t v2023.05.0 as well, to no avail.

Next I've tried to force debug output from jetty by overwriting the startup command in the docker-compose.yml file:

  blazegraph:
    image: gridappsd/blazegraph${GRIDAPPSD_TAG}
    container_name: blazegraph
    ports:
      - 8889:8080
    command: "java -Xmx1g -Xdebug -Dorg.eclipse.jetty.util.log.class=org.eclipse.jetty.util.log.StdErrLog -Dorg.eclipse.jetty.server.LEVEL=DEBUG -Dcom.bigdata.rdf.sail.webapp.ConfigParams.propertyFile=/RWStore.properties -jar /usr/local/jetty/start.jar"

This gives, at least, minimal log output:

2023-07-31 13:36:34 2023-07-31 11:36:34.000:INFO:docker-entrypoint:jetty start from /var/lib/jetty/jetty.start
2023-07-31 13:42:09 2023-07-31 11:42:09.000:INFO:docker-entrypoint:jetty start from /var/lib/jetty/jetty.start

But it still doesn't seem like the jetty process is coming up properly; there does not seem to be any process listening on the internal :8080 port as well. Using the -d debug switch with run.sh didn't help either, as it just shows the 60 timed out attempts to reach port 8889 on the container.

Is there any way I can force the jetty container to spit out debug information during startup so I can figure out why it won't start the blazegraph instance? Or is there something else I might try?

[poorva1209]

@tonya1 Can you have a look at this?

[tonya1]

@lyrixderaven Blazegraph issues are usually due to a lack of memory allocated for docker. Can you try increasing the amount of memory allocated for docker?

[lyrixderaven]

My docker engine has had 16GB memory allowance to start with, which should be more than enough for a java-based application that is allocating a max heap size of 1GB as per the default "-Xmx1g" parameter. Just to confirm I've upped the memory limit to 24GB, to no avail.

I'm running this on an M2 Max Apple Mac (ARM processor), but normally, this has little impact on java being run inside a container; do you think this could be a problem in this case? Is there no way to get some debug logs from the jetty container, or blazegraph itself on startup?

[tonya1]

Great. That should be more than enough memory. Please try running an empty blazegraph container outside of the platform to see if it will startup and run. docker run --name blazegraph -d -p 8889:8080 lyrasis/blazegraph:2.1.4 docker logs blazegraph

[poorva1209]

@lyrixderaven Were you able to get it working?

[tonya1]

@lyrixderaven Were you able to get blazegraph running?

[lyrixderaven]

Apologies for the late response, just got back to the office now. I've tried to run a standalone blazegraph container as instructed, but to no avail - same symptom as before - no log output, connection reset when trying to access the blazegraph web UI, and the container is running at 100% cpu load.

[lyrixderaven]

Just a quick followup: it does appear that the lyrasis/blazegraph container is the culprit and not blazegraph itself; having tried https://hub.docker.com/r/openkbs/blazegraph-docker as an alternative I can get blazegraph up and running as expected, access the blazgraph webinterface and even get some log output for both the jvm and blazegraph. It may be prudent to adapt the gridappsd docker setup to use this alternative image instead, but that might require some additional adaptations to the setup procedures.