Open Robert-Adelard opened 3 years ago
I tried to reproduce this error locally. I'm able to store nodegroups to a non-localhost address.
To do this I first ran export BASE_URL=http://192.168.1.208
before ./setup-rack.sh
. I suspect you're doing something very similar.
Could your repeat this import while having the journal open as shown below to see if there's anything interesting in the log work sharing on this ticket? The code below shows me using docker container ls
to find the hash name of the container and then docker exec 447f8d5c8bc5 $ tail -f /var/log/journal/nodeGroupStoreService.service.log
to dump the log on that container live.
➜ ~ $ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
447f8d5c8bc5 gehighassurance/rack-box:dev "/usr/bin/python3 /u…" 8 days ago Up 8 days 0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:3030->3030/tcp, :::3030->3030/tcp, 0.0.0.0:12050-12092->12050-12092/tcp, :::12050-12092->12050-12092/tcp confident_elbakyan
➜ ~ $ docker exec 447f8d5c8bc5 tail -f /var/log/journal/nodeGroupStoreService.service.log
To clarify, when you say a "non-localhost" address, do you mean a server that is not localhost, or the real IP address of localhost, as opposed to the 127.0.0.1 IP address.
The reason I ask is because the docker commands you suggested would only work if the Docker was running on the local machine - they don't work if Docker is running on a separate server, which is my situation.
I asked my sys admin to take a look in the log file on the server, and we found lots of JSON nodegroups between START MESSAGE and END MESSAGE, but also the following error message from today:
2021-08-16 08:17:38 ----------END MESSAGE---------------
2021-08-16 08:17:38.264 INFO 241 --- [io-12056-exec-5] o.apache.coyote.http11.Http11Processor : Error parsing HTTP request header
Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.
java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:495) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]
2021-08-16 08:18:16 ----------START MESSAGE-------------
Regardless of the underlying cause, it would appear that SemTK is not handling this error condition properly and converting it into an JSON HTTP Error response, as documented here:
https://github.com/ge-semtk/semtk/wiki/REST-API#error-responses
As for the underlying cause, the problem seems to be caused by trying to store the nodegroup ingest_INTERFACE.json
.
In particular, I am seeing two failure modes for
rack nodegroups import ../nodegroups/ingestion/arcos.rack
In one case, only 7 nodegroups are loaded - the 8th nodegroup is ingest_INTERFACE
.
In the other case, 57 / 58 nodegroups are loaded - the missing nodegroup is ingest_INTERFACE
.
The strange thing is that my sys admin cannot reproduce the problem. So the problem seems to be somehow specific to my local computer / setup. Do you have any thoughts on what might be wrong?
That looks like Tomcat throwing the error, not SemTK. Which log file is quoted above?
I'm happy to join the hunt, but I don't feel like I see anything I understand yet. @tuxji is good with this type of thing, so I hope he could also help.
Perhaps the most important clue is that it works on most system configurations, but not on this one.
This is the NodeGroupStoreService
log file - it looks as though the exception is being thrown by the embedded Tomcat server in Spring Boot, but is somehow bypassing the error handling mechanism in SemTK, which I would expect to catch the exception and return an appropriate JSON formatted error message.
Our RACK server runs on a dedicated server rather than localhost.
I wanted to re-initialise the RACK so I did
rack data clear
,rack model clear
and tried to runsetup-rack.sh
.The output looked like this:
I ran a web proxy to capture the REST API requests and the "Storing nodegroups..." step completed without error.
This is reproducible - why would
storeNodeGroup
return HTML rather than a properly formatted JSON error message?