search

ge-high-assurance / RACK

DARPA's Automated Rapid Certification of Software (ARCOS) project called Rapid Assurance Curation Kit (RACK)
BSD 3-Clause "New" or "Revised" License
20 stars 6 forks source link

RACK - unable to run setup-rack.sh except through a proxy #526

Open Robert-Adelard opened 3 years ago

Robert-Adelard commented 3 years ago

Our RACK server runs on a dedicated server rather than localhost.

I wanted to re-initialise the RACK so I did rack data clear, rack model clear and tried to run setup-rack.sh.

The output looked like this:

./setup-rack.sh
Ingesting ../RACK-Ontology/OwlModels/AGENTS.owl... OK
Ingesting ../RACK-Ontology/OwlModels/ANALYSIS.owl...OK
Ingesting ../RACK-Ontology/OwlModels/CONFIDENCE.owl...OK
Ingesting ../RACK-Ontology/OwlModels/DOCUMENT.owl...OK
Ingesting ../RACK-Ontology/OwlModels/FILE.owl...   OK
Ingesting ../RACK-Ontology/OwlModels/HAZARD.owl... OK
Ingesting ../RACK-Ontology/OwlModels/MODEL.owl...  OK
Ingesting ../RACK-Ontology/OwlModels/PROCESS.owl...OK
Ingesting ../RACK-Ontology/OwlModels/PROV-S.owl... OK
Ingesting ../RACK-Ontology/OwlModels/REQUIREMENTS.owl...OK
Ingesting ../RACK-Ontology/OwlModels/REVIEW.owl... OK
Ingesting ../RACK-Ontology/OwlModels/SOFTWARE.owl...OK
Ingesting ../RACK-Ontology/OwlModels/SYSTEM.owl... OK
Ingesting ../RACK-Ontology/OwlModels/TESTING.owl...OK
The following nodegroups do not exist: ^query
Deleting ingest_CODE_GEN...                        OK
Deleting ingest_SYSTEM_DEVELOPMENT...              OK
Deleting ingest_MODEL...                           OK
Deleting ingest_TEST_DEVELOPMENT...                OK
Deleting ingest_AGENT...                           OK
Deleting ingest_SECTION...                         OK
Deleting ingest_THING...                           OK
Storing nodegroups...                                       FAIL
[rack:74] ERROR: REST Endpoint Failure
http://rack.office.adelard.com:12056/nodeGroupStore/storeNodeGroup: failed with reason: <!doctype html><html lang="en"><head><title>HTTP Status 400 – Bad Request</title><style type="text/css">body {font-family:Tahoma,Arial,sans-serif;} h1, h2, h3, b {color:white;background-color:#525D76;} h1 {font-size:22px;} h2 {font-size:16px;} h3 {font-size:14px;} p {font-size:12px;} a {color:black;} .line {height:1px;background-color:#525D76;border:none;}</style></head><body><h1>HTTP Status 400 – Bad Request</h1></body></html>
Details:

I ran a web proxy to capture the REST API requests and the "Storing nodegroups..." step completed without error.

This is reproducible - why would storeNodeGroup return HTML rather than a properly formatted JSON error message?

glguy commented 3 years ago

I tried to reproduce this error locally. I'm able to store nodegroups to a non-localhost address.

To do this I first ran export BASE_URL=http://192.168.1.208 before ./setup-rack.sh. I suspect you're doing something very similar.

Could your repeat this import while having the journal open as shown below to see if there's anything interesting in the log work sharing on this ticket? The code below shows me using docker container ls to find the hash name of the container and then docker exec 447f8d5c8bc5 $ tail -f /var/log/journal/nodeGroupStoreService.service.log to dump the log on that container live.

➜  ~ $ docker container ls
CONTAINER ID   IMAGE                          COMMAND                  CREATED      STATUS      PORTS                                                                                                                                                 NAMES
447f8d5c8bc5   gehighassurance/rack-box:dev   "/usr/bin/python3 /u…"   8 days ago   Up 8 days   0.0.0.0:80->80/tcp, :::80->80/tcp, 0.0.0.0:3030->3030/tcp, :::3030->3030/tcp, 0.0.0.0:12050-12092->12050-12092/tcp, :::12050-12092->12050-12092/tcp   confident_elbakyan
➜  ~ $ docker exec 447f8d5c8bc5 tail -f /var/log/journal/nodeGroupStoreService.service.log
Robert-Adelard commented 3 years ago

To clarify, when you say a "non-localhost" address, do you mean a server that is not localhost, or the real IP address of localhost, as opposed to the 127.0.0.1 IP address.

The reason I ask is because the docker commands you suggested would only work if the Docker was running on the local machine - they don't work if Docker is running on a separate server, which is my situation.

Robert-Adelard commented 3 years ago

I asked my sys admin to take a look in the log file on the server, and we found lots of JSON nodegroups between START MESSAGE and END MESSAGE, but also the following error message from today:

2021-08-16 08:17:38 ----------END MESSAGE---------------
2021-08-16 08:17:38.264  INFO 241 --- [io-12056-exec-5] o.apache.coyote.http11.Http11Processor   : Error parsing HTTP request header
 Note: further occurrences of HTTP request parsing errors will be logged at DEBUG level.

java.lang.IllegalArgumentException: Invalid character found in the request target. The valid characters are defined in RFC 7230 and RFC 3986
        at org.apache.coyote.http11.Http11InputBuffer.parseRequestLine(Http11InputBuffer.java:495) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:260) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:65) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:868) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1594) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[na:na]
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[na:na]
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) ~[tomcat-embed-core-9.0.33.jar:9.0.33]
        at java.base/java.lang.Thread.run(Thread.java:829) ~[na:na]

2021-08-16 08:18:16 ----------START MESSAGE-------------

Regardless of the underlying cause, it would appear that SemTK is not handling this error condition properly and converting it into an JSON HTTP Error response, as documented here:

https://github.com/ge-semtk/semtk/wiki/REST-API#error-responses

Robert-Adelard commented 3 years ago

As for the underlying cause, the problem seems to be caused by trying to store the nodegroup ingest_INTERFACE.json.

In particular, I am seeing two failure modes for 

rack nodegroups import ../nodegroups/ingestion/arcos.rack

In one case, only 7 nodegroups are loaded - the 8th nodegroup is ingest_INTERFACE.

In the other case, 57 / 58 nodegroups are loaded - the missing nodegroup is ingest_INTERFACE.

The strange thing is that my sys admin cannot reproduce the problem. So the problem seems to be somehow specific to my local computer / setup. Do you have any thoughts on what might be wrong?

cuddihyge commented 3 years ago

That looks like Tomcat throwing the error, not SemTK. Which log file is quoted above?

I'm happy to join the hunt, but I don't feel like I see anything I understand yet. @tuxji is good with this type of thing, so I hope he could also help.

Perhaps the most important clue is that it works on most system configurations, but not on this one.

Robert-Adelard commented 3 years ago

This is the NodeGroupStoreService log file - it looks as though the exception is being thrown by the embedded Tomcat server in Spring Boot, but is somehow bypassing the error handling mechanism in SemTK, which I would expect to catch the exception and return an appropriate JSON formatted error message.