REST API sometimes needs to be reset between tests

[Sophietje]

In GitLab by @david.heck.iee.fraunhofer.de on Jun 24, 2021, 13:22

Currently I am using the REST API locally by running: \knowledge-engine\smart-connector-rest-dist\target>java -Dorg.slf4j.simpleLogger.logFile=ke.log -cp "smart-connector-rest-dist-0.1.10.jar;dependency/*" eu.interconnectproject.knowledge_engine.rest.Main 8280

After which I start a service, which reacts to interactions, and another one that posts interactions.

Now when I stop a service, change something in the code, build and run the service again sometimes the posts do not trigger the reacts. I have to stop the REST API and restart my services, after which the posts trigger the reacts as expected.

My services are written in java and the post/react knowledge interaction is mainly based on the clients in: https://gitlab.inesctec.pt/interconnect/knowledge-engine/-/tree/master/smart-connector-rest-server/src/main/java/eu/interconnectproject/knowledge_engine/rest/api/client_example/post_react

As a result of this I am resetting the REST API after every change which is kind of tiresome.

[Sophietje]

In GitLab by @barry.nouwt.tno.nl on Jun 24, 2021, 13:37

Hey @david.heck.iee.fraunhofer.de, thanks for posting.

You mention that the problem does not happen the first time that you start your post and react KBs, but does sometimes happen when you restart them. Could it be the case that the post and react services do not clean up their smart connectors correctly? This would mean that there are more than two KBs (with Smart Connectors) registered with the Knowledge Engine (2 new ones, but also the two previously registered Knowledge Bases). Since the graph patterns of the react are probably the same, the Knowledge Engine allows both of them to react one after the other. But if the first one never reacts, the Knowledge Engine keeps waiting. So, after closing your services the first time, double check whether the GET /rest/sc returns an empty list. If not, than that might be the cause for the instabilities.

The latest version of the clients you mention use a stop() method to unregister everything correctly and make sure it still works the second time. See this line of code.

See also the related FAQ whose question starts with: I use a very generic graph pattern like (I should definitely give FAQs a number).

Let me know the result of the POST /rest/sc call after shutting down your services, but before restarting them.

Kind regards, Barry

[Sophietje]

In GitLab by @david.heck.iee.fraunhofer.de on Jun 25, 2021, 07:17

Hey, @barry.nouwt.tno.nl,

I am using

Runtime.getRuntime().addShutdownHook(new Thread(() -> {
    client.close();
}));

client.startLongPoll();

Though I am not sure when exactly this gets triggered.

It is difficult to reproduce the behavior, sometimes interaction cannot register at all, sometimes nothing is received similar to your description.

Do I understand this correctly? If there is more than one knowledgebase with the same react graph pattern and the knowledgebase first in line does not answer, the interaction will stall indefinitely. What happens if one knowledgebase goes offline, cause of some unforeseen event? Is this really the desired behavior or should there never be more than one knowledgebase with the same graph pattern in the first place?

The error I was able to reproduce is as followed:

I start the REST API, start my service, post something to it, everything works as expected. Now I exit my service, start it again, but when I post something to it I get the following error from the service, while the poster gets a timeout:

Exception in thread "OkHttp Dispatcher" java.lang.NullPointerException
    at eu.interconnectproject.knowledge_engine.rest.api.client_example.KnowledgeEngineRestApiClient$2.onResponse(KnowledgeEngineRestApiClient.java:240)
    at okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:519)
    at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
    at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
    at java.base/java.lang.Thread.run(Thread.java:830)

Regards, David

[Sophietje]

In GitLab by @barry.nouwt.tno.nl on Apr 25, 2022, 09:33

Hi @david.heck.iee.fraunhofer.de, has this issue been resolved?

Regarding your question "Do I understand this correctly?": yes, you understand correctly. One of the ways in which the InterConnect Framework tries to improve the interoperability is by allowing other Knowledge Bases (KBs) to hook into existing data exchanges. To achieve this, the Knowledge Engine indeed triggers all KBs that have a matching graph pattern and expects them to respond accordingly. This assumption of the Knowledge Engine about a KB always being able to respond is risky, because it means that behavior as you described above could happen. This is why we stress the best practice to always unregister your KB from the REST server if something goes wrong. So, your shutdown hook is correct. It ensures that the KB is unregistered whenever the Java Virtual Machine (JVM) ends, although there are still some situations in which this shutdown hook is not triggered. The latest versions of the Knowledge Engine also have a mechanism to automatically remove inactive KB.

I am unsure what is causing the NullPointerException. Could you check this with the latest version?

Best regards, Barry

p.s. sorry for the late response

[Sophietje]

In GitLab by @david.heck.iee.fraunhofer.de on Apr 28, 2022, 11:00

Hi @barry.nouwt.tno.nl, I am not sure if it has been fully resolved.

However I have not seen this error in quite a while. I currently am on the newest version of the Knowledge Engine so my guess is the KBs are unregistered correctly and the situation with the NullPointer never has a chance to occur.

I did quite a lot of debugging the last weeks during which I tested many changes to my code with the same Knowledge Engine instance and I had no problem with it.

Regards, David

[Sophietje]

In GitLab by @barry.nouwt.tno.nl on May 3, 2022, 06:48

Ok, thanks! I'll close the issue, but if it pops up again you can reopen it.