Server Crashing for concurrent agents

[aryankhare2907]

I am working on a spring boot application where we are creating a restful service which on each call creates a Kernel in New Thread, The application works perfectly fine when we calls are made separately(one after another).However when we use 2 threads using (JMeter) to call the endpoint, it is causing the server to crash,

public boolean installAgent(String path, String name) {

 System,out,println("test"); //test gets printed twice 

 agent = kernel.CreateAgent(name + UUID.randomUUID().toString());

  System.out.println(agent.GetAgentName());
  boolean result = agent.LoadProductions(path); // code Breaks somewhere around here " only for concurrent calls"

    }

    I am using windows 10 jdk-11,Soar 9.6.0 (with soar.dll path setup in env variables)

 Is this a known issue ?

[marinier]

CSoar (the C++ implementation of Soar) is not really threaded in the way you want it to be. You're calling create kernel in new thread, but behind the scenes the kernel is a singleton and they are all the same, and you can't interact with that kernel from multiple threads. That is, all the agents are running on the same thread. There is more information here: https://soar.eecs.umich.edu/articles/articles/soar-markup-language-sml/203-threads-in-sml

You may consider JSoar (the java implementation of Soar), which has threaded agents that run completely independently from each other. JSoar is a bit behind CSoar (it's roughly equivalent to CSoar 9.4), but if that doesn't matter to you, this will be much easier from a threading perspective.

Some info:

https://github.com/soartech/jsoar/wiki https://github.com/soartech/jsoar/wiki/JSoarUsersGuide (see the ThreadedAgent section)

Full disclosure: I am the maintainer of JSoar.

[aryankhare2907]

Thanks @marinier for the response!! I am little confused may be I am not able to explain my question very well! So when I open two JAVA Debuggers windows on my system and have two agents running on them parallelly , doesn't that mean that we are having two different kernels and agents running together. All we we want to achieve is have multiple threads call a same restful endpoint. And in response they create multiple kernels and each having a agent instance on them? I have attached the code snippets. Is it not possible using CSOAR Java Code snippet.txt Pyhton Snippet.txt

[marinier]

CSoar does not run the agents in parallel. Internally, the kernel has a scheduler and it runs the agents in a round-robin fashion, with various parameters controlling how long an agent runs until the next one gets to run. By default, I think it runs each agent for 1 decision cycle at a time. (See the Soar manual's section on the run command for information about changing the interleave size and other parameters.)

You can confirm that all your agents are in the same underlying kernel by looking at the agent list in the debugger (Agents -> Select Current Agent will bring up a list of agents). Anything shown here is running in the same thread.

We can trace the code to see what's happening:

• CreateKernelInNewThread calls
• CreateEmbeddedConnnection, which calls
• CreateSoarManagers, which calls
• Soar_Instance::Get_Soar_Instance

This last one is a static method in Core/SoarKernel/src/shared/soar_instance.h. There is only one Soar_Instance object, and thus every time you create a kernel, what you're really doing is creating a new wrapper for this same object.

The only way I'm aware of to have multiple agents running in parallel in CSoar is to have them in separate processes. Essentially, you have to have a way to create separate Soar_Instance objects even though it's static. I'm not a C++ expert, so maybe there's a clever way of forcing that to happen within a single process.

JSoar doesn't have this problem because it doesn't have the concept of a Kernel at all. You create individual agents, which are completely separate from each other, and your application is responsible for running them however you want (i.e., there is no scheduler -- your application is responsible for running them however you want, which is usually to just let them run completely separately). (ThreadedAgents take care of setting up independent threads for you, but you can do it on your own if you have a reason to.)

[aryankhare2907]

Thanks a lot @marinie totally makes sense!! I see agents running parallelly and working fine when they are in separate process!! However I noticed it also works on same process when two threads (each creating creating agent and kernel in sml) are running sequentially when I connect remotely to both agents and do "select current agent" shows only different agent for both kernel . However when two threads start at same time code just hangs for some reason (deadlock probably) And I also see an old conversation where it seems this was fixed? Link to the conversation- https://github.com/SoarGroup/Soar/issues/63 We have CSOAR code already developed and we are on our way for deployments so changing to JSOAR might not be very good option for us for this particular project. You have any thoughts or ideas if we can try to make this work on CSOAR ?

[marinier]

Unfortunately, that issue is so old the commit number refers to an SVN commit. While the SVN history was imported to git at some point, I can't find this particular commit on or around that date, so I can't tell for sure what was actually done there. But walking through the code as I did above seems to indicate that, at least at the SML level, things are still single threaded (or at least all executing within the same Soar instance), which is single threaded.

One thing you might try is making sure that any interaction with Soar that you do happens on the Soar thread. E.g., instead of just loading productions in the agent directly, register for a Soar event (e.g., before input phase on the agent, or system start on the kernel) and load the productions in the event callback. If things actually are single threaded, this will ensure that only one thing is happening to the Soar agent at a time, as all actions will come from the same thread (i.e., the Soar thread). If things are actually multi-threaded this may not help, but it's hard to say -- there's a good chance it still will.

[garfieldnate]

We have an application at CIC that suffers from this as well; I think it might be beneficial to print a warning of some kind when Soar agents are created on different threads from where the kernel resides. This wouldn't prevent clients from creating an agent on one thread and trying to use it on another, but it might be a good start.