search

POETSII / Orchestrator

The Orchestrator is the configuration and run-time management system for POETS platforms.
1 stars 1 forks source link

Orchestrator takes around 15 seconds to startup and shutdown on machines with hardware #250

Closed m8pple closed 3 years ago

m8pple commented 3 years ago

This is on development-dt10-changes(ac3abc5e0b629f78d97d50461040da43d66a43bc). I checked out 1.0.0-alpha, recompiled and so on, and it applies there too.

On machines without hardware the orchestrator can be started and stopped quite quickly, which is very useful for running automated tests, particularly related to parsing and automation.

However, on machines with hardware, the minimum time between the orchestrator starting and it exiting normally is about 15 seconds:

dt10@ayres:~/Orchestrator$ time (echo "exit" | ./orchestrate.sh )
POETS>Exiting...
Mothership:CommonBase (ayres) Main closing down.

real    0m14.576s
user    0m56.721s
sys     0m1.416s

So even if a test passes or fails quickly, it still takes 15 seconds for the pass/fail to complete. Some testers may explicitly kill the orchestrator, or close stdout, but well-behaved ones that send in exit will be hanging around a while.

This is probably not a big deal, as people writing automated tests can explicitly kill the orchestrator, but is quite annoying for interactive use. You end up typing exit, and then hitting ctrl-c to actually quit.

The timing of ~15 seconds is very suggestive, as that is how long HostLink/DebugLink often takes to actually acquire hardware - maybe they are related. Also, if I start the orchestrator, wait 10 seconds and type exit, then it takes about another 4 seconds to exit.

mvousden commented 3 years ago

You're quite right - they are related. The Orchestrator, in this case, is waiting for Motherships to close down, which is blocked by HostLink's startup and teardown.

From the test-writer's perspective, run ./orchestrate.sh -n. The launcher supports the -n flag to start an Orchestrator without Motherships (which is what you want to do, if you are just testing XML).

heliosfa commented 3 years ago

Given the nature of things (e.g. not wanting to leave stale processes floating around on close down and it being related to hardware acquisition), there is not a lot that we can do to speed this up.

One question is should testers be running tests that don't involve the hardware on the POETS boxes in the general case? Surely it would make more sense to run tests that don't involve hardware locally/on another suitably beefy machine. At which point, the Orchestrator starts without a Mothership and the delay disappears.

m8pple commented 3 years ago

From a test point of view I generally prefer to log into the machine and run every single test that exists and can run on that machine. So when I log into a machine with hardware I just do make test and it tests the entire orchestrator. Generally I've found that if you have to manually run each batch of tests then if you are testing hardware you'll only run the hardware batch tests, and may commit a change that fails the parsing regression tests.

However, the ./orchestrate.sh -n thing should fix that, as the non hardware ones can opt out.

Thanks.