Finally fixed. SymbolFinder build has a thread interference issue. Resolved for now using single threaded build for SymbolFinder. Seems to be a thread being killed for some reason. Hard to pinpoint. Think it's around printing error output from SourceFileResource::Compile but cannot be sure.
Extensive work around CommandThread, GenericThread, CommandOutputHandler and thread safety. Removes vast majority of build issues, including an old issue I'd noticed but not logged where build errors would sometimes be incomplete - the command would exit before stderr is fully processed, resulting in lost information. Now doesn't occur. Test suite added to prove. (Ran this 2500 times to ensure no failures). Was producing 0.4% failure rate in any command execution from Paladin before fix. Now 0% (for the mechanism).
Finally fixed. SymbolFinder build has a thread interference issue. Resolved for now using single threaded build for SymbolFinder. Seems to be a thread being killed for some reason. Hard to pinpoint. Think it's around printing error output from SourceFileResource::Compile but cannot be sure.
Extensive work around CommandThread, GenericThread, CommandOutputHandler and thread safety. Removes vast majority of build issues, including an old issue I'd noticed but not logged where build errors would sometimes be incomplete - the command would exit before stderr is fully processed, resulting in lost information. Now doesn't occur. Test suite added to prove. (Ran this 2500 times to ensure no failures). Was producing 0.4% failure rate in any command execution from Paladin before fix. Now 0% (for the mechanism).