search

Terasology / ModuleTestingEnvironment

3 stars 15 forks source link

Tests can be overly resource hungry or otherwise get out of control - add hardening somehow? #30

Open Cervator opened 3 years ago

Cervator commented 3 years ago

Maybe mildly similar to #25 but distinct - in this case testExample literally went out of memory:

http://jenkins.terasology.io/teraorg/job/Terasology/job/Modules/job/M/job/ModuleTestingEnvironment/job/PR-29/1/console

I'm not sure what the best way to address this is. It doesn't look like the build agent itself ran out of memory, it had a little more space allocated, although maybe if Java requested a large enough chunk of memory (more than 300MB in one reqest?) that could be related

image

Maybe it is something more specific? The build log talks about "GC overhead limit exceeded"

DeprecationTest > testExample() STANDARD_ERROR
    Exception in thread "Chunk-Processing-Reactor" java.lang.OutOfMemoryError: GC overhead limit exceeded
    Exception in thread "Thread-148" java.lang.OutOfMemoryError: GC overhead limit exceeded

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "Chunk-Processing-6"
DarkWeird commented 3 years ago

I suspect it is not terminated engines during all tests run. Something with shutdowning. Every client engine takes 600MB+ by chunks (default view distance and server and client sides)

Cervator commented 3 years ago

Yeah I tried to add more memory to the build agent but it still failed eventually. On the plus side we now have the watchful http://jenkins.terasology.io/teraorg/job/Utilities/job/BuildWatchinator/ which will find and terminate such stalled builds. So at least they won't stall forever anymore