Jenkins CI failing to complete builds

[jonahgraham]

We haven't had a successful build since 2024-03 M1 build a couple of weeks ago. And we haven't had a completed build since 18 Jan. The builds since are failing with timeout exceptions.

Here is a sample:

[2024-01-24T21:39:01.726Z] Running org.eclipse.cdt.tests.dsf.gdb.tests.MIDisassemblyTest
[2024-01-24T21:39:04.525Z] 341,829 "readMixedWithValidAddress[gdb.10]" requesting /shared/common/gdb/gdb-all/bin/gdb.10.Uncaught exception in session executor thread: java.lang.AssertionError: Exception thrown by a IServiceEventListener.ServiceHandlerMethod method
[2024-01-24T21:39:04.525Z]  at org.eclipse.cdt.dsf.service.DsfSession.doDispatchEvent(DsfSession.java:550)
-- snip --
[2024-01-24T21:39:16.347Z]  at java.base/java.lang.Thread.run(Thread.java:833)
[2024-01-24T21:39:16.347Z] 
[2024-01-24T22:03:28.798Z] Cancelling nested steps due to timeout

Looking at the workspace log I see this:

java.lang.AssertionError: Exited event should only be received for a started inferior
    at org.eclipse.cdt.dsf.mi.service.command.MIInferiorProcess_7_3.eventDispatched(MIInferiorProcess_7_3.java:86)

[jonahgraham]

Exited event should only be received for a started inferior implies that something about the synchronization of gdb and dsf has gone wrong. The only changes since the last successful build are the asm changes (233c6d8696eddb78c2104a91473e933a0c1a88b4), but I don't think that is the cause and is more likely that it is just a coincidence on timing as I don't how that can be related to the failures.

[jonahgraham]

I can see it looks like something changed in the CI environment as gdb isn't working even for simple (non-attach) cases with ptrace errors (extract from this test build):

[2024-01-29T21:04:26.288Z] + /shared/common/gdb/gdb-all/bin/gdb.10 --interpreter=mi2 -ex run -ex quit MemoryTestApp.exe
[2024-01-29T21:04:26.547Z] =thread-group-added,id="i1"
[2024-01-29T21:04:26.547Z] ~"GNU gdb (GDB) 10.2\n"
[2024-01-29T21:04:26.547Z] ~"Copyright (C) 2021 Free Software Foundation, Inc.\n"
[2024-01-29T21:04:26.547Z] ~"License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>\nThis is free software: you are free to change and redistribute it.\nThere is NO WARRANTY, to the extent permitted by law."
[2024-01-29T21:04:26.547Z] ~"\nType \"show copying\" and \"show warranty\" for details.\n"
[2024-01-29T21:04:26.547Z] ~"This GDB was configured as \"x86_64-pc-linux-gnu\".\n"
[2024-01-29T21:04:26.547Z] ~"Type \"show configuration\" for configuration details.\n"
[2024-01-29T21:04:26.547Z] ~"For bug reporting instructions, please see:\n"
[2024-01-29T21:04:26.547Z] ~"<https://www.gnu.org/software/gdb/bugs/>.\n"
[2024-01-29T21:04:26.547Z] ~"Find the GDB manual and other documentation resources online at:\n    <http://www.gnu.org/software/gdb/documentation/>."
[2024-01-29T21:04:26.547Z] ~"\n\n"
[2024-01-29T21:04:26.547Z] ~"For help, type \"help\".\n"
[2024-01-29T21:04:26.547Z] ~"Type \"apropos word\" to search for commands related to \"word\"...\n"
[2024-01-29T21:04:26.547Z] ~"Reading symbols from MemoryTestApp.exe...\n"
[2024-01-29T21:04:26.547Z] &"warning: Could not trace the inferior process.\n"
[2024-01-29T21:04:26.547Z] &"warning: ptrace: Operation not permitted\n"
[2024-01-29T21:04:26.547Z] =thread-group-started,id="i1",pid="343"
[2024-01-29T21:04:26.547Z] =thread-created,id="1",group-id="i1"
[2024-01-29T21:04:26.547Z] =thread-exited,id="1",group-id="i1"
[2024-01-29T21:04:26.547Z] =thread-group-exited,id="i1"
[2024-01-29T21:04:26.547Z] &"During startup program exited with code 127.\n"

[jonahgraham]

I have added to https://gitlab.eclipse.org/eclipsefdn/helpdesk/-/issues/4217 because it looks like we aren't the only affected project.

In the meantime I will probably have to disable the tests so that we can get build to complete.

[jonahgraham]

The workaround in #685 means we have clean Jenkins builds - but ones that don't run DSF-GDB tests.