search

eclipse / gemoc-studio

gemoc-studio
Eclipse Public License 1.0
24 stars 21 forks source link

Flaky tests #123

Open dvojtise opened 5 years ago

dvojtise commented 5 years ago

as seen in #64, some system test (often failing on SWTBot) however, it would be better to understand why it fails and add the correct timing (or control in the actions themselves) at the right place rather that just retry automatically as suggested in #64

So this issue will try to trace facts and relevant messages in order to fix them at the source: (we keep the title of the issue geneiric as we don't know yet the real cause(s) as there might have several.

For example, looking at ://ci.eclipse.org/gemoc/job/gemoc-studio/job/move_dsl_default_lochttpation_to_project_root/ the build 6 an d 8 there is no commit.

let's try to identify the real cause: current flaky tests in build 6: https://ci.eclipse.org/gemoc/job/gemoc-studio/job/move_dsl_default_location_to_project_root/6/

all 3 seems related to the modeling workbench

One has a strange issue about java version incompatibility (https://ci.eclipse.org/gemoc/job/gemoc-studio/job/move_dsl_default_location_to_project_root/6/testReport/junit/org.eclipse.gemoc.studio.tests.system.mwb.userstory/DebugOfficialExampleK3FSM_Test/test01_TwoStatesUpCast_Model_some_steps_stop_and_clear/) (UnsupportedClassVersionError: org/eclipse/gemoc/addon/multidimensional/timeline/views/MultidimensionalTimelineViewPart has been compiled by a more recent version of the Java Runtime (class file version 54.0)

this is strange since the test will pass later ...

in build 7: only 2 remains https://ci.eclipse.org/gemoc/job/gemoc-studio/job/move_dsl_default_location_to_project_root/7/testReport/

and pass completly in build 8

dvojtise commented 5 years ago

54 is java 10 and 52 is java 8. I wonder where we compile with java 10 ...

dvojtise commented 5 years ago

Actually SWTBot does a screenshot for failed test, thet are stored in a /screenshots/ folder of the test plugin.

I've enabled jenkins archive of these folders in order to ease the access to these files (don't know a better solution yet)

one of the strange fact about these flaky tests is that the windows seem to display the expected dialog but this dialog does not have a title bar

ex: for test failing with the following message

Message d'erreur

Could not find shell matching: with text 'Confirm Perspective Switch'

Pile d'exécution

org.eclipse.swtbot.swt.finder.exceptions.WidgetNotFoundException: Could not find shell matching: with text 'Confirm Perspective Switch'
    at org.eclipse.gemoc.studio.tests.system.mwb.userstory.DebugOfficialExampleK3FSM_Test.startDebugwoStatesUpCast_Model(DebugOfficialExampleK3FSM_Test.java:214)
    at org.eclipse.gemoc.studio.tests.system.mwb.userstory.DebugOfficialExampleK3FSM_Test.TwoStatesUpCast_Model_some_steps_stop_and_clear(DebugOfficialExampleK3FSM_Test.java:133)
    at org.eclipse.gemoc.studio.tests.system.mwb.userstory.DebugOfficialExampleK3FSM_Test.test01_TwoStatesUpCast_Model_some_steps_stop_and_clear(DebugOfficialExampleK3FSM_Test.java:123)
Caused by: org.eclipse.swtbot.swt.finder.widgets.TimeoutException: Timeout after: 8000 ms.: Could not find shell matching: with text 'Confirm Perspective Switch'
    at org.eclipse.gemoc.studio.tests.system.mwb.userstory.DebugOfficialExampleK3FSM_Test.startDebugwoStatesUpCast_Model(DebugOfficialExampleK3FSM_Test.java:214)
    at org.eclipse.gemoc.studio.tests.system.mwb.userstory.DebugOfficialExampleK3FSM_Test.TwoStatesUpCast_Model_some_steps_stop_and_clear(DebugOfficialExampleK3FSM_Test.java:133)
    at org.eclipse.gemoc.studio.tests.system.mwb.userstory.DebugOfficialExampleK3FSM_Test.test01_TwoStatesUpCast_Model_some_steps_stop_and_clear(DebugOfficialExampleK3FSM_Test.java:123)

image

dvojtise commented 5 years ago

Apparently, some flaky tests may also be due to the Discovery that checks for update site availability. As these sites may be very long to answer, this cause errors. in the discovery test and in some subsequent tests.

test03_toolbarContent(org eclipse gemoc studio tests system lwb feature XDSMLPerspective_Test)

We need to make sure to cancel this dialog in the test that launch it.

dvojtise commented 4 years ago

About other possible cause of flaky tests: In some situations the SWT name of the buttons ( Resume suspend etc) may change depending on which plugin contributes it first to the UI. I already had the pb with Step &Into (F5) versus Step &Into

Apparently our test sometimes fails on Timeout after: 20000 ms.: The widget (of type 'ToolItem' and with tooltip 'Resu&me (F8)' and with style 'SWT.PUSH') was not enabled.

which looks similar to the step into problem.

We need to make sure to search for both name variant in each action.

dvojtise commented 4 years ago

Sometimes the test fails with a timeout (currently set to 40min max to run the system tests)

When this occurs this is often on: org.eclipse.gemoc.studio.tests.system.lwb.userstory.CreateMelangeBasedSingleSequentialLanguageFromOfficialFSM_Test.test05_CreateSiriusEditorForBaseLanguage

that seems stuck while in normal condition it runs in less than 10s

Possible action: enable video recording in case of timeout in order to help identify this issue

dvojtise commented 2 years ago

in current version the flaky test occurs in - test02_DebugPredefinedK3Fsm_TwoStatesUpCast_Model_some_steps_and_stopafter having reported 

!MESSAGE Exception received Transaction is already closed, stopping engine.
!STACK 0
java.lang.IllegalStateException: Transaction is already closed
    at org.eclipse.emf.transaction.impl.TransactionImpl.commit(TransactionImpl.java:384)
    at org.eclipse.gemoc.executionframework.engine.core.AbstractExecutionEngine.commitCurrentTransaction(AbstractExecutionEngine.java:392)
    at org.eclipse.gemoc.executionframework.engine.core.AbstractExecutionEngine.startSynchronous(AbstractExecutionEngine.java:355)
    at org.eclipse.gemoc.executionframework.engine.core.AbstractExecutionEngine$1.run(AbstractExecutionEngine.java:335)
    at java.base/java.lang.Thread.run(Thread.java:834)

messages

dvojtise commented 1 year ago

May be consider migrating our SWTbot based test to RedDeer ?

https://projects.eclipse.org/projects/technology.reddeer https://www.eclipse.org/lists/reddeer-dev/msg00006.html