search

MarathonLabs / marathon

Cross-platform test runner
https://docs.marathonlabs.io
GNU General Public License v2.0
577 stars 119 forks source link

Reports build failure when all the test cases pass #660

Closed umesh-onshape closed 1 year ago

umesh-onshape commented 2 years ago

Describe the bug I explicitly specified 208 test cases to run using filter with fully qualified test names. All these 208 test cases pass but marathon reports failure as it say s it expected results from 220 test cases.

To Reproduce Steps to reproduce the behaviour:

  1. Full configuration: specified in the marathon file

name: "Onshape" outputDir: "build/reports/marathon" vendorConfiguration: type: "Android" installOptions: "-r -t" autoGrantPermission: true androidSdk: "$HOME/build_tools/android-sdk-31" applicationApk: "./build/outputs/apk/debug/Onshape-debug.apk" testApplicationApk: "./build/outputs/apk/androidTest/debug/Onshape-debug-androidTest.apk" batchingStrategy: type: "isolate" flakinessStrategy: type: "ignore" filteringConfiguration: allowlist:

Expected behavior Marathon should send report success as all the specified test cases pass.

Logs and reports

2022-06-16 01:07:36 D 01:07:36.879 [main @coroutine#34] <QueueActor[DevicePoolId(name=omni)]> request next batch for device 127.0.0.1:5037:emulator-5554 2022-06-16 01:07:37 Allure environment data saved. 2022-06-16 01:07:37 Marathon run finished: 2022-06-16 01:07:37 Device pool omni: 2022-06-16 01:07:37 208 passed, 0 failed, 0 ignored tests 2022-06-16 01:07:37 Flakiness overhead: 10361ms 2022-06-16 01:07:37 Raw: 220 passed, 0 failed, 0 ignored, 0 incomplete tests 2022-06-16 01:07:37 Total time: 1H 2m 42s 2022-06-16 01:07:37 2022-06-16 01:07:37 E 01:07:37.743 [main @coroutine#1] <c.m.m.e.p.tracker.PoolProgressTracker> Expected to run 220 tests but received results for only 208 Devices (please complete the following information):

Additional context Add any other context about the problem here.

Malinskiy commented 2 years ago

Hey @umesh-onshape

So where do you get the other 12 test runs from? You should be able to understand why from the standard output.

In terms of solution - pretty sure #644 should solve this regardless of the source of retries, but I still want to understand why there are more retries invoked here

umesh-onshape commented 2 years ago

I see some test cases run twice, even with no retries asked for in the marathon configuration file as shown above.

Malinskiy commented 2 years ago

@umesh-onshape can you please investigate why the retries are happening for at least one test? E.g. when they’re starting, what’s the result, etc. Maybe they’re mentioned twice in the list requested

umesh-onshape commented 2 years ago

Consider the case of testPermissionFull, testCanDeleteAsOwner and testPermissionCopy , these are present two times in the overall test list before marathon start executing tests, though it is present only once in the app/MarathonFile

Also note right at the beginning, log shows marathon will execute 220 test cases, because some of them are duplicated. Again these occur only once in the config file. Let me know what else I should look out for.

2022-06-16 00:04:56 I 00:04:55.742 [main @coroutine#1] <com.malinskiy.marathon.Marathon> java.vm.version: 11.0.11+9-LTS
2022-06-16 00:04:56 I 00:04:55.742 [main @coroutine#1] <com.malinskiy.marathon.Marathon> java.class.version: 55.0
2022-06-16 00:04:56 D 00:04:56.244 [main @coroutine#1] <AdamDeviceProvider> Android Debug Bridge /127.0.0.1:5037: version 41
2022-06-16 00:04:56 D 00:04:56.302 [DeviceMonitor @coroutine#6] <AdamDeviceProvider> Device emulator-5554 changed state to CONNECTED
2022-06-16 00:04:56 D 00:04:56.342 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#12] <AndroidDevice> Device 127.0.0.1:5037:emulator-5554 booted!
**2022-06-16 00:04:56 I 00:04:56.463 [main @coroutine#1] <com.malinskiy.marathon.Marathon> Scheduling 220 tests**
2022-06-16 00:04:56 D 00:04:56.464 [main @coroutine#1] <com.malinskiy.marathon.Marathon> com.belmonttech.app.activities.BTDocumentListActivityTest#testCapability, com.belmonttech.app.activities.BTDocumentListActivityTest#testDeleteAndRestore, com.belmonttech.app.activities.BTDocumentListActivityTest#testOpenDocument, com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithCompany, com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithIndividual, com.belmonttech.app.activities.BTLoginActivityTest#openDocumentTest, com.belmonttech.app.activities.BTLoginActivityTest#testBadInput, com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateEduAccount, com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateFreeAccount, com.belmonttech.app.activities.BTNewSignupLoginTest#eduAccountSignupTest, com.belmonttech.app.activities.BTNewSignupLoginTest#freeAccountSignupTest, com.belmonttech.app.activities.GestureTutorialTest#gestureTutorialTest, com.belmonttech.app.activities.ReadOnlyFeatureDialogTest#readOnlyFeatureDialogTest, com.belmonttech.app.adapters.BTContactsAdapterTest#testCustomContact, com.belmonttech.app.adapters.BTContactsAdapterTest#testEmptyCursor, com.belmonttech.app.adapters.BTContactsAdapterTest#testExistingContactsAreFiltered, com.belmonttech.app.adapters.BTContactsAdapterTest#testGetSelectedContacts, com.belmonttech.app.adapters.BTContactsAdapterTest#testSaveAndRestoreState, com.belmonttech.app.adapters.BTContactsAdapterTest#testSearchQuery, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull,
Malinskiy commented 2 years ago

Hmmm, that’s a very interesting behaviour. The problem is not with the execution side of marathon, it’s with the actual calculation of the tests to execute. The logic here is basically 3 steps:

  1. Parse all tests from the apk
  2. Apply filtering configuration (can’t add duplicates)
  3. Apply sharding configuration (may add duplicates, but you don’t have any sharding configuration, so not affecting the execution)

The only reasonable explanation I can see here is that you have multiple tests with the same fully-qualified-test-names. Can you please verify this hypothesis?

By the way, what kind of test parser are you using? Default?

umesh-onshape commented 2 years ago

No, that's not the case I have them specified only once. I haven't specified any parser so I am guessing it's default one. I am pasting the initial log of marathon before test run and the config file. Can you tell me more about how to use the solution in #644 ?

2022-06-16 00:04:56 D 00:04:56.244 [main @coroutine#1] <AdamDeviceProvider> Android Debug Bridge /127.0.0.1:5037: version 41
2022-06-16 00:04:56 D 00:04:56.302 [DeviceMonitor @coroutine#6] <AdamDeviceProvider> Device emulator-5554 changed state to CONNECTED
2022-06-16 00:04:56 D 00:04:56.342 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#12] <AndroidDevice> Device 127.0.0.1:5037:emulator-5554 booted!
2022-06-16 00:04:56 I 00:04:56.463 [main @coroutine#1] <com.malinskiy.marathon.Marathon> Scheduling 220 tests
2022-06-16 00:04:56 D 00:04:56.464 [main @coroutine#1] <com.malinskiy.marathon.Marathon> com.belmonttech.app.activities.BTDocumentListActivityTest#testCapability, com.belmonttech.app.activities.BTDocumentListActivityTest#testDeleteAndRestore, com.belmonttech.app.activities.BTDocumentListActivityTest#testOpenDocument, com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithCompany, com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithIndividual, com.belmonttech.app.activities.BTLoginActivityTest#openDocumentTest, com.belmonttech.app.activities.BTLoginActivityTest#testBadInput, com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateEduAccount, com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateFreeAccount, com.belmonttech.app.activities.BTNewSignupLoginTest#eduAccountSignupTest, com.belmonttech.app.activities.BTNewSignupLoginTest#freeAccountSignupTest, com.belmonttech.app.activities.GestureTutorialTest#gestureTutorialTest, com.belmonttech.app.activities.ReadOnlyFeatureDialogTest#readOnlyFeatureDialogTest, com.belmonttech.app.adapters.BTContactsAdapterTest#testCustomContact, com.belmonttech.app.adapters.BTContactsAdapterTest#testEmptyCursor, com.belmonttech.app.adapters.BTContactsAdapterTest#testExistingContactsAreFiltered, com.belmonttech.app.adapters.BTContactsAdapterTest#testGetSelectedContacts, com.belmonttech.app.adapters.BTContactsAdapterTest#testSaveAndRestoreState, com.belmonttech.app.adapters.BTContactsAdapterTest#testSearchQuery, **com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionNoAccess, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionNoAccess, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionRead, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionRead, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionWrite, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionWrite,** com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testEOAF, com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testGetSelectedOrganizations, com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testSaveAndRestoreState, com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testASFSOF, com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testASFSOF, com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testAllowedSelectionForFilter, com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testAllowedSelectionForFilter, com.belmonttech.gesture.BTChainedGestureListenerTest#testClaimedGesture, com.belmonttech.gesture.BTGestureDetectorTest#testDoubleTap, com.belmonttech.gesture.BTGestureDetectorTest#testDrag, com.belmonttech.gesture.BTGestureDetectorTest#testLongPress, com.belmonttech.gesture.BTGestureDetectorTest#testLongPressThenScroll, com.belmonttech.gesture.BTGestureDetectorTest#testSeparateDoubleTap, com.belmonttech.gesture.BTGestureDetectorTest#testTap, com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerDelayedTap, com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerDrag, com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerTap, com.belmonttech.integration.AssemblyBrokenTest#testBrokenSubassembly, com.belmonttech.integration.AssemblyBrokenTest#testDeletedSubassembly, com.belmonttech.integration.AssemblyContextMenuTest#testACMTV, com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuDelete, com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuFix, com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuSuppress, com.belmonttech.integration.AssemblyContextMenuTest#testItemsWithPartInstanceSelected, com.belmonttech.integration.AssemblyFeatureTest#testMateFeature, com.belmonttech.integration.AssemblyFeatureTest#testMateGroup, com.belmonttech.integration.AssemblyFeatureTest#testMateLimit, com.belmonttech.integration.AssemblyFeatureTest#testMateRelation, com.belmonttech.integration.AssemblyFeatureTest#testNewMateFeatures, com.belmonttech.integration.AssemblyFeatureTest#testNewMateRelations, com.belmonttech.integration.AssemblyFeatureTest#testPreselectMates, com.belmonttech.integration.AssemblyImportTest#testCancelAssemblyImport, com.belmonttech.integration.AssemblyImportTest#testCommitAssemblyImport, com.belmonttech.integration.AssemblyRotationTest#testAssemblyQueryAfterRotation, com.belmonttech.integration.AssemblyTreeTest#testAssemblyDelete, com.belmonttech.integration.AssemblyTreeTest#testAssemblyFix, com.belmonttech.integration.AssemblyTreeTest#testAssemblyMove, com.belmonttech.integration.AssemblyTreeTest#testAssemblyRename, com.belmonttech.integration.AssemblyTreeTest#testAssemblySuppress, com.belmonttech.integration.AssemblyTreeTest#testAssemblyTree, com.belmonttech.integration.AssemblyTreeTest#testAssemblyVisibility, com.belmonttech.integration.AssemblyTreeTest#testFeatureEditorBlocksAssemblyModification, com.belmonttech.integration.AssemblyTreeTest#testToSwitchTab, com.belmonttech.integration.AssemblyUndoTest#testAssemblyUndoRedo, com.belmonttech.integration.CommentTest#testAssignComments, com.belmonttech.integration.CommentTest#testCommentNotification, com.belmonttech.integration.CommentTest#testCreateComment, com.belmonttech.integration.CommentTest#testCreateReply, com.belmonttech.integration.CommentTest#testOpenComments, com.belmonttech.integration.CommentTest#testResolveComments, com.belmonttech.integration.CommentTest#testSoftKeyboardCloseOnExit, com.belmonttech.integration.CustomFeatureImporterTest#testOpenCustomFeaturesImporter, com.belmonttech.integration.DerivedTest#testImportParts, com.belmonttech.integration.DerivedTest#testRestrictToElement, com.belmonttech.integration.DerivedTest#testSearchFiltering, com.belmonttech.integration.DocumentIncompatibilityTest#testUnfathomableDocument, com.belmonttech.integration.DocumentTest#testBackButton, com.belmonttech.integration.DocumentTest#testChangeLengthUnit, com.belmonttech.integration.DocumentTest#testChangeMassUnit, com.belmonttech.integration.DocumentTest#testCopyWorkspace, com.belmonttech.integration.DocumentTest#testModelingAfterReturningFromHelp, com.belmonttech.integration.DocumentTest#testRenameDocument, com.belmonttech.integration.ElementSwitchTest#testSwitchElementViewLock, com.belmonttech.integration.ElementSwitchTest#testSwitchElementWhileEditing, com.belmonttech.integration.ExportTest#testStoreInATab, com.belmonttech.integration.ExportTest#testVerifyExportOptionExist, com.belmonttech.integration.FeatureListTest#testDeleteFeature, com.belmonttech.integration.FeatureListTest#testDeletePart, com.belmonttech.integration.FeatureListTest#testEditPartColor, com.belmonttech.integration.FeatureListTest#testHelixIsACurve, com.belmonttech.integration.FeatureListTest#testRenameDefaultGeometry, com.belmonttech.integration.FeatureListTest#testRenameFeature, com.belmonttech.integration.FeatureListTest#testRenamePart, com.belmonttech.integration.FeatureListTest#testRollback, com.belmonttech.integration.FeatureListTest#testSuppress, com.belmonttech.integration.FeatureListTest#testVisibilityForFeature, com.belmonttech.integration.FeatureListTest#testVisibilityForPart, com.belmonttech.integration.FeatureParameterTest#testBooleanParameter, com.belmonttech.integration.FeatureParameterTest#testDeadFeature, com.belmonttech.integration.FeatureParameterTest#testEnumParameter, com.belmonttech.integration.FeatureParameterTest#testFinal, com.belmonttech.integration.FeatureParameterTest#testGraphicsSelectionFilter, com.belmonttech.integration.FeatureParameterTest#testKeypadAfterParameterChanged, com.belmonttech.integration.FeatureParameterTest#testPSMateConnectorQuery, com.belmonttech.integration.FeatureParameterTest#testQuantityParameter, com.belmonttech.integration.FeatureParameterTest#testQueryListMultipleLists, com.belmonttech.integration.FeatureParameterTest#testQueryListPreselection, com.belmonttech.integration.FeatureParameterTest#testQueryListRemove, com.belmonttech.integration.FeatureParameterTest#testQueryListSelectionFromList, com.belmonttech.integration.FeatureScriptTest#testLocalCustomFeature, com.belmonttech.integration.KeypadTest#testBasicExpression, com.belmonttech.integration.KeypadTest#testInitialDecimalPoint, com.belmonttech.integration.KeypadTest#testInvalidExpression, com.belmonttech.integration.KeypadTest#testInvalidNegativeDimension, com.belmonttech.integration.KeypadTest#testNegativeDimension, com.belmonttech.integration.LoftTest#loftTest, com.belmonttech.integration.MaterialAndMassPropertiesTest#testMassProperties, com.belmonttech.integration.MaterialAndMassPropertiesTest#testMaterialsDialog, com.belmonttech.integration.PartStudioContextMenuTest#testCreatePart, com.belmonttech.integration.PartStudioContextMenuTest#testHidePartsWhenUsingDeletePartTool, com.belmonttech.integration.PartStudioContextMenuTest#testIISMWES, com.belmonttech.integration.PartStudioContextMenuTest#testIISMWPS, com.belmonttech.integration.PartStudioContextMenuTest#testItemsWithPartEntitySelected, com.belmonttech.integration.PartStudioContextMenuTest#testItemsWithSurfaceEdgeSelected, com.belmonttech.integration.PartStudioContextMenuTest#testShowAll, com.belmonttech.integration.PartStudioContextMenuTest#testViewNormalToSketchFeature, com.belmonttech.integration.PartStudioUndoTest#testKeyboardIsClosedAfterUndo, com.belmonttech.integration.PartStudioUndoTest#testSketchToolDisabledAfterUndo, com.belmonttech.integration.PartStudioUndoTest#testUndoAfterFillet, com.belmonttech.integration.PartStudioUndoTest#testUndoIsEnabledAfterCreatingSketch, com.belmonttech.integration.PartStudioUndoTest#testUndoStatusRetainedAcrossRotation, com.belmonttech.integration.PartStudioUndoTest#testVisibilityAfterUndo, com.belmonttech.integration.PrecisionSelectorTest#testSimplePrecisionSelection, com.belmonttech.integration.RectangleAndArcInferencingTest#testRectangleAndArcInferencing, com.belmonttech.integration.RotationTest#testActiveQueryAfterRotation, com.belmonttech.integration.RotationTest#testRotationBasic, com.belmonttech.integration.RotationTest#testRotationDuringSketching, com.belmonttech.integration.RotationTest#testSlidingPanelCollapsedState, com.belmonttech.integration.RotationTest#testSlidingPanelReorderState, com.belmonttech.integration.RotationTest#testToolbarItemsAfterRotation, com.belmonttech.integration.SketchActionsTest#testDeleteEntity, com.belmonttech.integration.SketchActionsTest#testExtrudeFromSketch, com.belmonttech.integration.SketchActionsTest#testRemoveDimension, com.belmonttech.integration.SketchActionsTest#testSketchPlaneQuery, com.belmonttech.integration.SketchActionsTest#testSuppressInferencing, com.belmonttech.integration.SketchActionsTest#testUndoRedo, com.belmonttech.integration.SketchCreatorTest#testClosedSpline, com.belmonttech.integration.SketchCreatorTest#testDragArc, com.belmonttech.integration.SketchCreatorTest#testDragCenterRectangle, com.belmonttech.integration.SketchCreatorTest#testDragCenterpointArc, com.belmonttech.integration.SketchCreatorTest#testDragCircle, com.belmonttech.integration.SketchCreatorTest#testDragCircumscribedPolygon, com.belmonttech.integration.SketchCreatorTest#testDragInscribedPolygon, com.belmonttech.integration.SketchCreatorTest#testDragLine, com.belmonttech.integration.SketchCreatorTest#testDragPerimeterCircle, com.belmonttech.integration.SketchCreatorTest#testDragPoint, com.belmonttech.integration.SketchCreatorTest#testDragRectangle, com.belmonttech.integration.SketchCreatorTest#testDragSpline, com.belmonttech.integration.SketchCreatorTest#testOriginInferencing, com.belmonttech.integration.SketchCreatorTest#testSketchPerpendicularError, com.belmonttech.integration.SketchCreatorTest#testTapArc, com.belmonttech.integration.SketchCreatorTest#testTapCenterRectangle, com.belmonttech.integration.SketchCreatorTest#testTapCenterpointArc, com.belmonttech.integration.SketchCreatorTest#testTapCircle, com.belmonttech.integration.SketchCreatorTest#testTapCircumscribedPolygon, com.belmonttech.integration.SketchCreatorTest#testTapCreateText, com.belmonttech.integration.SketchCreatorTest#testTapInscribedPolygon, com.belmonttech.integration.SketchCreatorTest#testTapLine, com.belmonttech.integration.SketchCreatorTest#testTapLineQuick, com.belmonttech.integration.SketchCreatorTest#testTapPattern, com.belmonttech.integration.SketchCreatorTest#testTapPerimeterCircle, com.belmonttech.integration.SketchCreatorTest#testTapPoint, com.belmonttech.integration.SketchCreatorTest#testTapRectangle, com.belmonttech.integration.SketchCreatorTest#testTapSpline, com.belmonttech.integration.SketchManipulateTest#testAddSplinePoint, com.belmonttech.integration.SketchManipulateTest#testAutoViewLock, com.belmonttech.integration.SketchManipulateTest#testConstraint, com.belmonttech.integration.SketchManipulateTest#testConstruction, com.belmonttech.integration.SketchManipulateTest#testDimension, com.belmonttech.integration.SketchManipulateTest#testDimensionCircle, com.belmonttech.integration.SketchManipulateTest#testExplicitViewLock, com.belmonttech.integration.SketchManipulateTest#testExtend, com.belmonttech.integration.SketchManipulateTest#testFillet, com.belmonttech.integration.SketchManipulateTest#testMirror, com.belmonttech.integration.SketchManipulateTest#testOffsetEdge, com.belmonttech.integration.SketchManipulateTest#testOffsetFace, com.belmonttech.integration.SketchManipulateTest#testPierceConstraint, com.belmonttech.integration.SketchManipulateTest#testProjectBody, com.belmonttech.integration.SketchManipulateTest#testProjectFace, com.belmonttech.integration.SketchManipulateTest#testProjectSilhouette, com.belmonttech.integration.SketchManipulateTest#testProjectWithConstruction, com.belmonttech.integration.SketchManipulateTest#testSlot, com.belmonttech.integration.SketchManipulateTest#testSlotDimension, com.belmonttech.integration.SketchManipulateTest#testSplitEntity, com.belmonttech.integration.SketchManipulateTest#testTangentArc, com.belmonttech.integration.SketchManipulateTest#testTangentArcDrag, com.belmonttech.integration.SketchManipulateTest#testTrim, com.belmonttech.integration.SketchManipulateTest#testTrimMidSection, com.belmonttech.integration.smoketests.AdvancedSearchTest#testPartResultSearch, com.belmonttech.integration.smoketests.AdvancedSearchTest#testQFDTSE, com.belmonttech.integration.smoketests.AdvancedSearchTest#testRQFSETD

the config file:

name: "Onshape"
outputDir: "build/reports/marathon"
vendorConfiguration:
  type: "Android"
  installOptions: "-r -t"
  autoGrantPermission: true
  androidSdk: "$HOME/build_tools/android-sdk-31"
  applicationApk: "./build/outputs/apk/debug/Onshape-debug.apk"
  testApplicationApk: "./build/outputs/apk/androidTest/debug/Onshape-debug-androidTest.apk"
batchingStrategy:
  type: "isolate"
flakinessStrategy:
  type: "ignore"
filteringConfiguration:
  allowlist:
    - type: "fully-qualified-test-name"
      values:
        - "com.belmonttech.app.activities.BTDocumentListActivityTest#testCapability"
        - "com.belmonttech.app.activities.BTDocumentListActivityTest#testDeleteAndRestore"
        - "com.belmonttech.app.activities.BTDocumentListActivityTest#testOpenDocument"
        - "com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithCompany"
        - "com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithIndividual"
        - "com.belmonttech.app.activities.BTLoginActivityTest#testBadInput"
        - "com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateEduAccount"
        - "com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateFreeAccount"
        - "com.belmonttech.app.activities.BTLoginActivityTest#openDocumentTest"
        - "com.belmonttech.app.activities.BTNewSignupLoginTest#eduAccountSignupTest"
        - "com.belmonttech.app.activities.BTNewSignupLoginTest#freeAccountSignupTest"
        - "com.belmonttech.app.activities.GestureTutorialTest#gestureTutorialTest"
        - "com.belmonttech.app.activities.ReadOnlyFeatureDialogTest#readOnlyFeatureDialogTest"
        - "com.belmonttech.app.adapters.BTContactsAdapterTest#testSearchQuery"
        - "com.belmonttech.app.adapters.BTContactsAdapterTest#testCustomContact"
        - "com.belmonttech.app.adapters.BTContactsAdapterTest#testEmptyCursor"
        - "com.belmonttech.app.adapters.BTContactsAdapterTest#testGetSelectedContacts"
        - "com.belmonttech.app.adapters.BTContactsAdapterTest#testSaveAndRestoreState"
        - "com.belmonttech.app.adapters.BTContactsAdapterTest#testExistingContactsAreFiltered"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionNoAccess"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionOwner"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionRead"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare"
        - "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionWrite"
        #- "com.belmonttech.app.rest.data.HTTPUnitTests#testUserAgent"
        - "com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testEOAF"
        - "com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testGetSelectedOrganizations"
        - "com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testSaveAndRestoreState"
        - "com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testAllowedSelectionForFilter"
        - "com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testASFSOF"
        - "com.belmonttech.gesture.BTChainedGestureListenerTest#testClaimedGesture"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testTap"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testDrag"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testLongPressThenScroll"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testSeparateDoubleTap"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerDrag"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testDoubleTap"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerDelayedTap"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testLongPress"
        - "com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerTap"
        - "com.belmonttech.integration.AssemblyBrokenTest#testDeletedSubassembly"
        - "com.belmonttech.integration.AssemblyBrokenTest#testBrokenSubassembly"
        - "com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuDelete"
        - "com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuFix"
        - "com.belmonttech.integration.AssemblyContextMenuTest#testACMTV"
        - "com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuSuppress"
        - "com.belmonttech.integration.AssemblyContextMenuTest#testItemsWithPartInstanceSelected"
        - "com.belmonttech.integration.AssemblyFeatureTest#testPreselectMates"
        - "com.belmonttech.integration.AssemblyFeatureTest#testMateRelation"
        - "com.belmonttech.integration.AssemblyFeatureTest#testNewMateRelations"
        - "com.belmonttech.integration.AssemblyFeatureTest#testMateFeature"
        - "com.belmonttech.integration.AssemblyFeatureTest#testMateGroup"
        - "com.belmonttech.integration.AssemblyFeatureTest#testMateLimit"
        - "com.belmonttech.integration.AssemblyFeatureTest#testNewMateFeatures"
        - "com.belmonttech.integration.AssemblyImportTest#testCancelAssemblyImport"
        - "com.belmonttech.integration.AssemblyImportTest#testCommitAssemblyImport"
        - "com.belmonttech.integration.AssemblyRotationTest#testAssemblyQueryAfterRotation"
        - "com.belmonttech.integration.AssemblyTreeTest#testAssemblyMove"
        - "com.belmonttech.integration.AssemblyTreeTest#testAssemblyTree"
        - "com.belmonttech.integration.AssemblyTreeTest#testAssemblyDelete"
        - "com.belmonttech.integration.AssemblyTreeTest#testAssemblyFix"
        - "com.belmonttech.integration.AssemblyTreeTest#testAssemblyRename"
        - "com.belmonttech.integration.AssemblyTreeTest#testToSwitchTab"
        - "com.belmonttech.integration.AssemblyTreeTest#testAssemblyVisibility"
        - "com.belmonttech.integration.AssemblyTreeTest#testAssemblySuppress"
        - "com.belmonttech.integration.AssemblyTreeTest#testFeatureEditorBlocksAssemblyModification"
        - "com.belmonttech.integration.AssemblyUndoTest#testAssemblyUndoRedo"
        - "com.belmonttech.integration.CommentTest#testCommentNotification"
        - "com.belmonttech.integration.CommentTest#testOpenComments"
        - "com.belmonttech.integration.CommentTest#testCreateReply"
        - "com.belmonttech.integration.CommentTest#testAssignComments"
        - "com.belmonttech.integration.CommentTest#testSoftKeyboardCloseOnExit"
        - "com.belmonttech.integration.CommentTest#testCreateComment"
        - "com.belmonttech.integration.CommentTest#testResolveComments"
        - "com.belmonttech.integration.CustomFeatureImporterTest#testOpenCustomFeaturesImporter"
        - "com.belmonttech.integration.DerivedTest#testImportParts"
        - "com.belmonttech.integration.DerivedTest#testSearchFiltering"
        - "com.belmonttech.integration.DerivedTest#testRestrictToElement"
        - "com.belmonttech.integration.DocumentIncompatibilityTest#testUnfathomableDocument"
        - "com.belmonttech.integration.DocumentTest#testChangeMassUnit"
        - "com.belmonttech.integration.DocumentTest#testRenameDocument"
        - "com.belmonttech.integration.DocumentTest#testChangeLengthUnit"
        - "com.belmonttech.integration.DocumentTest#testBackButton"
        - "com.belmonttech.integration.DocumentTest#testCopyWorkspace"
        - "com.belmonttech.integration.DocumentTest#testModelingAfterReturningFromHelp"
        - "com.belmonttech.integration.ElementSwitchTest#testSwitchElementViewLock"
        - "com.belmonttech.integration.ElementSwitchTest#testSwitchElementWhileEditing"
        - "com.belmonttech.integration.ExportTest#testStoreInATab"
        - "com.belmonttech.integration.ExportTest#testVerifyExportOptionExist"
        - "com.belmonttech.integration.FeatureListTest#testRenameFeature"
        - "com.belmonttech.integration.FeatureListTest#testRenamePart"
        - "com.belmonttech.integration.FeatureListTest#testHelixIsACurve"
        - "com.belmonttech.integration.FeatureListTest#testSuppress"
        - "com.belmonttech.integration.FeatureListTest#testVisibilityForFeature"
        - "com.belmonttech.integration.FeatureListTest#testRenameDefaultGeometry"
        - "com.belmonttech.integration.FeatureListTest#testRollback"
        - "com.belmonttech.integration.FeatureListTest#testVisibilityForPart"
        - "com.belmonttech.integration.FeatureListTest#testDeleteFeature"
        - "com.belmonttech.integration.FeatureListTest#testDeletePart"
        - "com.belmonttech.integration.FeatureListTest#testEditPartColor"
        - "com.belmonttech.integration.FeatureParameterTest#testFinal"
        - "com.belmonttech.integration.FeatureParameterTest#testGraphicsSelectionFilter"
        - "com.belmonttech.integration.FeatureParameterTest#testEnumParameter"
        - "com.belmonttech.integration.FeatureParameterTest#testBooleanParameter"
        - "com.belmonttech.integration.FeatureParameterTest#testKeypadAfterParameterChanged"
        - "com.belmonttech.integration.FeatureParameterTest#testQueryListMultipleLists"
        - "com.belmonttech.integration.FeatureParameterTest#testQueryListRemove"
        - "com.belmonttech.integration.FeatureParameterTest#testDeadFeature"
        - "com.belmonttech.integration.FeatureParameterTest#testQueryListPreselection"
        - "com.belmonttech.integration.FeatureParameterTest#testQueryListSelectionFromList"
        - "com.belmonttech.integration.FeatureParameterTest#testPSMateConnectorQuery"
        - "com.belmonttech.integration.FeatureParameterTest#testQuantityParameter"
        - "com.belmonttech.integration.FeatureScriptTest#testLocalCustomFeature"
        - "com.belmonttech.integration.KeypadTest#testInvalidNegativeDimension"
        - "com.belmonttech.integration.KeypadTest#testInitialDecimalPoint"
        - "com.belmonttech.integration.KeypadTest#testBasicExpression"
        - "com.belmonttech.integration.KeypadTest#testNegativeDimension"
        - "com.belmonttech.integration.KeypadTest#testInvalidExpression"
        - "com.belmonttech.integration.LoftTest#loftTest"
        - "com.belmonttech.integration.MaterialAndMassPropertiesTest#testMaterialsDialog"
        - "com.belmonttech.integration.MaterialAndMassPropertiesTest#testMassProperties"
        - "com.belmonttech.integration.PartStudioContextMenuTest#testHidePartsWhenUsingDeletePartTool"
        - "com.belmonttech.integration.PartStudioContextMenuTest#testIISMWPS"
        - "com.belmonttech.integration.PartStudioContextMenuTest#testItemsWithPartEntitySelected"
        - "com.belmonttech.integration.PartStudioContextMenuTest#testCreatePart"
        - "com.belmonttech.integration.PartStudioContextMenuTest#testItemsWithSurfaceEdgeSelected"
        - "com.belmonttech.integration.PartStudioContextMenuTest#testViewNormalToSketchFeature"
        - "com.belmonttech.integration.PartStudioContextMenuTest#testIISMWES"
        - "com.belmonttech.integration.PartStudioContextMenuTest#testShowAll"
        - "com.belmonttech.integration.PartStudioUndoTest#testUndoStatusRetainedAcrossRotation"
        - "com.belmonttech.integration.PartStudioUndoTest#testKeyboardIsClosedAfterUndo"
        - "com.belmonttech.integration.PartStudioUndoTest#testUndoIsEnabledAfterCreatingSketch"
        - "com.belmonttech.integration.PartStudioUndoTest#testUndoAfterFillet"
        - "com.belmonttech.integration.PartStudioUndoTest#testVisibilityAfterUndo"
        - "com.belmonttech.integration.PartStudioUndoTest#testSketchToolDisabledAfterUndo"
        - "com.belmonttech.integration.PrecisionSelectorTest#testSimplePrecisionSelection"
        - "com.belmonttech.integration.RectangleAndArcInferencingTest#testRectangleAndArcInferencing"
        - "com.belmonttech.integration.RotationTest#testActiveQueryAfterRotation"
        - "com.belmonttech.integration.RotationTest#testSlidingPanelCollapsedState"
        - "com.belmonttech.integration.RotationTest#testToolbarItemsAfterRotation"
        - "com.belmonttech.integration.RotationTest#testSlidingPanelReorderState"
        - "com.belmonttech.integration.RotationTest#testRotationDuringSketching"
        - "com.belmonttech.integration.RotationTest#testRotationBasic"
        - "com.belmonttech.integration.SketchActionsTest#testSketchPlaneQuery"
        - "com.belmonttech.integration.SketchActionsTest#testDeleteEntity"
        - "com.belmonttech.integration.SketchActionsTest#testExtrudeFromSketch"
        - "com.belmonttech.integration.SketchActionsTest#testUndoRedo"
        - "com.belmonttech.integration.SketchActionsTest#testRemoveDimension"
        - "com.belmonttech.integration.SketchActionsTest#testSuppressInferencing"
        - "com.belmonttech.integration.SketchCreatorTest#testDragSpline"
        - "com.belmonttech.integration.SketchCreatorTest#testTapLine"
        - "com.belmonttech.integration.SketchCreatorTest#testTapInscribedPolygon"
        - "com.belmonttech.integration.SketchCreatorTest#testDragCircumscribedPolygon"
        - "com.belmonttech.integration.SketchCreatorTest#testSketchPerpendicularError"
        - "com.belmonttech.integration.SketchCreatorTest#testDragCenterRectangle"
        - "com.belmonttech.integration.SketchCreatorTest#testDragInscribedPolygon"
        - "com.belmonttech.integration.SketchCreatorTest#testTapRectangle"
        - "com.belmonttech.integration.SketchCreatorTest#testTapCenterRectangle"
        - "com.belmonttech.integration.SketchCreatorTest#testClosedSpline"
        - "com.belmonttech.integration.SketchCreatorTest#testDragLine"
        - "com.belmonttech.integration.SketchCreatorTest#testTapPoint"
        - "com.belmonttech.integration.SketchCreatorTest#testTapLineQuick"
        - "com.belmonttech.integration.SketchCreatorTest#testDragPoint"
        - "com.belmonttech.integration.SketchCreatorTest#testTapCircumscribedPolygon"
        - "com.belmonttech.integration.SketchCreatorTest#testTapCenterpointArc"
        - "com.belmonttech.integration.SketchCreatorTest#testTapCircle"
        - "com.belmonttech.integration.SketchCreatorTest#testDragPerimeterCircle"
        - "com.belmonttech.integration.SketchCreatorTest#testDragArc"
        - "com.belmonttech.integration.SketchCreatorTest#testTapPattern"
        - "com.belmonttech.integration.SketchCreatorTest#testTapCreateText"
        - "com.belmonttech.integration.SketchCreatorTest#testDragCircle"
        - "com.belmonttech.integration.SketchCreatorTest#testTapArc"
        - "com.belmonttech.integration.SketchCreatorTest#testTapSpline"
        - "com.belmonttech.integration.SketchCreatorTest#testDragCenterpointArc"
        - "com.belmonttech.integration.SketchCreatorTest#testTapPerimeterCircle"
        - "com.belmonttech.integration.SketchCreatorTest#testDragRectangle"
        - "com.belmonttech.integration.SketchCreatorTest#testOriginInferencing"
        - "com.belmonttech.integration.SketchManipulateTest#testTrimMidSection"
        - "com.belmonttech.integration.SketchManipulateTest#testProjectBody"
        - "com.belmonttech.integration.SketchManipulateTest#testProjectFace"
        - "com.belmonttech.integration.SketchManipulateTest#testOffsetEdge"
        - "com.belmonttech.integration.SketchManipulateTest#testOffsetFace"
        - "com.belmonttech.integration.SketchManipulateTest#testExplicitViewLock"
        - "com.belmonttech.integration.SketchManipulateTest#testSlot"
        - "com.belmonttech.integration.SketchManipulateTest#testTrim"
        - "com.belmonttech.integration.SketchManipulateTest#testPierceConstraint"
        - "com.belmonttech.integration.SketchManipulateTest#testProjectWithConstruction"
        - "com.belmonttech.integration.SketchManipulateTest#testConstruction"
        - "com.belmonttech.integration.SketchManipulateTest#testConstraint"
        - "com.belmonttech.integration.SketchManipulateTest#testDimension"
        - "com.belmonttech.integration.SketchManipulateTest#testTangentArcDrag"
        - "com.belmonttech.integration.SketchManipulateTest#testDimensionCircle"
        - "com.belmonttech.integration.SketchManipulateTest#testAddSplinePoint"
        - "com.belmonttech.integration.SketchManipulateTest#testSplitEntity"
        - "com.belmonttech.integration.SketchManipulateTest#testSlotDimension"
        - "com.belmonttech.integration.SketchManipulateTest#testExtend"
        - "com.belmonttech.integration.SketchManipulateTest#testFillet"
        - "com.belmonttech.integration.SketchManipulateTest#testTangentArc"
        - "com.belmonttech.integration.SketchManipulateTest#testProjectSilhouette"
        - "com.belmonttech.integration.SketchManipulateTest#testMirror"
        - "com.belmonttech.integration.SketchManipulateTest#testAutoViewLock"
        #- "com.belmonttech.integration.smoketests.AdvancedSearchTest#testAddCriteria"
        - "com.belmonttech.integration.smoketests.AdvancedSearchTest#testRQFSETD"
        - "com.belmonttech.integration.smoketests.AdvancedSearchTest#testPartResultSearch"
        - "com.belmonttech.integration.smoketests.AdvancedSearchTest#testQFDTSE"
        - "com.belmonttech.integration.smoketests.SteeringWheel#testSteeringWheel"
umesh-onshape commented 2 years ago

Coming back to your main concern that you would still want to understand why there are more retries invoked here. I am more than interested to help.

As you say:

The logic here is basically 3 steps:

Parse all tests from the apk Apply filtering configuration (can’t add duplicates) Apply sharding configuration (may add duplicates, but you don’t have any sharding configuration, so not affecting the execution)

There could be a bug in calculation (in context of my marathon file or could be a general cause) ? Do you have automated tests that can tell this off-hand?

umesh-onshape commented 2 years ago

Or the 12 "extra" test cases did not report success or failure....I see two of these statements, one in highlighted mode, may or may not be related.

2022-06-16 01:04:03 I 01:04:03.326 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#3781] <DebugTestRunListener> testRunStarted 127.0.0.1:5037:emulator-5554
2022-06-16 01:04:03 I 01:04:03.326 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#3781] <DebugTestRunListener> testStarted 127.0.0.1:5037:emulator-5554 test = TestIdentifier(className=com.belmonttech.app.rest.data.BTDocumentDescriptorTest, testName=testPermissionReshare)
2022-06-16 01:04:03 84% | [omni]-[127.0.0.1:5037:emulator-5554] com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare started
2022-06-16 01:04:03 D 01:04:03.458 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#3785] <AndroidDevice> screenrecord output:
### **2022-06-16 01:04:03  ShellCommandResult(output=FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x7fbbbd532e18)**
2022-06-16 01:04:03 , exitCode=235)
2022-06-16 01:04:03 D 01:04:03.458 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#3785] <ScreenRecorder> Recording finished in 132ms /sdcard/com.belmonttech.app.rest.data.BTDocumentDescriptorTest-testPermissionReshare-7667b4ca-9558-4d68-a735-c1d93ce3344c.mp4
2022-06-16 01:04:04 I 01:04:04.084 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#3781] <DebugTestRunListener> testEnded 127.0.0.1:5037:emulator-5554 test = TestIdentifier(className=com.belmonttech.app.rest.data.BTDocumentDescriptorTest, testName=testPermissionReshare)
2022-06-16 01:04:04 85% | [omni]-[127.0.0.1:5037:emulator-5554] com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare ended
2022-06-16 01:04:04 I 01:04:04.084 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#3781] <DebugTestRunListener> testRunEnded elapsedTime 5
2022-06-16 01:04:04 D 01:04:04.084 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#3781] <AndroidDeviceTestRunner> Execution finished
Malinskiy commented 2 years ago

testEnded means test successfully finished in the am instrument terminology, so I don't think it's a problem with the execution since the execute flow follows the logic of scheduling 220 tests (not expecting duplicates here at all), execute all, only 208 unique results received, fail with mismatch.

Technically you don't have retries created for tests anywhere: the retries are coming from the parsing stage as a duplicate of a test case here. To investigate further I need you to:

  1. Try out the remote test parser. It's a bit involved to set it up but I want to understand if it gives you the same result to exclude failure of the dex-test-parser

  2. If possible, create a reproducer sample with just one test in the source that marathon sees as multiple tests. If you want to collaborate in a private chat - feel free to ping me in slack, link on the main README of the project. I may be able to just use your apks to figure out the problem much faster this way

Malinskiy commented 2 years ago

Problems with remote parser for this use-case were solved in #666 To continue investigation with dex-test-parser either a demo reproducer is needed or the actual apks with the problem.

Malinskiy commented 1 year ago

Closing due to inactivity. Please reopen with a demo reproducer.