Closed umesh-onshape closed 1 year ago
Hey @umesh-onshape
So where do you get the other 12 test runs from? You should be able to understand why from the standard output.
In terms of solution - pretty sure #644 should solve this regardless of the source of retries, but I still want to understand why there are more retries invoked here
I see some test cases run twice, even with no retries asked for in the marathon configuration file as shown above.
@umesh-onshape can you please investigate why the retries are happening for at least one test? E.g. when they’re starting, what’s the result, etc. Maybe they’re mentioned twice in the list requested
Consider the case of testPermissionFull, testCanDeleteAsOwner and testPermissionCopy , these are present two times in the overall test list before marathon start executing tests, though it is present only once in the app/MarathonFile
Also note right at the beginning, log shows marathon will execute 220 test cases, because some of them are duplicated. Again these occur only once in the config file. Let me know what else I should look out for.
2022-06-16 00:04:56 I 00:04:55.742 [main @coroutine#1] <com.malinskiy.marathon.Marathon> java.vm.version: 11.0.11+9-LTS
2022-06-16 00:04:56 I 00:04:55.742 [main @coroutine#1] <com.malinskiy.marathon.Marathon> java.class.version: 55.0
2022-06-16 00:04:56 D 00:04:56.244 [main @coroutine#1] <AdamDeviceProvider> Android Debug Bridge /127.0.0.1:5037: version 41
2022-06-16 00:04:56 D 00:04:56.302 [DeviceMonitor @coroutine#6] <AdamDeviceProvider> Device emulator-5554 changed state to CONNECTED
2022-06-16 00:04:56 D 00:04:56.342 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#12] <AndroidDevice> Device 127.0.0.1:5037:emulator-5554 booted!
**2022-06-16 00:04:56 I 00:04:56.463 [main @coroutine#1] <com.malinskiy.marathon.Marathon> Scheduling 220 tests**
2022-06-16 00:04:56 D 00:04:56.464 [main @coroutine#1] <com.malinskiy.marathon.Marathon> com.belmonttech.app.activities.BTDocumentListActivityTest#testCapability, com.belmonttech.app.activities.BTDocumentListActivityTest#testDeleteAndRestore, com.belmonttech.app.activities.BTDocumentListActivityTest#testOpenDocument, com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithCompany, com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithIndividual, com.belmonttech.app.activities.BTLoginActivityTest#openDocumentTest, com.belmonttech.app.activities.BTLoginActivityTest#testBadInput, com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateEduAccount, com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateFreeAccount, com.belmonttech.app.activities.BTNewSignupLoginTest#eduAccountSignupTest, com.belmonttech.app.activities.BTNewSignupLoginTest#freeAccountSignupTest, com.belmonttech.app.activities.GestureTutorialTest#gestureTutorialTest, com.belmonttech.app.activities.ReadOnlyFeatureDialogTest#readOnlyFeatureDialogTest, com.belmonttech.app.adapters.BTContactsAdapterTest#testCustomContact, com.belmonttech.app.adapters.BTContactsAdapterTest#testEmptyCursor, com.belmonttech.app.adapters.BTContactsAdapterTest#testExistingContactsAreFiltered, com.belmonttech.app.adapters.BTContactsAdapterTest#testGetSelectedContacts, com.belmonttech.app.adapters.BTContactsAdapterTest#testSaveAndRestoreState, com.belmonttech.app.adapters.BTContactsAdapterTest#testSearchQuery, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull,
Hmmm, that’s a very interesting behaviour. The problem is not with the execution side of marathon, it’s with the actual calculation of the tests to execute. The logic here is basically 3 steps:
The only reasonable explanation I can see here is that you have multiple tests with the same fully-qualified-test-names. Can you please verify this hypothesis?
By the way, what kind of test parser are you using? Default?
No, that's not the case I have them specified only once. I haven't specified any parser so I am guessing it's default one. I am pasting the initial log of marathon before test run and the config file. Can you tell me more about how to use the solution in #644 ?
2022-06-16 00:04:56 D 00:04:56.244 [main @coroutine#1] <AdamDeviceProvider> Android Debug Bridge /127.0.0.1:5037: version 41
2022-06-16 00:04:56 D 00:04:56.302 [DeviceMonitor @coroutine#6] <AdamDeviceProvider> Device emulator-5554 changed state to CONNECTED
2022-06-16 00:04:56 D 00:04:56.342 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#12] <AndroidDevice> Device 127.0.0.1:5037:emulator-5554 booted!
2022-06-16 00:04:56 I 00:04:56.463 [main @coroutine#1] <com.malinskiy.marathon.Marathon> Scheduling 220 tests
2022-06-16 00:04:56 D 00:04:56.464 [main @coroutine#1] <com.malinskiy.marathon.Marathon> com.belmonttech.app.activities.BTDocumentListActivityTest#testCapability, com.belmonttech.app.activities.BTDocumentListActivityTest#testDeleteAndRestore, com.belmonttech.app.activities.BTDocumentListActivityTest#testOpenDocument, com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithCompany, com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithIndividual, com.belmonttech.app.activities.BTLoginActivityTest#openDocumentTest, com.belmonttech.app.activities.BTLoginActivityTest#testBadInput, com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateEduAccount, com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateFreeAccount, com.belmonttech.app.activities.BTNewSignupLoginTest#eduAccountSignupTest, com.belmonttech.app.activities.BTNewSignupLoginTest#freeAccountSignupTest, com.belmonttech.app.activities.GestureTutorialTest#gestureTutorialTest, com.belmonttech.app.activities.ReadOnlyFeatureDialogTest#readOnlyFeatureDialogTest, com.belmonttech.app.adapters.BTContactsAdapterTest#testCustomContact, com.belmonttech.app.adapters.BTContactsAdapterTest#testEmptyCursor, com.belmonttech.app.adapters.BTContactsAdapterTest#testExistingContactsAreFiltered, com.belmonttech.app.adapters.BTContactsAdapterTest#testGetSelectedContacts, com.belmonttech.app.adapters.BTContactsAdapterTest#testSaveAndRestoreState, com.belmonttech.app.adapters.BTContactsAdapterTest#testSearchQuery, **com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionNoAccess, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionNoAccess, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionOwner, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionRead, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionRead, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionWrite, com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionWrite,** com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testEOAF, com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testGetSelectedOrganizations, com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testSaveAndRestoreState, com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testASFSOF, com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testASFSOF, com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testAllowedSelectionForFilter, com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testAllowedSelectionForFilter, com.belmonttech.gesture.BTChainedGestureListenerTest#testClaimedGesture, com.belmonttech.gesture.BTGestureDetectorTest#testDoubleTap, com.belmonttech.gesture.BTGestureDetectorTest#testDrag, com.belmonttech.gesture.BTGestureDetectorTest#testLongPress, com.belmonttech.gesture.BTGestureDetectorTest#testLongPressThenScroll, com.belmonttech.gesture.BTGestureDetectorTest#testSeparateDoubleTap, com.belmonttech.gesture.BTGestureDetectorTest#testTap, com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerDelayedTap, com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerDrag, com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerTap, com.belmonttech.integration.AssemblyBrokenTest#testBrokenSubassembly, com.belmonttech.integration.AssemblyBrokenTest#testDeletedSubassembly, com.belmonttech.integration.AssemblyContextMenuTest#testACMTV, com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuDelete, com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuFix, com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuSuppress, com.belmonttech.integration.AssemblyContextMenuTest#testItemsWithPartInstanceSelected, com.belmonttech.integration.AssemblyFeatureTest#testMateFeature, com.belmonttech.integration.AssemblyFeatureTest#testMateGroup, com.belmonttech.integration.AssemblyFeatureTest#testMateLimit, com.belmonttech.integration.AssemblyFeatureTest#testMateRelation, com.belmonttech.integration.AssemblyFeatureTest#testNewMateFeatures, com.belmonttech.integration.AssemblyFeatureTest#testNewMateRelations, com.belmonttech.integration.AssemblyFeatureTest#testPreselectMates, com.belmonttech.integration.AssemblyImportTest#testCancelAssemblyImport, com.belmonttech.integration.AssemblyImportTest#testCommitAssemblyImport, com.belmonttech.integration.AssemblyRotationTest#testAssemblyQueryAfterRotation, com.belmonttech.integration.AssemblyTreeTest#testAssemblyDelete, com.belmonttech.integration.AssemblyTreeTest#testAssemblyFix, com.belmonttech.integration.AssemblyTreeTest#testAssemblyMove, com.belmonttech.integration.AssemblyTreeTest#testAssemblyRename, com.belmonttech.integration.AssemblyTreeTest#testAssemblySuppress, com.belmonttech.integration.AssemblyTreeTest#testAssemblyTree, com.belmonttech.integration.AssemblyTreeTest#testAssemblyVisibility, com.belmonttech.integration.AssemblyTreeTest#testFeatureEditorBlocksAssemblyModification, com.belmonttech.integration.AssemblyTreeTest#testToSwitchTab, com.belmonttech.integration.AssemblyUndoTest#testAssemblyUndoRedo, com.belmonttech.integration.CommentTest#testAssignComments, com.belmonttech.integration.CommentTest#testCommentNotification, com.belmonttech.integration.CommentTest#testCreateComment, com.belmonttech.integration.CommentTest#testCreateReply, com.belmonttech.integration.CommentTest#testOpenComments, com.belmonttech.integration.CommentTest#testResolveComments, com.belmonttech.integration.CommentTest#testSoftKeyboardCloseOnExit, com.belmonttech.integration.CustomFeatureImporterTest#testOpenCustomFeaturesImporter, com.belmonttech.integration.DerivedTest#testImportParts, com.belmonttech.integration.DerivedTest#testRestrictToElement, com.belmonttech.integration.DerivedTest#testSearchFiltering, com.belmonttech.integration.DocumentIncompatibilityTest#testUnfathomableDocument, com.belmonttech.integration.DocumentTest#testBackButton, com.belmonttech.integration.DocumentTest#testChangeLengthUnit, com.belmonttech.integration.DocumentTest#testChangeMassUnit, com.belmonttech.integration.DocumentTest#testCopyWorkspace, com.belmonttech.integration.DocumentTest#testModelingAfterReturningFromHelp, com.belmonttech.integration.DocumentTest#testRenameDocument, com.belmonttech.integration.ElementSwitchTest#testSwitchElementViewLock, com.belmonttech.integration.ElementSwitchTest#testSwitchElementWhileEditing, com.belmonttech.integration.ExportTest#testStoreInATab, com.belmonttech.integration.ExportTest#testVerifyExportOptionExist, com.belmonttech.integration.FeatureListTest#testDeleteFeature, com.belmonttech.integration.FeatureListTest#testDeletePart, com.belmonttech.integration.FeatureListTest#testEditPartColor, com.belmonttech.integration.FeatureListTest#testHelixIsACurve, com.belmonttech.integration.FeatureListTest#testRenameDefaultGeometry, com.belmonttech.integration.FeatureListTest#testRenameFeature, com.belmonttech.integration.FeatureListTest#testRenamePart, com.belmonttech.integration.FeatureListTest#testRollback, com.belmonttech.integration.FeatureListTest#testSuppress, com.belmonttech.integration.FeatureListTest#testVisibilityForFeature, com.belmonttech.integration.FeatureListTest#testVisibilityForPart, com.belmonttech.integration.FeatureParameterTest#testBooleanParameter, com.belmonttech.integration.FeatureParameterTest#testDeadFeature, com.belmonttech.integration.FeatureParameterTest#testEnumParameter, com.belmonttech.integration.FeatureParameterTest#testFinal, com.belmonttech.integration.FeatureParameterTest#testGraphicsSelectionFilter, com.belmonttech.integration.FeatureParameterTest#testKeypadAfterParameterChanged, com.belmonttech.integration.FeatureParameterTest#testPSMateConnectorQuery, com.belmonttech.integration.FeatureParameterTest#testQuantityParameter, com.belmonttech.integration.FeatureParameterTest#testQueryListMultipleLists, com.belmonttech.integration.FeatureParameterTest#testQueryListPreselection, com.belmonttech.integration.FeatureParameterTest#testQueryListRemove, com.belmonttech.integration.FeatureParameterTest#testQueryListSelectionFromList, com.belmonttech.integration.FeatureScriptTest#testLocalCustomFeature, com.belmonttech.integration.KeypadTest#testBasicExpression, com.belmonttech.integration.KeypadTest#testInitialDecimalPoint, com.belmonttech.integration.KeypadTest#testInvalidExpression, com.belmonttech.integration.KeypadTest#testInvalidNegativeDimension, com.belmonttech.integration.KeypadTest#testNegativeDimension, com.belmonttech.integration.LoftTest#loftTest, com.belmonttech.integration.MaterialAndMassPropertiesTest#testMassProperties, com.belmonttech.integration.MaterialAndMassPropertiesTest#testMaterialsDialog, com.belmonttech.integration.PartStudioContextMenuTest#testCreatePart, com.belmonttech.integration.PartStudioContextMenuTest#testHidePartsWhenUsingDeletePartTool, com.belmonttech.integration.PartStudioContextMenuTest#testIISMWES, com.belmonttech.integration.PartStudioContextMenuTest#testIISMWPS, com.belmonttech.integration.PartStudioContextMenuTest#testItemsWithPartEntitySelected, com.belmonttech.integration.PartStudioContextMenuTest#testItemsWithSurfaceEdgeSelected, com.belmonttech.integration.PartStudioContextMenuTest#testShowAll, com.belmonttech.integration.PartStudioContextMenuTest#testViewNormalToSketchFeature, com.belmonttech.integration.PartStudioUndoTest#testKeyboardIsClosedAfterUndo, com.belmonttech.integration.PartStudioUndoTest#testSketchToolDisabledAfterUndo, com.belmonttech.integration.PartStudioUndoTest#testUndoAfterFillet, com.belmonttech.integration.PartStudioUndoTest#testUndoIsEnabledAfterCreatingSketch, com.belmonttech.integration.PartStudioUndoTest#testUndoStatusRetainedAcrossRotation, com.belmonttech.integration.PartStudioUndoTest#testVisibilityAfterUndo, com.belmonttech.integration.PrecisionSelectorTest#testSimplePrecisionSelection, com.belmonttech.integration.RectangleAndArcInferencingTest#testRectangleAndArcInferencing, com.belmonttech.integration.RotationTest#testActiveQueryAfterRotation, com.belmonttech.integration.RotationTest#testRotationBasic, com.belmonttech.integration.RotationTest#testRotationDuringSketching, com.belmonttech.integration.RotationTest#testSlidingPanelCollapsedState, com.belmonttech.integration.RotationTest#testSlidingPanelReorderState, com.belmonttech.integration.RotationTest#testToolbarItemsAfterRotation, com.belmonttech.integration.SketchActionsTest#testDeleteEntity, com.belmonttech.integration.SketchActionsTest#testExtrudeFromSketch, com.belmonttech.integration.SketchActionsTest#testRemoveDimension, com.belmonttech.integration.SketchActionsTest#testSketchPlaneQuery, com.belmonttech.integration.SketchActionsTest#testSuppressInferencing, com.belmonttech.integration.SketchActionsTest#testUndoRedo, com.belmonttech.integration.SketchCreatorTest#testClosedSpline, com.belmonttech.integration.SketchCreatorTest#testDragArc, com.belmonttech.integration.SketchCreatorTest#testDragCenterRectangle, com.belmonttech.integration.SketchCreatorTest#testDragCenterpointArc, com.belmonttech.integration.SketchCreatorTest#testDragCircle, com.belmonttech.integration.SketchCreatorTest#testDragCircumscribedPolygon, com.belmonttech.integration.SketchCreatorTest#testDragInscribedPolygon, com.belmonttech.integration.SketchCreatorTest#testDragLine, com.belmonttech.integration.SketchCreatorTest#testDragPerimeterCircle, com.belmonttech.integration.SketchCreatorTest#testDragPoint, com.belmonttech.integration.SketchCreatorTest#testDragRectangle, com.belmonttech.integration.SketchCreatorTest#testDragSpline, com.belmonttech.integration.SketchCreatorTest#testOriginInferencing, com.belmonttech.integration.SketchCreatorTest#testSketchPerpendicularError, com.belmonttech.integration.SketchCreatorTest#testTapArc, com.belmonttech.integration.SketchCreatorTest#testTapCenterRectangle, com.belmonttech.integration.SketchCreatorTest#testTapCenterpointArc, com.belmonttech.integration.SketchCreatorTest#testTapCircle, com.belmonttech.integration.SketchCreatorTest#testTapCircumscribedPolygon, com.belmonttech.integration.SketchCreatorTest#testTapCreateText, com.belmonttech.integration.SketchCreatorTest#testTapInscribedPolygon, com.belmonttech.integration.SketchCreatorTest#testTapLine, com.belmonttech.integration.SketchCreatorTest#testTapLineQuick, com.belmonttech.integration.SketchCreatorTest#testTapPattern, com.belmonttech.integration.SketchCreatorTest#testTapPerimeterCircle, com.belmonttech.integration.SketchCreatorTest#testTapPoint, com.belmonttech.integration.SketchCreatorTest#testTapRectangle, com.belmonttech.integration.SketchCreatorTest#testTapSpline, com.belmonttech.integration.SketchManipulateTest#testAddSplinePoint, com.belmonttech.integration.SketchManipulateTest#testAutoViewLock, com.belmonttech.integration.SketchManipulateTest#testConstraint, com.belmonttech.integration.SketchManipulateTest#testConstruction, com.belmonttech.integration.SketchManipulateTest#testDimension, com.belmonttech.integration.SketchManipulateTest#testDimensionCircle, com.belmonttech.integration.SketchManipulateTest#testExplicitViewLock, com.belmonttech.integration.SketchManipulateTest#testExtend, com.belmonttech.integration.SketchManipulateTest#testFillet, com.belmonttech.integration.SketchManipulateTest#testMirror, com.belmonttech.integration.SketchManipulateTest#testOffsetEdge, com.belmonttech.integration.SketchManipulateTest#testOffsetFace, com.belmonttech.integration.SketchManipulateTest#testPierceConstraint, com.belmonttech.integration.SketchManipulateTest#testProjectBody, com.belmonttech.integration.SketchManipulateTest#testProjectFace, com.belmonttech.integration.SketchManipulateTest#testProjectSilhouette, com.belmonttech.integration.SketchManipulateTest#testProjectWithConstruction, com.belmonttech.integration.SketchManipulateTest#testSlot, com.belmonttech.integration.SketchManipulateTest#testSlotDimension, com.belmonttech.integration.SketchManipulateTest#testSplitEntity, com.belmonttech.integration.SketchManipulateTest#testTangentArc, com.belmonttech.integration.SketchManipulateTest#testTangentArcDrag, com.belmonttech.integration.SketchManipulateTest#testTrim, com.belmonttech.integration.SketchManipulateTest#testTrimMidSection, com.belmonttech.integration.smoketests.AdvancedSearchTest#testPartResultSearch, com.belmonttech.integration.smoketests.AdvancedSearchTest#testQFDTSE, com.belmonttech.integration.smoketests.AdvancedSearchTest#testRQFSETD
the config file:
name: "Onshape"
outputDir: "build/reports/marathon"
vendorConfiguration:
type: "Android"
installOptions: "-r -t"
autoGrantPermission: true
androidSdk: "$HOME/build_tools/android-sdk-31"
applicationApk: "./build/outputs/apk/debug/Onshape-debug.apk"
testApplicationApk: "./build/outputs/apk/androidTest/debug/Onshape-debug-androidTest.apk"
batchingStrategy:
type: "isolate"
flakinessStrategy:
type: "ignore"
filteringConfiguration:
allowlist:
- type: "fully-qualified-test-name"
values:
- "com.belmonttech.app.activities.BTDocumentListActivityTest#testCapability"
- "com.belmonttech.app.activities.BTDocumentListActivityTest#testDeleteAndRestore"
- "com.belmonttech.app.activities.BTDocumentListActivityTest#testOpenDocument"
- "com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithCompany"
- "com.belmonttech.app.activities.BTDocumentListActivityTest#testShareWithIndividual"
- "com.belmonttech.app.activities.BTLoginActivityTest#testBadInput"
- "com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateEduAccount"
- "com.belmonttech.app.activities.BTLoginActivityTest#testBadInputCreateFreeAccount"
- "com.belmonttech.app.activities.BTLoginActivityTest#openDocumentTest"
- "com.belmonttech.app.activities.BTNewSignupLoginTest#eduAccountSignupTest"
- "com.belmonttech.app.activities.BTNewSignupLoginTest#freeAccountSignupTest"
- "com.belmonttech.app.activities.GestureTutorialTest#gestureTutorialTest"
- "com.belmonttech.app.activities.ReadOnlyFeatureDialogTest#readOnlyFeatureDialogTest"
- "com.belmonttech.app.adapters.BTContactsAdapterTest#testSearchQuery"
- "com.belmonttech.app.adapters.BTContactsAdapterTest#testCustomContact"
- "com.belmonttech.app.adapters.BTContactsAdapterTest#testEmptyCursor"
- "com.belmonttech.app.adapters.BTContactsAdapterTest#testGetSelectedContacts"
- "com.belmonttech.app.adapters.BTContactsAdapterTest#testSaveAndRestoreState"
- "com.belmonttech.app.adapters.BTContactsAdapterTest#testExistingContactsAreFiltered"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testCanDeleteAsOwner"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testGetBestThumbnailUrl"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionComment"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionCopy"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionFull"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionNoAccess"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionOwner"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionRead"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare"
- "com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionWrite"
#- "com.belmonttech.app.rest.data.HTTPUnitTests#testUserAgent"
- "com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testEOAF"
- "com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testGetSelectedOrganizations"
- "com.belmonttech.app.share.BTOrganizationSelectorAdapterTest#testSaveAndRestoreState"
- "com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testAllowedSelectionForFilter"
- "com.belmonttech.app.utils.BTAllowedSelectionUtilsTest#testASFSOF"
- "com.belmonttech.gesture.BTChainedGestureListenerTest#testClaimedGesture"
- "com.belmonttech.gesture.BTGestureDetectorTest#testTap"
- "com.belmonttech.gesture.BTGestureDetectorTest#testDrag"
- "com.belmonttech.gesture.BTGestureDetectorTest#testLongPressThenScroll"
- "com.belmonttech.gesture.BTGestureDetectorTest#testSeparateDoubleTap"
- "com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerDrag"
- "com.belmonttech.gesture.BTGestureDetectorTest#testDoubleTap"
- "com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerDelayedTap"
- "com.belmonttech.gesture.BTGestureDetectorTest#testLongPress"
- "com.belmonttech.gesture.BTGestureDetectorTest#testTwoFingerTap"
- "com.belmonttech.integration.AssemblyBrokenTest#testDeletedSubassembly"
- "com.belmonttech.integration.AssemblyBrokenTest#testBrokenSubassembly"
- "com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuDelete"
- "com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuFix"
- "com.belmonttech.integration.AssemblyContextMenuTest#testACMTV"
- "com.belmonttech.integration.AssemblyContextMenuTest#testAssemblyContextMenuSuppress"
- "com.belmonttech.integration.AssemblyContextMenuTest#testItemsWithPartInstanceSelected"
- "com.belmonttech.integration.AssemblyFeatureTest#testPreselectMates"
- "com.belmonttech.integration.AssemblyFeatureTest#testMateRelation"
- "com.belmonttech.integration.AssemblyFeatureTest#testNewMateRelations"
- "com.belmonttech.integration.AssemblyFeatureTest#testMateFeature"
- "com.belmonttech.integration.AssemblyFeatureTest#testMateGroup"
- "com.belmonttech.integration.AssemblyFeatureTest#testMateLimit"
- "com.belmonttech.integration.AssemblyFeatureTest#testNewMateFeatures"
- "com.belmonttech.integration.AssemblyImportTest#testCancelAssemblyImport"
- "com.belmonttech.integration.AssemblyImportTest#testCommitAssemblyImport"
- "com.belmonttech.integration.AssemblyRotationTest#testAssemblyQueryAfterRotation"
- "com.belmonttech.integration.AssemblyTreeTest#testAssemblyMove"
- "com.belmonttech.integration.AssemblyTreeTest#testAssemblyTree"
- "com.belmonttech.integration.AssemblyTreeTest#testAssemblyDelete"
- "com.belmonttech.integration.AssemblyTreeTest#testAssemblyFix"
- "com.belmonttech.integration.AssemblyTreeTest#testAssemblyRename"
- "com.belmonttech.integration.AssemblyTreeTest#testToSwitchTab"
- "com.belmonttech.integration.AssemblyTreeTest#testAssemblyVisibility"
- "com.belmonttech.integration.AssemblyTreeTest#testAssemblySuppress"
- "com.belmonttech.integration.AssemblyTreeTest#testFeatureEditorBlocksAssemblyModification"
- "com.belmonttech.integration.AssemblyUndoTest#testAssemblyUndoRedo"
- "com.belmonttech.integration.CommentTest#testCommentNotification"
- "com.belmonttech.integration.CommentTest#testOpenComments"
- "com.belmonttech.integration.CommentTest#testCreateReply"
- "com.belmonttech.integration.CommentTest#testAssignComments"
- "com.belmonttech.integration.CommentTest#testSoftKeyboardCloseOnExit"
- "com.belmonttech.integration.CommentTest#testCreateComment"
- "com.belmonttech.integration.CommentTest#testResolveComments"
- "com.belmonttech.integration.CustomFeatureImporterTest#testOpenCustomFeaturesImporter"
- "com.belmonttech.integration.DerivedTest#testImportParts"
- "com.belmonttech.integration.DerivedTest#testSearchFiltering"
- "com.belmonttech.integration.DerivedTest#testRestrictToElement"
- "com.belmonttech.integration.DocumentIncompatibilityTest#testUnfathomableDocument"
- "com.belmonttech.integration.DocumentTest#testChangeMassUnit"
- "com.belmonttech.integration.DocumentTest#testRenameDocument"
- "com.belmonttech.integration.DocumentTest#testChangeLengthUnit"
- "com.belmonttech.integration.DocumentTest#testBackButton"
- "com.belmonttech.integration.DocumentTest#testCopyWorkspace"
- "com.belmonttech.integration.DocumentTest#testModelingAfterReturningFromHelp"
- "com.belmonttech.integration.ElementSwitchTest#testSwitchElementViewLock"
- "com.belmonttech.integration.ElementSwitchTest#testSwitchElementWhileEditing"
- "com.belmonttech.integration.ExportTest#testStoreInATab"
- "com.belmonttech.integration.ExportTest#testVerifyExportOptionExist"
- "com.belmonttech.integration.FeatureListTest#testRenameFeature"
- "com.belmonttech.integration.FeatureListTest#testRenamePart"
- "com.belmonttech.integration.FeatureListTest#testHelixIsACurve"
- "com.belmonttech.integration.FeatureListTest#testSuppress"
- "com.belmonttech.integration.FeatureListTest#testVisibilityForFeature"
- "com.belmonttech.integration.FeatureListTest#testRenameDefaultGeometry"
- "com.belmonttech.integration.FeatureListTest#testRollback"
- "com.belmonttech.integration.FeatureListTest#testVisibilityForPart"
- "com.belmonttech.integration.FeatureListTest#testDeleteFeature"
- "com.belmonttech.integration.FeatureListTest#testDeletePart"
- "com.belmonttech.integration.FeatureListTest#testEditPartColor"
- "com.belmonttech.integration.FeatureParameterTest#testFinal"
- "com.belmonttech.integration.FeatureParameterTest#testGraphicsSelectionFilter"
- "com.belmonttech.integration.FeatureParameterTest#testEnumParameter"
- "com.belmonttech.integration.FeatureParameterTest#testBooleanParameter"
- "com.belmonttech.integration.FeatureParameterTest#testKeypadAfterParameterChanged"
- "com.belmonttech.integration.FeatureParameterTest#testQueryListMultipleLists"
- "com.belmonttech.integration.FeatureParameterTest#testQueryListRemove"
- "com.belmonttech.integration.FeatureParameterTest#testDeadFeature"
- "com.belmonttech.integration.FeatureParameterTest#testQueryListPreselection"
- "com.belmonttech.integration.FeatureParameterTest#testQueryListSelectionFromList"
- "com.belmonttech.integration.FeatureParameterTest#testPSMateConnectorQuery"
- "com.belmonttech.integration.FeatureParameterTest#testQuantityParameter"
- "com.belmonttech.integration.FeatureScriptTest#testLocalCustomFeature"
- "com.belmonttech.integration.KeypadTest#testInvalidNegativeDimension"
- "com.belmonttech.integration.KeypadTest#testInitialDecimalPoint"
- "com.belmonttech.integration.KeypadTest#testBasicExpression"
- "com.belmonttech.integration.KeypadTest#testNegativeDimension"
- "com.belmonttech.integration.KeypadTest#testInvalidExpression"
- "com.belmonttech.integration.LoftTest#loftTest"
- "com.belmonttech.integration.MaterialAndMassPropertiesTest#testMaterialsDialog"
- "com.belmonttech.integration.MaterialAndMassPropertiesTest#testMassProperties"
- "com.belmonttech.integration.PartStudioContextMenuTest#testHidePartsWhenUsingDeletePartTool"
- "com.belmonttech.integration.PartStudioContextMenuTest#testIISMWPS"
- "com.belmonttech.integration.PartStudioContextMenuTest#testItemsWithPartEntitySelected"
- "com.belmonttech.integration.PartStudioContextMenuTest#testCreatePart"
- "com.belmonttech.integration.PartStudioContextMenuTest#testItemsWithSurfaceEdgeSelected"
- "com.belmonttech.integration.PartStudioContextMenuTest#testViewNormalToSketchFeature"
- "com.belmonttech.integration.PartStudioContextMenuTest#testIISMWES"
- "com.belmonttech.integration.PartStudioContextMenuTest#testShowAll"
- "com.belmonttech.integration.PartStudioUndoTest#testUndoStatusRetainedAcrossRotation"
- "com.belmonttech.integration.PartStudioUndoTest#testKeyboardIsClosedAfterUndo"
- "com.belmonttech.integration.PartStudioUndoTest#testUndoIsEnabledAfterCreatingSketch"
- "com.belmonttech.integration.PartStudioUndoTest#testUndoAfterFillet"
- "com.belmonttech.integration.PartStudioUndoTest#testVisibilityAfterUndo"
- "com.belmonttech.integration.PartStudioUndoTest#testSketchToolDisabledAfterUndo"
- "com.belmonttech.integration.PrecisionSelectorTest#testSimplePrecisionSelection"
- "com.belmonttech.integration.RectangleAndArcInferencingTest#testRectangleAndArcInferencing"
- "com.belmonttech.integration.RotationTest#testActiveQueryAfterRotation"
- "com.belmonttech.integration.RotationTest#testSlidingPanelCollapsedState"
- "com.belmonttech.integration.RotationTest#testToolbarItemsAfterRotation"
- "com.belmonttech.integration.RotationTest#testSlidingPanelReorderState"
- "com.belmonttech.integration.RotationTest#testRotationDuringSketching"
- "com.belmonttech.integration.RotationTest#testRotationBasic"
- "com.belmonttech.integration.SketchActionsTest#testSketchPlaneQuery"
- "com.belmonttech.integration.SketchActionsTest#testDeleteEntity"
- "com.belmonttech.integration.SketchActionsTest#testExtrudeFromSketch"
- "com.belmonttech.integration.SketchActionsTest#testUndoRedo"
- "com.belmonttech.integration.SketchActionsTest#testRemoveDimension"
- "com.belmonttech.integration.SketchActionsTest#testSuppressInferencing"
- "com.belmonttech.integration.SketchCreatorTest#testDragSpline"
- "com.belmonttech.integration.SketchCreatorTest#testTapLine"
- "com.belmonttech.integration.SketchCreatorTest#testTapInscribedPolygon"
- "com.belmonttech.integration.SketchCreatorTest#testDragCircumscribedPolygon"
- "com.belmonttech.integration.SketchCreatorTest#testSketchPerpendicularError"
- "com.belmonttech.integration.SketchCreatorTest#testDragCenterRectangle"
- "com.belmonttech.integration.SketchCreatorTest#testDragInscribedPolygon"
- "com.belmonttech.integration.SketchCreatorTest#testTapRectangle"
- "com.belmonttech.integration.SketchCreatorTest#testTapCenterRectangle"
- "com.belmonttech.integration.SketchCreatorTest#testClosedSpline"
- "com.belmonttech.integration.SketchCreatorTest#testDragLine"
- "com.belmonttech.integration.SketchCreatorTest#testTapPoint"
- "com.belmonttech.integration.SketchCreatorTest#testTapLineQuick"
- "com.belmonttech.integration.SketchCreatorTest#testDragPoint"
- "com.belmonttech.integration.SketchCreatorTest#testTapCircumscribedPolygon"
- "com.belmonttech.integration.SketchCreatorTest#testTapCenterpointArc"
- "com.belmonttech.integration.SketchCreatorTest#testTapCircle"
- "com.belmonttech.integration.SketchCreatorTest#testDragPerimeterCircle"
- "com.belmonttech.integration.SketchCreatorTest#testDragArc"
- "com.belmonttech.integration.SketchCreatorTest#testTapPattern"
- "com.belmonttech.integration.SketchCreatorTest#testTapCreateText"
- "com.belmonttech.integration.SketchCreatorTest#testDragCircle"
- "com.belmonttech.integration.SketchCreatorTest#testTapArc"
- "com.belmonttech.integration.SketchCreatorTest#testTapSpline"
- "com.belmonttech.integration.SketchCreatorTest#testDragCenterpointArc"
- "com.belmonttech.integration.SketchCreatorTest#testTapPerimeterCircle"
- "com.belmonttech.integration.SketchCreatorTest#testDragRectangle"
- "com.belmonttech.integration.SketchCreatorTest#testOriginInferencing"
- "com.belmonttech.integration.SketchManipulateTest#testTrimMidSection"
- "com.belmonttech.integration.SketchManipulateTest#testProjectBody"
- "com.belmonttech.integration.SketchManipulateTest#testProjectFace"
- "com.belmonttech.integration.SketchManipulateTest#testOffsetEdge"
- "com.belmonttech.integration.SketchManipulateTest#testOffsetFace"
- "com.belmonttech.integration.SketchManipulateTest#testExplicitViewLock"
- "com.belmonttech.integration.SketchManipulateTest#testSlot"
- "com.belmonttech.integration.SketchManipulateTest#testTrim"
- "com.belmonttech.integration.SketchManipulateTest#testPierceConstraint"
- "com.belmonttech.integration.SketchManipulateTest#testProjectWithConstruction"
- "com.belmonttech.integration.SketchManipulateTest#testConstruction"
- "com.belmonttech.integration.SketchManipulateTest#testConstraint"
- "com.belmonttech.integration.SketchManipulateTest#testDimension"
- "com.belmonttech.integration.SketchManipulateTest#testTangentArcDrag"
- "com.belmonttech.integration.SketchManipulateTest#testDimensionCircle"
- "com.belmonttech.integration.SketchManipulateTest#testAddSplinePoint"
- "com.belmonttech.integration.SketchManipulateTest#testSplitEntity"
- "com.belmonttech.integration.SketchManipulateTest#testSlotDimension"
- "com.belmonttech.integration.SketchManipulateTest#testExtend"
- "com.belmonttech.integration.SketchManipulateTest#testFillet"
- "com.belmonttech.integration.SketchManipulateTest#testTangentArc"
- "com.belmonttech.integration.SketchManipulateTest#testProjectSilhouette"
- "com.belmonttech.integration.SketchManipulateTest#testMirror"
- "com.belmonttech.integration.SketchManipulateTest#testAutoViewLock"
#- "com.belmonttech.integration.smoketests.AdvancedSearchTest#testAddCriteria"
- "com.belmonttech.integration.smoketests.AdvancedSearchTest#testRQFSETD"
- "com.belmonttech.integration.smoketests.AdvancedSearchTest#testPartResultSearch"
- "com.belmonttech.integration.smoketests.AdvancedSearchTest#testQFDTSE"
- "com.belmonttech.integration.smoketests.SteeringWheel#testSteeringWheel"
Coming back to your main concern that you would still want to understand why there are more retries invoked here. I am more than interested to help.
As you say:
The logic here is basically 3 steps:
Parse all tests from the apk Apply filtering configuration (can’t add duplicates) Apply sharding configuration (may add duplicates, but you don’t have any sharding configuration, so not affecting the execution)
There could be a bug in calculation (in context of my marathon file or could be a general cause) ? Do you have automated tests that can tell this off-hand?
Or the 12 "extra" test cases did not report success or failure....I see two of these statements, one in highlighted mode, may or may not be related.
2022-06-16 01:04:03 I 01:04:03.326 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#3781] <DebugTestRunListener> testRunStarted 127.0.0.1:5037:emulator-5554
2022-06-16 01:04:03 I 01:04:03.326 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#3781] <DebugTestRunListener> testStarted 127.0.0.1:5037:emulator-5554 test = TestIdentifier(className=com.belmonttech.app.rest.data.BTDocumentDescriptorTest, testName=testPermissionReshare)
2022-06-16 01:04:03 84% | [omni]-[127.0.0.1:5037:emulator-5554] com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare started
2022-06-16 01:04:03 D 01:04:03.458 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#3785] <AndroidDevice> screenrecord output:
### **2022-06-16 01:04:03 ShellCommandResult(output=FORTIFY: pthread_mutex_lock called on a destroyed mutex (0x7fbbbd532e18)**
2022-06-16 01:04:03 , exitCode=235)
2022-06-16 01:04:03 D 01:04:03.458 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-2 @coroutine#3785] <ScreenRecorder> Recording finished in 132ms /sdcard/com.belmonttech.app.rest.data.BTDocumentDescriptorTest-testPermissionReshare-7667b4ca-9558-4d68-a735-c1d93ce3344c.mp4
2022-06-16 01:04:04 I 01:04:04.084 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#3781] <DebugTestRunListener> testEnded 127.0.0.1:5037:emulator-5554 test = TestIdentifier(className=com.belmonttech.app.rest.data.BTDocumentDescriptorTest, testName=testPermissionReshare)
2022-06-16 01:04:04 85% | [omni]-[127.0.0.1:5037:emulator-5554] com.belmonttech.app.rest.data.BTDocumentDescriptorTest#testPermissionReshare ended
2022-06-16 01:04:04 I 01:04:04.084 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#3781] <DebugTestRunListener> testRunEnded elapsedTime 5
2022-06-16 01:04:04 D 01:04:04.084 [AndroidDevice - execution - 127.0.0.1:5037:emulator-5554-1 @coroutine#3781] <AndroidDeviceTestRunner> Execution finished
testEnded
means test successfully finished in the am instrument
terminology, so I don't think it's a problem with the execution since the execute flow follows the logic of scheduling 220 tests (not expecting duplicates here at all), execute all, only 208 unique results received, fail with mismatch.
Technically you don't have retries created for tests anywhere: the retries are coming from the parsing stage as a duplicate of a test case here. To investigate further I need you to:
Try out the remote test parser. It's a bit involved to set it up but I want to understand if it gives you the same result to exclude failure of the dex-test-parser
If possible, create a reproducer sample with just one test in the source that marathon sees as multiple tests. If you want to collaborate in a private chat - feel free to ping me in slack, link on the main README of the project. I may be able to just use your apks to figure out the problem much faster this way
Problems with remote parser for this use-case were solved in #666 To continue investigation with dex-test-parser either a demo reproducer is needed or the actual apks with the problem.
Closing due to inactivity. Please reopen with a demo reproducer.
Describe the bug I explicitly specified 208 test cases to run using filter with fully qualified test names. All these 208 test cases pass but marathon reports failure as it say s it expected results from 220 test cases.
To Reproduce Steps to reproduce the behaviour:
name: "Onshape" outputDir: "build/reports/marathon" vendorConfiguration: type: "Android" installOptions: "-r -t" autoGrantPermission: true androidSdk: "$HOME/build_tools/android-sdk-31" applicationApk: "./build/outputs/apk/debug/Onshape-debug.apk" testApplicationApk: "./build/outputs/apk/androidTest/debug/Onshape-debug-androidTest.apk" batchingStrategy: type: "isolate" flakinessStrategy: type: "ignore" filteringConfiguration: allowlist:
Expected behavior Marathon should send report success as all the specified test cases pass.
Logs and reports
2022-06-16 01:07:36 D 01:07:36.879 [main @coroutine#34] <QueueActor[DevicePoolId(name=omni)]> request next batch for device 127.0.0.1:5037:emulator-5554 2022-06-16 01:07:37 Allure environment data saved. 2022-06-16 01:07:37 Marathon run finished: 2022-06-16 01:07:37 Device pool omni: 2022-06-16 01:07:37 208 passed, 0 failed, 0 ignored tests 2022-06-16 01:07:37 Flakiness overhead: 10361ms 2022-06-16 01:07:37 Raw: 220 passed, 0 failed, 0 ignored, 0 incomplete tests 2022-06-16 01:07:37 Total time: 1H 2m 42s 2022-06-16 01:07:37 2022-06-16 01:07:37 E 01:07:37.743 [main @coroutine#1] <c.m.m.e.p.tracker.PoolProgressTracker> Expected to run 220 tests but received results for only 208 Devices (please complete the following information):
Additional context Add any other context about the problem here.