search

openconfig / featureprofiles

Feature Profiles are groups of OpenConfig paths and tests which verify their behavior
Apache License 2.0
48 stars 138 forks source link

featureprofiles CI: go test check flakiness #1759

Closed bstoll closed 1 year ago

bstoll commented 1 year ago

1755 saw a bug where 'go test' CI check reported a failure. A followup rerun passed. Need to investigate how/why this check is flaky.

Log of failure:

Run go test -v -coverprofile=profile.cov $(go list ./... | grep -v /.*test.*)
  go test -v -coverprofile=profile.cov $(go list ./... | grep -v /.*test.*)
  shell: /usr/bin/bash -e {0}
  env:
    GOROOT: /opt/hostedtoolcache/go/1.19.10/x64
?       github.com/openconfig/featureprofiles/feature/experimental/p4rt/internal/p4rtutils  [no test files]
?       github.com/openconfig/featureprofiles/internal/args [no test files]
?       github.com/openconfig/featureprofiles/internal/attrs    [no test files]
=== RUN   TestCheck
=== RUN   TestCheck/Equal/Correct
E0614 14:12:15.146160    2494 client.go:226] Client  error: end of updates
=== RUN   TestCheck/Equal/Incorrect
=== RUN   TestCheck/Equal/Missing
=== RUN   TestCheck/NotEqual/Correct
=== RUN   TestCheck/NotEqual/Incorrect
=== RUN   TestCheck/EqualOrNil/Correct
=== RUN   TestCheck/EqualOrNil/Nil
=== RUN   TestCheck/EqualOrNil/Incorrect
E0614 14:12:15.146604    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.146945    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.147215    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.147527    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.147850    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.151581    2494 client.go:226] Client  error: end of updates
=== RUN   TestCheck/Present/Correct
E0614 14:12:15.148114    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.154127    2494 client.go:226] Client  error: end of updates
=== RUN   TestCheck/Present/Incorrect
E0614 14:12:15.154554    2494 client.go:226] Client  error: end of updates
=== RUN   TestCheck/NotPresent/Correct
E0614 14:12:15.155041    2494 client.go:226] Client  error: end of updates
=== RUN   TestCheck/NotPresent/Incorrect
E0614 14:12:15.155919    2494 client.go:226] Client  error: end of updates
--- PASS: TestCheck (0.01s)
    --- PASS: TestCheck/Equal/Correct (0.00s)
    --- PASS: TestCheck/Equal/Incorrect (0.00s)
    --- PASS: TestCheck/Equal/Missing (0.00s)
    --- PASS: TestCheck/NotEqual/Correct (0.00s)
    --- PASS: TestCheck/NotEqual/Incorrect (0.00s)
    --- PASS: TestCheck/EqualOrNil/Correct (0.00s)
    --- PASS: TestCheck/EqualOrNil/Nil (0.00s)
    --- PASS: TestCheck/EqualOrNil/Incorrect (0.00s)
    --- PASS: TestCheck/Present/Correct (0.00s)
    --- PASS: TestCheck/Present/Incorrect (0.00s)
    --- PASS: TestCheck/NotPresent/Correct (0.00s)
    --- PASS: TestCheck/NotPresent/Incorrect (0.00s)
=== RUN   TestAwait
=== RUN   TestAwait/Immediately_correct/AwaitFor
E0614 14:12:15.157361    2494 client.go:226] Client  error: end of updates
=== RUN   TestAwait/Immediately_correct/AwaitUntil
E0614 14:12:15.157822    2494 client.go:226] Client  error: end of updates
=== RUN   TestAwait/Delayed_correct/AwaitFor
E0614 14:12:15.158325    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.158707    2494 client.go:226] Client  error: end of updates
=== RUN   TestAwait/Delayed_correct/AwaitUntil
E0614 14:12:15.159513    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.160049    2494 client.go:226] Client  error: end of updates
=== RUN   TestAwait/Too_slow/AwaitFor
=== RUN   TestAwait/Too_slow/AwaitUntil
=== RUN   TestAwait/Too_slow/multiple_values/AwaitFor
=== RUN   TestAwait/Too_slow/multiple_values/AwaitUntil
=== RUN   TestAwait/EOF_before_any_value/AwaitFor
E0614 14:12:15.364065    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.364332    2494 client.go:226] Client  error: end of updates
    check_test.go:268: Error [/parent/child/state/two: got no value, want "correct" (deadline exceeded)]: Missing substring "EOF"
=== RUN   TestAwait/EOF_before_any_value/AwaitUntil
E0614 14:12:15.414950    2494 client.go:226] Client  error: end of updates
E0614 14:12:15.415143    2494 client.go:226] Client  error: end of updates
    check_test.go:279: Error [/parent/child/state/two: got no value, want "correct" (deadline exceeded)]: Missing substring "EOF"
--- FAIL: TestAwait (0.31s)
    --- PASS: TestAwait/Immediately_correct/AwaitFor (0.00s)
    --- PASS: TestAwait/Immediately_correct/AwaitUntil (0.00s)
    --- PASS: TestAwait/Delayed_correct/AwaitFor (0.00s)
    --- PASS: TestAwait/Delayed_correct/AwaitUntil (0.00s)
    --- PASS: TestAwait/Too_slow/AwaitFor (0.05s)
    --- PASS: TestAwait/Too_slow/AwaitUntil (0.05s)
    --- PASS: TestAwait/Too_slow/multiple_values/AwaitFor (0.05s)
    --- PASS: TestAwait/Too_slow/multiple_values/AwaitUntil (0.05s)
    --- FAIL: TestAwait/EOF_before_any_value/AwaitFor (0.05s)
    --- FAIL: TestAwait/EOF_before_any_value/AwaitUntil (0.05s)
=== RUN   TestContext
=== RUN   TestContext/Canceled_context
E0614 14:12:15.920915    2494 client.go:226] Client  error: end of updates
=== RUN   TestContext/Past_context
E0614 14:12:15.921337    2494 client.go:226] Client  error: end of updates
=== RUN   TestContext/AwaitFor/past
E0614 14:12:15.926075    2494 client.go:226] Client  error: end of updates
=== RUN   TestContext/AwaitUntil/past
E0614 14:12:15.926460    2494 client.go:226] Client  error: end of updates
--- PASS: TestContext (0.46s)
    --- PASS: TestContext/Canceled_context (0.46s)
    --- PASS: TestContext/Past_context (0.00s)
    --- PASS: TestContext/AwaitFor/past (0.00s)
    --- PASS: TestContext/AwaitUntil/past (0.00s)
=== RUN   TestNetworkErrors
--- PASS: TestNetworkErrors (0.00s)
=== RUN   TestFormatValue
--- PASS: TestFormatValue (0.00s)
=== RUN   TestFormatPath
--- PASS: TestFormatPath (0.00s)
FAIL
coverage: 92.6% of statements
FAIL    github.com/openconfig/featureprofiles/internal/check    0.879s
=== RUN   TestFindMatchingStrings
--- PASS: TestFindMatchingStrings (0.00s)
PASS
coverage: 7.6% of statements
ok      github.com/openconfig/featureprofiles/internal/components   0.430s  coverage: 7.6% of statements
?       github.com/openconfig/featureprofiles/internal/confirm  [no test files]
?       github.com/openconfig/featureprofiles/internal/deviations   [no test files]
?       github.com/openconfig/featureprofiles/internal/gribi    [no test files]
?       github.com/openconfig/featureprofiles/internal/helpers  [no test files]
=== RUN   TestInit
--- PASS: TestInit (0.00s)
PASS
coverage: 85.7% of statements
ok      github.com/openconfig/featureprofiles/internal/metadata 0.013s  coverage: 85.7% of statements
?       github.com/openconfig/featureprofiles/internal/otgutils [no test files]
=== RUN   TestComputeRootPath
--- PASS: TestComputeRootPath (0.00s)
=== RUN   TestComputeRootPathError
--- PASS: TestComputeRootPathError (0.00s)
PASS
coverage: 31.2% of statements
ok      github.com/openconfig/featureprofiles/internal/pathutil 0.004s  coverage: 31.2% of statements
=== RUN   TestDUTShortVendor
--- PASS: TestDUTShortVendor (0.00s)
=== RUN   TestDUTShortModel
--- PASS: TestDUTShortModel (0.00s)
=== RUN   TestDUTPut
=== RUN   TestDUTPut/Arista
=== RUN   TestDUTPut/Cisco
=== RUN   TestDUTPut/Juniper
=== RUN   TestDUTPut/Yoyodyne
--- PASS: TestDUTPut (0.00s)
    --- PASS: TestDUTPut/Arista (0.00s)
    --- PASS: TestDUTPut/Cisco (0.00s)
    --- PASS: TestDUTPut/Juniper (0.00s)
    --- PASS: TestDUTPut/Yoyodyne (0.00s)
=== RUN   TestBuildInfo
    local_test.go:43: map[build.go_version:go1.19.10 build.main.path: build.main.sum: build.main.version: build.path:]
--- PASS: TestBuildInfo (0.00s)
=== RUN   TestGitOrigin
--- PASS: TestGitOrigin (0.00s)
=== RUN   TestGitOrigin_NoOrigin
    local_test.go:132: gitOrigin got error: remote not found
--- PASS: TestGitOrigin_NoOrigin (0.00s)
=== RUN   TestGitHead
--- PASS: TestGitHead (0.00s)
=== RUN   TestGitHead_NoHead
    local_test.go:169: gitHead got error: reference not found
--- PASS: TestGitHead_NoHead (0.00s)
=== RUN   TestGitInfoWithRepo
    local_test.go:199: map[git.clean:true git.commit:b6b0097af12d14dc19e59cb492210a3f0f7bb029 git.commit_timestamp:1686751946 git.origin:https://git.example.com/origin.git git.status:]
--- PASS: TestGitInfoWithRepo (0.00s)
=== RUN   TestGitInfoWithRepo_NotClean
    local_test.go:243: map[git.clean:false git.commit:b6b0097af12d14dc19e59cb492210a3f0f7bb029 git.commit_timestamp:1686751946 git.origin:https://git.example.com/origin.git git.status:?? bar
        ]
--- PASS: TestGitInfoWithRepo_NotClean (0.00s)
=== RUN   TestGitInfoWithRepo_NoOriginHead
    local_test.go:263: map[git.clean:true git.status:]
--- PASS: TestGitInfoWithRepo_NoOriginHead (0.00s)
=== RUN   TestGitInfo
    local_test.go:323: map[git.clean:false git.commit:0f6a0f0c8707535127a17ded5235861d24aa6412 git.commit_timestamp:1686751881 git.origin:https://github.com/openconfig/featureprofiles git.status:?? profile.cov
        ]
--- PASS: TestGitInfo (0.12s)
=== RUN   TestTestPath
--- PASS: TestTestPath (0.00s)
=== RUN   TestTestPath_FromGit
--- PASS: TestTestPath_FromGit (0.00s)
=== RUN   TestDeviationInfo
=== RUN   TestDeviationInfo/Empty
    local_test.go:392: No deviation flag specified should result in the empty map.
    local_test.go:403: map[]
=== RUN   TestDeviationInfo/DefaultValues
    local_test.go:392: Explicitly given default values should result in the empty map.
    local_test.go:403: map[]
=== RUN   TestDeviationInfo/Deviated
    local_test.go:392: Only the deviations should be enumerated.
    local_test.go:403: map[deviation.bar:NOT_DEFAULT deviation.foo:true]
--- PASS: TestDeviationInfo (0.00s)
    --- PASS: TestDeviationInfo/Empty (0.00s)
    --- PASS: TestDeviationInfo/DefaultValues (0.00s)
    --- PASS: TestDeviationInfo/Deviated (0.00s)
=== RUN   TestLocal
    local_test.go:415: map[build.go_version:go1.19.10 build.main.path: build.main.sum: build.main.version: build.path: git.clean:false git.commit:0f6a0f0c8707535127a17ded5235861d24aa6412 git.commit_timestamp:1686751881 git.origin:https://github.com/openconfig/featureprofiles git.status:?? profile.cov
         test.path:internal/rundata]
--- PASS: TestLocal (0.09s)
=== RUN   TestTopology
=== RUN   TestTopology/empty
=== RUN   TestTopology/atedut_12
=== RUN   TestTopology/dutdut
--- PASS: TestTopology (0.00s)
    --- PASS: TestTopology/empty (0.00s)
    --- PASS: TestTopology/atedut_12 (0.00s)
    --- PASS: TestTopology/dutdut (0.00s)
=== RUN   TestProperties
    rundata_test.go:132: map[build.go_version:go1.19.10 build.main.path: build.main.sum: build.main.version: build.path: git.clean:false git.commit:0f6a0f0c8707535127a17ded5235861d24aa6412 git.commit_timestamp:1686751881 git.origin:https://github.com/openconfig/featureprofiles git.status:?? profile.cov
         known_issue_url:https://example.com test.description:TestProperties unit test test.path:internal/rundata test.plan_id:TestProperties test.uuid:TestProperties123 topology:]
--- PASS: TestProperties (0.11s)
=== RUN   TestTiming
    rundata_test.go:157: map[time.begin:1686751945 time.end:1686751946]
--- PASS: TestTiming (0.00s)
PASS
coverage: 56.3% of statements
ok      github.com/openconfig/featureprofiles/internal/rundata  0.795s  coverage: 56.3% of statements
?       github.com/openconfig/featureprofiles/proto/feature_go_proto    [no test files]
?       github.com/openconfig/featureprofiles/proto/metadata_go_proto   [no test files]
?       github.com/openconfig/featureprofiles/tools [no test files]
=== RUN   TestCase_Read
=== RUN   TestCase_Read/empty
=== RUN   TestCase_Read/bad_markdown
=== RUN   TestCase_Read/good_markdown
=== RUN   TestCase_Read/bad_metadata
=== RUN   TestCase_Read/good_metadata
--- PASS: TestCase_Read (0.00s)
    --- PASS: TestCase_Read/empty (0.00s)
    --- PASS: TestCase_Read/bad_markdown (0.00s)
    --- PASS: TestCase_Read/good_markdown (0.00s)
    --- PASS: TestCase_Read/bad_metadata (0.00s)
    --- PASS: TestCase_Read/good_metadata (0.00s)
=== RUN   TestCase_Check
=== RUN   TestCase_Check/good
    case_test.go:156: Errors from check: []
=== RUN   TestCase_Check/allbad
    case_test.go:156: Errors from check: [`metadata test plan ID needs update: was "YY-1.1", will be "XX-1.1"` `metadata test description needs update: was "Description from Test", will be "Description from Markdown"` `missing testbed in metadata` `bad UUID in metadata: 123e4567-e89b-12d3-a456-426614174000: got variant RFC4122 version 1; want variant RFC4122 version 4`]
=== RUN   TestCase_Check/noexisting
    case_test.go:156: Errors from check: [`existing metadata is missing`]
=== RUN   TestCase_Check/nodata
    case_test.go:156: Errors from check: [`existing metadata is missing` `existing markdown is missing`]
--- PASS: TestCase_Check (0.00s)
    --- PASS: TestCase_Check/good (0.00s)
    --- PASS: TestCase_Check/allbad (0.00s)
    --- PASS: TestCase_Check/noexisting (0.00s)
    --- PASS: TestCase_Check/nodata (0.00s)
=== RUN   TestCase_Fix
--- PASS: TestCase_Fix (0.00s)
=== RUN   TestCase_FixUUID
--- PASS: TestCase_FixUUID (0.00s)
=== RUN   TestCase_Write
--- PASS: TestCase_Write (0.00s)
=== RUN   TestLessVersion
--- PASS: TestLessVersion (0.00s)
=== RUN   TestListCSV
--- PASS: TestListCSV (0.00s)
=== RUN   TestFeatureFromTestDir
--- PASS: TestFeatureFromTestDir (0.00s)
=== RUN   TestSortedByTestPlanID
--- PASS: TestSortedByTestPlanID (0.00s)
=== RUN   TestListTestTracker
--- PASS: TestListTestTracker (0.00s)
=== RUN   TestListTestTracker_Merge
--- PASS: TestListTestTracker_Merge (0.00s)
=== RUN   TestTTBuildPlan
--- PASS: TestTTBuildPlan (0.00s)
=== RUN   TestTTBuildPlan_MissingData
Missing rundata: feature/xyzzy/tests/quuz_test
--- PASS: TestTTBuildPlan_MissingData (0.00s)
=== RUN   TestTTBuildPlan_DisallowReuseUUID
Test UUID c857db98-7b2c-433c-b9fb-4511b42edd78 has inconsistent data at feature/foo/baz/quuz_test and map[string]string{"ate_tests":"feature/foo/bar/ate_tests/qux_test"}
--- PASS: TestTTBuildPlan_DisallowReuseUUID (0.00s)
=== RUN   TestSection
--- PASS: TestSection (0.00s)
=== RUN   TestJSONQuote
--- PASS: TestJSONQuote (0.00s)
=== RUN   TestTTPlan_SortedKeys
--- PASS: TestTTPlan_SortedKeys (0.00s)
=== RUN   TestTTPlan_Merge
--- PASS: TestTTPlan_Merge (0.00s)
=== RUN   TestChildSuite
=== RUN   TestChildSuite/NotMap
=== RUN   TestChildSuite/NoText
=== RUN   TestChildSuite/NoMatch
=== RUN   TestChildSuite/Match
--- PASS: TestChildSuite (0.00s)
    --- PASS: TestChildSuite/NotMap (0.00s)
    --- PASS: TestChildSuite/NoText (0.00s)
    --- PASS: TestChildSuite/NoMatch (0.00s)
    --- PASS: TestChildSuite/Match (0.00s)
=== RUN   TestTTSuite_SortedKeys
--- PASS: TestTTSuite_SortedKeys (0.00s)
=== RUN   TestTTSuite_Merge
--- PASS: TestTTSuite_Merge (0.00s)
=== RUN   TestChildCase
=== RUN   TestChildCase/BadTitle_NotString
=== RUN   TestChildCase/BadTitle_BadString
=== RUN   TestChildCase/All
=== RUN   TestChildCase/NoAttrs
=== RUN   TestChildCase/NoTitle
=== RUN   TestChildCase/BadUUID_NotString
=== RUN   TestChildCase/BadUUID_BadString
=== RUN   TestChildCase/NormalizeUUID
=== RUN   TestChildCase/NotMap
=== RUN   TestChildCase/NoUUID
--- PASS: TestChildCase (0.00s)
    --- PASS: TestChildCase/BadTitle_NotString (0.00s)
    --- PASS: TestChildCase/BadTitle_BadString (0.00s)
    --- PASS: TestChildCase/All (0.00s)
    --- PASS: TestChildCase/NoAttrs (0.00s)
    --- PASS: TestChildCase/NoTitle (0.00s)
    --- PASS: TestChildCase/BadUUID_NotString (0.00s)
    --- PASS: TestChildCase/BadUUID_BadString (0.00s)
    --- PASS: TestChildCase/NormalizeUUID (0.00s)
    --- PASS: TestChildCase/NotMap (0.00s)
    --- PASS: TestChildCase/NoUUID (0.00s)
=== RUN   TestTTDesc
--- PASS: TestTTDesc (0.00s)
=== RUN   TestTTCase_Merge
--- PASS: TestTTCase_Merge (0.00s)
=== RUN   TestParseMarkdown
=== RUN   TestParseMarkdown/standard
=== RUN   TestParseMarkdown/excess_spaces
=== RUN   TestParseMarkdown/no_space
--- PASS: TestParseMarkdown (0.00s)
    --- PASS: TestParseMarkdown/standard (0.00s)
    --- PASS: TestParseMarkdown/excess_spaces (0.00s)
    --- PASS: TestParseMarkdown/no_space (0.00s)
=== RUN   TestWriteProto
--- PASS: TestWriteProto (0.00s)
=== RUN   TestSuite_Read
--- PASS: TestSuite_Read (0.00s)
=== RUN   TestSuite_Read_BadPath
Test found in a bad path: 001/foo/bar/qux_test/foo_test.go
--- PASS: TestSuite_Read_BadPath (0.00s)
=== RUN   TestSuite_Check
=== RUN   TestSuite_Check/NeedsUpdate
Found 1 errors in foo/bar/tests/qux_test
  - existing metadata is missing
Skipping check for duplicate test UUID due to missing value: foo/bar/tests/qux_test
=== RUN   TestSuite_Check/Updated
=== RUN   TestSuite_Check/DuplicateTestPlanID
Found 1 errors in foo/bar/tests/quuz_test
  - missing testbed in metadata
Duplicate test plan ID found at foo/bar/tests/quuz_test, already used by foo/bar/tests/qux_test
=== RUN   TestSuite_Check/DuplicateUUID
Found 1 errors in foo/bar/tests/quuz_test
  - missing testbed in metadata
Duplicate test UUID found at foo/bar/tests/quuz_test, already used by foo/bar/tests/qux_test
=== RUN   TestSuite_Check/SameATEOTG
=== RUN   TestSuite_Check/DifferentATEOTG
ATE and OTG tests have different test plan IDs: foo/bar/ate_tests/qux_test
  - ATE: XX-2.1
  - OTG: XX-2.2
ATE and OTG tests have different test descriptions: foo/bar/ate_tests/qux_test
  - ATE: Qux Functional Test
  - OTG: Quuz Functional Test
ATE and OTG tests have different UUIDs: foo/bar/ate_tests/qux_test
  - ATE: c857db98-7b2c-433c-b9fb-4511b42edd78
  - OTG: a5413d74-5b44-49d2-b4e7-84c9751d50be
--- PASS: TestSuite_Check (0.00s)
    --- PASS: TestSuite_Check/NeedsUpdate (0.00s)
    --- PASS: TestSuite_Check/Updated (0.00s)
    --- PASS: TestSuite_Check/DuplicateTestPlanID (0.00s)
    --- PASS: TestSuite_Check/DuplicateUUID (0.00s)
    --- PASS: TestSuite_Check/SameATEOTG (0.00s)
    --- PASS: TestSuite_Check/DifferentATEOTG (0.00s)
=== RUN   TestSuite_Fix
--- PASS: TestSuite_Fix (0.00s)
=== RUN   TestSuite_ReadFixWriteReadCheck
Updated 001/foo/bar/tests/quuz_test
Updated 001/foo/bar/ate_tests/qux_test
Updated 001/foo/bar/otg_tests/qux_test
--- PASS: TestSuite_ReadFixWriteReadCheck (0.00s)
PASS
coverage: 79.8% of statements
ok      github.com/openconfig/featureprofiles/tools/addrundata  0.014s  coverage: 79.8% of statements
=== RUN   TestSVGBadge
=== RUN   TestSVGBadge/Label_exists_in_output
=== RUN   TestSVGBadge/Message_exists_in_output
=== RUN   TestSVGBadge/Success_message_uses_color_#4C1
=== RUN   TestSVGBadge/Failure_message_uses_color_#E05D44
=== RUN   TestSVGBadge/Message_default_color_is_#9F9F9F
=== RUN   TestSVGBadge/Empty_inputs
--- PASS: TestSVGBadge (0.00s)
    --- PASS: TestSVGBadge/Label_exists_in_output (0.00s)
    --- PASS: TestSVGBadge/Message_exists_in_output (0.00s)
    --- PASS: TestSVGBadge/Success_message_uses_color_#4C1 (0.00s)
    --- PASS: TestSVGBadge/Failure_message_uses_color_#E05D44 (0.00s)
    --- PASS: TestSVGBadge/Message_default_color_is_#9F9F9F (0.00s)
    --- PASS: TestSVGBadge/Empty_inputs (0.00s)
=== RUN   TestEstimateStringWidth
=== RUN   TestEstimateStringWidth/Empty_String
=== RUN   TestEstimateStringWidth/Standard_String
--- PASS: TestEstimateStringWidth (0.00s)
    --- PASS: TestEstimateStringWidth/Empty_String (0.00s)
    --- PASS: TestEstimateStringWidth/Standard_String (0.00s)
=== RUN   TestCreateTGZArchive
--- PASS: TestCreateTGZArchive (0.00s)
=== RUN   TestModifiedFunctionalTests
=== RUN   TestModifiedFunctionalTests/path_test
=== RUN   TestModifiedFunctionalTests/nothing_shared
=== RUN   TestModifiedFunctionalTests/everything_shared
=== RUN   TestModifiedFunctionalTests/some_shared
=== RUN   TestModifiedFunctionalTests/different_size_inputs
=== RUN   TestModifiedFunctionalTests/empty_inputs
=== RUN   TestModifiedFunctionalTests/nil_inputs
--- PASS: TestModifiedFunctionalTests (0.00s)
    --- PASS: TestModifiedFunctionalTests/path_test (0.00s)
    --- PASS: TestModifiedFunctionalTests/nothing_shared (0.00s)
    --- PASS: TestModifiedFunctionalTests/everything_shared (0.00s)
    --- PASS: TestModifiedFunctionalTests/some_shared (0.00s)
    --- PASS: TestModifiedFunctionalTests/different_size_inputs (0.00s)
    --- PASS: TestModifiedFunctionalTests/empty_inputs (0.00s)
    --- PASS: TestModifiedFunctionalTests/nil_inputs (0.00s)
=== RUN   TestFunctionalTestPaths
--- PASS: TestFunctionalTestPaths (0.00s)
=== RUN   TestPopulateTestDetail
--- PASS: TestPopulateTestDetail (0.00s)
PASS
coverage: 20.1% of statements
ok      github.com/openconfig/featureprofiles/tools/ci-trigger  0.014s  coverage: 20.1% of statements
?       github.com/openconfig/featureprofiles/tools/lint2annotation [no test files]
?       github.com/openconfig/featureprofiles/tools/wikidoc [no test files]
=== RUN   TestReserveRelease
--- PASS: TestReserveRelease (0.00s)
=== RUN   TestReservation
--- PASS: TestReservation (0.00s)
=== RUN   TestReservation_Error
    binding_test.go:203: Got reservation errors: 5 errors occurred:
          * missing binding for DUT "dut.tb"
          * error binding DUT "dut.both": 2 errors occurred:
              * binding port "port3" not found in testbed
              * testbed port "port1" is missing in binding
          * binding DUT "dut.b" not found in testbed
          * missing binding for ATE "ate.tb"
          * error binding ATE "ate.both": 2 errors occurred:
              * binding port "port3" not found in testbed
              * testbed port "port1" is missing in binding
--- PASS: TestReservation_Error (0.00s)
=== RUN   TestCLI
    cli_test.go:154: Client begins handshake.
    cli_test.go:138: Server begins handshake.
    cli_test.go:147: Server is ready.
    cli_test.go:161: Client is ready.
    cli_test.go:288: Test is ready.
=== RUN   TestCLI/StdinStdout
=== RUN   TestCLI/StdinStderr
=== RUN   TestCLI/SendCommand
--- PASS: TestCLI (0.00s)
    --- PASS: TestCLI/StdinStdout (0.00s)
    --- PASS: TestCLI/StdinStderr (0.00s)
    --- PASS: TestCLI/SendCommand (0.00s)
=== RUN   TestMerge
=== RUN   TestMerge/One
=== RUN   TestMerge/Disjoint
=== RUN   TestMerge/MultipleOverride
--- PASS: TestMerge (0.00s)
    --- PASS: TestMerge/One (0.00s)
    --- PASS: TestMerge/Disjoint (0.00s)
    --- PASS: TestMerge/MultipleOverride (0.00s)
=== RUN   TestResolver_ByID
=== RUN   TestResolver_ByID/dutByID(dut)
=== RUN   TestResolver_ByID/dutByID(anotherdut)
=== RUN   TestResolver_ByID/ateByID(ate)
=== RUN   TestResolver_ByID/ateByID(anotherate)
=== RUN   TestResolver_ByID/dutByID(no.such.dut)
=== RUN   TestResolver_ByID/ateByID(no.such.ate)
=== RUN   TestResolver_ByID/ateByID(dut)
=== RUN   TestResolver_ByID/dutByID(ate)
--- PASS: TestResolver_ByID (0.00s)
    --- PASS: TestResolver_ByID/dutByID(dut) (0.00s)
    --- PASS: TestResolver_ByID/dutByID(anotherdut) (0.00s)
    --- PASS: TestResolver_ByID/ateByID(ate) (0.00s)
    --- PASS: TestResolver_ByID/ateByID(anotherate) (0.00s)
    --- PASS: TestResolver_ByID/dutByID(no.such.dut) (0.00s)
    --- PASS: TestResolver_ByID/ateByID(no.such.ate) (0.00s)
    --- PASS: TestResolver_ByID/ateByID(dut) (0.00s)
    --- PASS: TestResolver_ByID/dutByID(ate) (0.00s)
=== RUN   TestResolver_ByName_Protocols
=== RUN   TestResolver_ByName_Protocols/ssh
=== RUN   TestResolver_ByName_Protocols/gnmi
=== RUN   TestResolver_ByName_Protocols/gnoi
=== RUN   TestResolver_ByName_Protocols/gnsi
=== RUN   TestResolver_ByName_Protocols/gribi
=== RUN   TestResolver_ByName_Protocols/p4rt
=== RUN   TestResolver_ByName_Protocols/ixnetwork
=== RUN   TestResolver_ByName_Protocols/anotherdut
=== RUN   TestResolver_ByName_Protocols/anotherate
--- PASS: TestResolver_ByName_Protocols (0.00s)
    --- PASS: TestResolver_ByName_Protocols/ssh (0.00s)
    --- PASS: TestResolver_ByName_Protocols/gnmi (0.00s)
    --- PASS: TestResolver_ByName_Protocols/gnoi (0.00s)
    --- PASS: TestResolver_ByName_Protocols/gnsi (0.00s)
    --- PASS: TestResolver_ByName_Protocols/gribi (0.00s)
    --- PASS: TestResolver_ByName_Protocols/p4rt (0.00s)
    --- PASS: TestResolver_ByName_Protocols/ixnetwork (0.00s)
    --- PASS: TestResolver_ByName_Protocols/anotherdut (0.00s)
    --- PASS: TestResolver_ByName_Protocols/anotherate (0.00s)
=== RUN   TestResolver_ByName_Protocols_Error
=== RUN   TestResolver_ByName_Protocols_Error/no.such.dut.name
    options_test.go:368: Resolve got error: dut name "no.such.dut.name" is missing from the binding
=== RUN   TestResolver_ByName_Protocols_Error/no.such.ate.name
    options_test.go:368: Resolve got error: ate name "no.such.ate.name" is missing from the binding
=== RUN   TestResolver_ByName_Protocols_Error/ate.name
    options_test.go:368: Resolve got error: dut name "ate.name" is missing from the binding
=== RUN   TestResolver_ByName_Protocols_Error/ate.name#01
    options_test.go:368: Resolve got error: dut name "ate.name" is missing from the binding
=== RUN   TestResolver_ByName_Protocols_Error/ate.name#02
    options_test.go:368: Resolve got error: dut name "ate.name" is missing from the binding
=== RUN   TestResolver_ByName_Protocols_Error/ate.name#03
    options_test.go:368: Resolve got error: dut name "ate.name" is missing from the binding
=== RUN   TestResolver_ByName_Protocols_Error/ate.name#04
    options_test.go:368: Resolve got error: dut name "ate.name" is missing from the binding
=== RUN   TestResolver_ByName_Protocols_Error/ate.name#05
    options_test.go:368: Resolve got error: dut name "ate.name" is missing from the binding
=== RUN   TestResolver_ByName_Protocols_Error/dut.name
    options_test.go:368: Resolve got error: ate name "dut.name" is missing from the binding
--- PASS: TestResolver_ByName_Protocols_Error (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/no.such.dut.name (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/no.such.ate.name (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/ate.name (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/ate.name#01 (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/ate.name#02 (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/ate.name#03 (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/ate.name#04 (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/ate.name#05 (0.00s)
    --- PASS: TestResolver_ByName_Protocols_Error/dut.name (0.00s)
PASS
coverage: 31.6% of statements
ok      github.com/openconfig/featureprofiles/topologies/binding    0.447s  coverage: 31.6% of statements
?       github.com/openconfig/featureprofiles/topologies/proto/binding  [no test files]
FAIL
Error: Process completed with exit code 1.
greg-dennis commented 1 year ago

Yes, these have been flaky quite a bit. I've been meaning to flag it. Thank you for filing the issue

bstoll commented 1 year ago

Still seeing similar errors:

=== RUN   TestAwait/Too_slow/AwaitFor
    check_test.go:268: Error [/parent/child/state/two: context deadline exceeded]:
          Missing substring "wrong"
          Missing substring "correct"
=== RUN   TestAwait/Too_slow/AwaitUntil
    check_test.go:279: Error [/parent/child/state/two: context deadline exceeded]:
          Missing substring "wrong"
          Missing substring "correct"
=== RUN   TestAwait/Too_slow/multiple_values/AwaitFor
    check_test.go:268: Error [/parent/child/state/two: context deadline exceeded]:
          Missing substring "wrong2"
          Missing substring "correct"
=== RUN   TestAwait/Too_slow/multiple_values/AwaitUntil
    check_test.go:279: Error [/parent/child/state/two: context deadline exceeded]:
          Missing substring "wrong2"
          Missing substring "correct"
=== RUN   TestAwait/EOF_before_any_value/AwaitFor
E0731 23:54:20.177124    3558 client.go:226] Client  error: end of updates
E0731 23:54:20.177642    3558 client.go:226] Client  error: end of updates
=== RUN   TestAwait/EOF_before_any_value/AwaitUntil
E0731 23:54:20.178142    3558 client.go:226] Client  error: end of updates
E0731 23:54:20.178538    3558 client.go:226] Client  error: end of updates
--- FAIL: TestAwait (2.01s)
    --- PASS: TestAwait/Immediately_correct/AwaitFor (0.00s)
    --- PASS: TestAwait/Immediately_correct/AwaitUntil (0.00s)
    --- PASS: TestAwait/Delayed_correct/AwaitFor (0.00s)
    --- PASS: TestAwait/Delayed_correct/AwaitUntil (0.00s)
    --- FAIL: TestAwait/Too_slow/AwaitFor (0.50s)
    --- FAIL: TestAwait/Too_slow/AwaitUntil (0.50s)
    --- FAIL: TestAwait/Too_slow/multiple_values/AwaitFor (0.50s)
    --- FAIL: TestAwait/Too_slow/multiple_values/AwaitUntil (0.50s)
    --- PASS: TestAwait/EOF_before_any_value/AwaitFor (0.00s)
    --- PASS: TestAwait/EOF_before_any_value/AwaitUntil (0.00s)

in #1961. Need to figure out if there is a better way to address this issue.

bstoll commented 1 year ago

I missed @greg-dennis's followup commit updated the isTimeout check in #1961 that should have addressed this test failure.