search

xcherryio / xcherry

server and main repo of xCherry project
Apache License 2.0
26 stars 1 forks source link

Flaky test for TestStateExecutionLocalQueues #81

Closed longquanzheng closed 10 months ago

longquanzheng commented 10 months ago

repro at https://github.com/xdblab/xdb/actions/runs/6791516741/job/18463155053?pr=80

2023-11-07T23:24:19.9552536Z === RUN   TestStateExecutionLocalQueuesTryConsumeForStateExecution_Any_consumed
2023-11-07T23:24:19.9552756Z     state_execution_local_queues_json_test.go:258: 
2023-11-07T23:24:19.9553474Z            Error Trace:    /home/runner/work/xdb/xdb/persistence/persistencetest/state_execution_local_queues_json_test.go:258
2023-11-07T23:24:19.9553633Z            Error:          Not equal: 
2023-11-07T23:24:19.9555408Z                            expected: map[int][]persistence.InternalLocalQueueMessage{0:[]persistence.InternalLocalQueueMessage{persistence.InternalLocalQueueMessage{DedupId:"963bec49-7dc4-11ee-94c4-6045bd02b271", IsFull:false}}}
2023-11-07T23:24:19.9558183Z                            actual  : map[int][]persistence.InternalLocalQueueMessage{1:[]persistence.InternalLocalQueueMessage{persistence.InternalLocalQueueMessage{DedupId:"963bec4c-7dc4-11ee-94c4-6045bd02b271", IsFull:false}, persistence.InternalLocalQueueMessage{DedupId:"963bec4e-7dc4-11ee-94c4-6045bd02b271", IsFull:false}}}
2023-11-07T23:24:19.9558331Z                            
2023-11-07T23:24:19.9558501Z                            Diff:
2023-11-07T23:24:19.9558732Z                            --- Expected
2023-11-07T23:24:19.9558919Z                            +++ Actual
2023-11-07T23:24:19.9559168Z                            @@ -1,5 +1,9 @@
2023-11-07T23:24:19.9559686Z                             (map[int][]persistence.InternalLocalQueueMessage) (len=1) {
2023-11-07T23:24:19.9560260Z                            - (int) 0: ([]persistence.InternalLocalQueueMessage) (len=1) {
2023-11-07T23:24:19.9560772Z                            + (int) 1: ([]persistence.InternalLocalQueueMessage) (len=2) {
2023-11-07T23:24:19.9561192Z                               (persistence.InternalLocalQueueMessage) {
2023-11-07T23:24:19.9561809Z                            -   DedupId: (string) (len=36) "963bec49-7dc4-11ee-94c4-6045bd02b271",
2023-11-07T23:24:19.9562415Z                            +   DedupId: (string) (len=36) "963bec4c-7dc4-11ee-94c4-6045bd02b271",
2023-11-07T23:24:19.9562686Z                            +   IsFull: (bool) false
2023-11-07T23:24:19.9562853Z                            +  },
2023-11-07T23:24:19.9563251Z                            +  (persistence.InternalLocalQueueMessage) {
2023-11-07T23:24:19.9563854Z                            +   DedupId: (string) (len=36) "963bec4e-7dc4-11ee-94c4-6045bd02b271",
2023-11-07T23:24:19.9564330Z                                IsFull: (bool) false
2023-11-07T23:24:19.9564810Z            Test:           TestStateExecutionLocalQueuesTryConsumeForStateExecution_Any_consumed
2023-11-07T23:24:19.9565030Z     state_execution_local_queues_json_test.go:262: 
2023-11-07T23:24:19.9565737Z            Error Trace:    /home/runner/work/xdb/xdb/persistence/persistencetest/state_execution_local_queues_json_test.go:262
2023-11-07T23:24:19.9567134Z            Error:          "map[q1:[{963bec49-7dc4-11ee-94c4-6045bd02b271 %!s(bool=false)} {963bec4b-7dc4-11ee-94c4-6045bd02b271 %!s(bool=false)}] q3:[{963bec4f-7dc4-11ee-94c4-6045bd02b271 %!s(bool=false)}]]" should have 3 item(s), but has 2
2023-11-07T23:24:19.9567608Z            Test:           TestStateExecutionLocalQueuesTryConsumeForStateExecution_Any_consumed
2023-11-07T23:24:19.9567809Z     state_execution_local_queues_json_test.go:263: 
2023-11-07T23:24:19.9568499Z            Error Trace:    /home/runner/work/xdb/xdb/persistence/persistencetest/state_execution_local_queues_json_test.go:263
2023-11-07T23:24:19.9568666Z            Error:          Not equal: 
2023-11-07T23:24:19.9569998Z                            expected: []persistence.InternalLocalQueueMessage{persistence.InternalLocalQueueMessage{DedupId:"963bec4b-7dc4-11ee-94c4-6045bd02b271", IsFull:false}}
2023-11-07T23:24:19.9572346Z                            actual  : []persistence.InternalLocalQueueMessage{persistence.InternalLocalQueueMessage{DedupId:"963bec49-7dc4-11ee-94c4-6045bd02b271", IsFull:false}, persistence.InternalLocalQueueMessage{DedupId:"963bec4b-7dc4-11ee-94c4-6045bd02b271", IsFull:false}}
2023-11-07T23:24:19.9572487Z                            
2023-11-07T23:24:19.9572648Z                            Diff:
2023-11-07T23:24:19.9572881Z                            --- Expected
2023-11-07T23:24:19.9573065Z                            +++ Actual
2023-11-07T23:24:19.9573306Z                            @@ -1,2 +1,6 @@
2023-11-07T23:24:19.9573808Z                            -([]persistence.InternalLocalQueueMessage) (len=1) {
2023-11-07T23:24:19.9574252Z                            +([]persistence.InternalLocalQueueMessage) (len=2) {
2023-11-07T23:24:19.9574648Z                            + (persistence.InternalLocalQueueMessage) {
2023-11-07T23:24:19.9575267Z                            +  DedupId: (string) (len=36) "963bec49-7dc4-11ee-94c4-6045bd02b271",
2023-11-07T23:24:19.9575532Z                            +  IsFull: (bool) false
2023-11-07T23:24:19.9575696Z                            + },
2023-11-07T23:24:19.9576105Z                              (persistence.InternalLocalQueueMessage) {
2023-11-07T23:24:19.9576700Z            Test:           TestStateExecutionLocalQueuesTryConsumeForStateExecution_Any_consumed
2023-11-07T23:24:19.9576919Z     state_execution_local_queues_json_test.go:266: 
2023-11-07T23:24:19.9577628Z            Error Trace:    /home/runner/work/xdb/xdb/persistence/persistencetest/state_execution_local_queues_json_test.go:266
2023-11-07T23:24:19.9577794Z            Error:          Not equal: 
2023-11-07T23:24:19.9579991Z                            expected: []persistence.InternalLocalQueueMessage{persistence.InternalLocalQueueMessage{DedupId:"963bec4c-7dc4-11ee-94c4-6045bd02b271", IsFull:false}, persistence.InternalLocalQueueMessage{DedupId:"963bec4e-7dc4-11ee-94c4-6045bd02b271", IsFull:false}}
2023-11-07T23:24:19.9580464Z                            actual  : []persistence.InternalLocalQueueMessage(nil)
2023-11-07T23:24:19.9580611Z                            
2023-11-07T23:24:19.9580773Z                            Diff:
2023-11-07T23:24:19.9581001Z                            --- Expected
2023-11-07T23:24:19.9581198Z                            +++ Actual
2023-11-07T23:24:19.9581445Z                            @@ -1,11 +1,2 @@
2023-11-07T23:24:19.9581950Z                            -([]persistence.InternalLocalQueueMessage) (len=2) {
2023-11-07T23:24:19.9582391Z                            - (persistence.InternalLocalQueueMessage) {
2023-11-07T23:24:19.9582995Z                            -  DedupId: (string) (len=36) "963bec4c-7dc4-11ee-94c4-6045bd02b271",
2023-11-07T23:24:19.9583337Z                            -  IsFull: (bool) false
2023-11-07T23:24:19.9583521Z                            - },
2023-11-07T23:24:19.9583957Z                            - (persistence.InternalLocalQueueMessage) {
2023-11-07T23:24:19.9584557Z                            -  DedupId: (string) (len=36) "963bec4e-7dc4-11ee-94c4-6045bd02b271",
2023-11-07T23:24:19.9584855Z                            -  IsFull: (bool) false
2023-11-07T23:24:19.9585032Z                            - }
2023-11-07T23:24:19.9585203Z                            -}
2023-11-07T23:24:19.9585630Z                            +([]persistence.InternalLocalQueueMessage) <nil>
2023-11-07T23:24:19.9585770Z                             
2023-11-07T23:24:19.9586245Z            Test:           TestStateExecutionLocalQueuesTryConsumeForStateExecution_Any_consumed
2023-11-07T23:24:19.9586658Z --- FAIL: TestStateExecutionLocalQueuesTryConsumeForStateExecution_Any_consumed (0.00s)
longquanzheng commented 10 months ago

Another repro https://github.com/xdblab/xdb/actions/runs/6816783922/job/18538881254?pr=84

zklgame commented 10 months ago

Found a bug introduced in the implementation when switching to use Map from Array: https://github.com/xdblab/xdb/pull/87

longquanzheng commented 10 months ago

@zklgame it's still happening:

2023-11-10T18:32:36.3392944Z === RUN   TestStateExecutionLocalQueuesTryConsumeForStateExecution_Any_consumed
2023-11-10T18:32:36.3393157Z     state_execution_local_queues_json_test.go:258: 
2023-11-10T18:32:36.3393874Z            Error Trace:    /home/runner/work/xdb/xdb/persistence/persistencetest/state_execution_local_queues_json_test.go:258
2023-11-10T18:32:36.3394033Z            Error:          Not equal: 
2023-11-10T18:32:36.3395838Z                            expected: map[int][]persistence.InternalLocalQueueMessage{0:[]persistence.InternalLocalQueueMessage{persistence.InternalLocalQueueMessage{DedupId:"4d6200f7-7ff7-11ee-8735-000d3a5509ab", IsFull:false}}}
2023-11-10T18:32:36.3398654Z                            actual  : map[int][]persistence.InternalLocalQueueMessage{1:[]persistence.InternalLocalQueueMessage{persistence.InternalLocalQueueMessage{DedupId:"4d6200fa-7ff7-11ee-8735-000d3a5509ab", IsFull:false}, persistence.InternalLocalQueueMessage{DedupId:"4d6200fb-7ff7-11ee-8735-000d3a5509ab", IsFull:false}}}
2023-11-10T18:32:36.3398796Z                            
2023-11-10T18:32:36.3398962Z                            Diff:
2023-11-10T18:32:36.3399195Z                            --- Expected
2023-11-10T18:32:36.3399385Z                            +++ Actual
2023-11-10T18:32:36.3399638Z                            @@ -1,5 +1,9 @@
2023-11-10T18:32:36.3400147Z                             (map[int][]persistence.InternalLocalQueueMessage) (len=1) {
2023-11-10T18:32:36.3400735Z                            - (int) 0: ([]persistence.InternalLocalQueueMessage) (len=1) {
2023-11-10T18:32:36.3401225Z                            + (int) 1: ([]persistence.InternalLocalQueueMessage) (len=2) {
2023-11-10T18:32:36.3401638Z                               (persistence.InternalLocalQueueMessage) {
2023-11-10T18:32:36.3402276Z                            -   DedupId: (string) (len=36) "4d6200f7-7ff7-11ee-8735-000d3a5509ab",
2023-11-10T18:32:36.3402895Z                            +   DedupId: (string) (len=36) "4d6200fa-7ff7-11ee-8735-000d3a5509ab",
2023-11-10T18:32:36.3403163Z                            +   IsFull: (bool) false
2023-11-10T18:32:36.3403327Z                            +  },
2023-11-10T18:32:36.3403719Z                            +  (persistence.InternalLocalQueueMessage) {
2023-11-10T18:32:36.3404343Z                            +   DedupId: (string) (len=36) "4d6200fb-7ff7-11ee-8735-000d3a5509ab",
2023-11-10T18:32:36.3404630Z                                IsFull: (bool) false

see https://github.com/xdblab/xdb/actions/runs/6828404432/job/18572463957?pr=89

zklgame commented 10 months ago

Can you provide the branch name and the commit hash?

longquanzheng commented 10 months ago

Can you provide the branch name and the commit hash?

this one: https://github.com/xdblab/xdb/pull/89/commits/8980cf503bdd7ef5ac874e9a6307a2ecdf86c312

it's on the current latest commit of the branch. I ran the test again and succeeded on the second attempt

zklgame commented 10 months ago

Found the issune in the unit test. For anyOfCimplete, there is no guarantee about which command will be picked to complete if there are several commands that are able to complete.

Updated the test case: https://github.com/xdblab/xdb/pull/91