search

zhouqingqing / qpmodel

A Relational Optimizer and Executor
MIT License
66 stars 18 forks source link

[bug] race condition in emulated distributed execution #173

Closed zhouqingqing closed 4 years ago

zhouqingqing commented 4 years ago

Test case Redistribute(), which does not reliably repro but you can make it easy to repro with a endless loop:

       public void Redistribute()
        {
            for (; ; )
            {
                // needs order by to force result order
                var phyplan = "";

when it is fixed, change for(;;) to 3 times.

zhouqingqing commented 4 years ago

@zhouruiapple updates?

zhouruiapple commented 4 years ago

@zhouqingqing In progressing right now

zhouruiapple commented 4 years ago

@zhouqingqing Can you explain the bug with more information? I don't find the race condition in the code of Redistribute().

zhouqingqing commented 4 years ago

Have you tried putting the loop around the unittest?

zhouqingqing commented 4 years ago

another issue is profiling nodes actual rows = 0.

Check out here: https://github.com/zhouqingqing/qpmodel/pull/187/files

Look at tpch query plans with actual stats. You can see many with "actual rows = 0" under gather. I think what's happened here is that each individual threads have do their profiling correctly but there is no process to "summarize" profilings into the main thread's profiling nodes, so we see all zeros.

zhouruiapple commented 4 years ago

@zhouqingqing Got it.

zhouruiapple commented 4 years ago

The @totalfreq_ of @MCVList of ColumnStat cause EstSelectivity which return different value when unit test issued. Is it changed in different threads? @zhouqingqing

zhouqingqing commented 4 years ago

check out how ANALYZE is implemented - there is a GATHER on top of select statement, so analyze itself is not multi-threaded. Why ANALYZE on a non-distributed table is stable? One difference I see is that rows are in fixed order in non-distributed table while distributed table rows coming to analyze are in random order.

zhouruiapple commented 4 years ago

@zhouqingqing ths. This is the only problem left. I will fix it as soon as possible.