search

stoneatom / stonedb

StoneDB is an Open-Source MySQL HTAP and MySQL-Native DataBase for OLTP, Real-Time Analytics, a counterpart of MySQLHeatWave. (https://stonedb.io)
https://stonedb.io/
GNU General Public License v2.0
861 stars 140 forks source link

feature: make aggregation parallel process to reduce execution time #422

Open adofsauron opened 2 years ago

adofsauron commented 2 years ago

Documentation for mysql aggregation

https://dev.mysql.com/doc/refman/8.0/en/aggregate-functions-and-modifiers.html

Functional Requirements:

  1. Consistent with aggregation conditions handled by existing logic
  2. The Aggregate function is used as the entry and ResultSender is used as the result output

Performance Requirements:

Table data limit:

  1. The upper limit of a physical drive
  2. The memory limit is exceeded. Procedure

Memory usage limit:

  1. Add parameters to control the maximum amount of memory that can be used for aggregation
  2. Otherwise temporary disk files if the limit is exceeded

Execution time limit:

  1. Under the same conditions, it is in the same order of magnitude as InnoDB aggregate query at most
  2. Aim to take less time than InnoDB for aggregated queries

Aggregate query SQL


    select
        c_name,
        c_custkey,
        o_orderkey,
        o_orderdate,
        o_totalprice,
        sum(l_quantity)
    from
        customer,
        orders,
        lineitem
    where
        c_custkey = o_custkey
        and o_orderkey = l_orderkey
    group by
        c_name,
        c_custkey,
        o_orderkey,
        o_orderdate,
        o_totalprice
    order by
        o_totalprice desc,
        o_orderdate
    limit 10;

mysql> desc part;
+---------------+---------------+------+-----+---------+-------+
| Field         | Type          | Null | Key | Default | Extra |
+---------------+---------------+------+-----+---------+-------+
| p_partkey     | int(11)       | NO   | PRI | NULL    |       |
| p_name        | varchar(55)   | NO   |     | NULL    |       |
| p_mfgr        | char(25)      | NO   |     | NULL    |       |
| p_brand       | char(10)      | NO   |     | NULL    |       |
| p_type        | varchar(25)   | NO   |     | NULL    |       |
| p_size        | int(11)       | NO   |     | NULL    |       |
| p_container   | char(10)      | NO   |     | NULL    |       |
| p_retailprice | decimal(15,2) | NO   |     | NULL    |       |
| p_comment     | varchar(23)   | NO   |     | NULL    |       |
+---------------+---------------+------+-----+---------+-------+
9 rows in set (0.00 sec)


mysql> desc partsupp;
+---------------+---------------+------+-----+---------+-------+
| Field         | Type          | Null | Key | Default | Extra |
+---------------+---------------+------+-----+---------+-------+
| ps_partkey    | int(11)       | NO   | PRI | NULL    |       |
| ps_suppkey    | int(11)       | NO   | PRI | NULL    |       |
| ps_availqty   | int(11)       | NO   |     | NULL    |       |
| ps_supplycost | decimal(15,2) | NO   |     | NULL    |       |
| ps_comment    | varchar(199)  | NO   |     | NULL    |       |
+---------------+---------------+------+-----+---------+-------+
5 rows in set (0.00 sec)


    select
        p_brand,
        p_type,
        p_size,
        count(distinct ps_suppkey) as supplier_cnt
    from
        partsupp,
        part
    where
        p_partkey = ps_partkey
        and p_brand <> 'Brand#45'
        and p_type not like 'MEDIUM POLISHED%'
        and p_size in (49, 14, 23, 45, 19, 3, 36, 9)
        and ps_suppkey not in (
            select
                s_suppkey
            from
                supplier
            where
                s_comment like '%Customer%Complaints%'
        )
    group by
        p_brand,
        p_type,
        p_size
    order by
        supplier_cnt desc,
        p_brand,
        p_type,
        p_size
    limit 10;
adofsauron commented 1 year ago

Obtain data:

a90c62931c306261c0e6dcb9847c972a


(gdb) bt
#0  Tianmu::compress::TextCompressor::DecompressCopy (this=0x7f75e1e39fc0, dest=0x7f745d286000 "", dlen=1179648, 
    src=0x7f745d165019 "Supplier#000000001Supplier#000000002Supplier#000000003Supplier#000000004Supplier#000000005Supplier#000000006Supplier#000000007Supplier#000000008Supplier#000000009Supplier#000000010Supplier#000000011Su"..., slen=1179648, index=0x7f745cf8b000, lens=0x7f745d00c000, nrec=65536) at /root/work/stonedb-dev-20230103/storage/tianmu/compress/text_compressor.cpp:129
#1  0x000000000306a677 in Tianmu::compress::TextCompressor::Decompress (this=0x7f75e1e39fc0, dest=0x7f745d286000 "", dlen=1179648, src=0x7f745d165017 "", slen=1179650, index=0x7f745cf8b000, 
    lens=0x7f745d00c000, nrec=65536) at /root/work/stonedb-dev-20230103/storage/tianmu/compress/text_compressor.cpp:562
#2  0x000000000301bee3 in Tianmu::core::PackStr::LoadCompressed (this=0x7f7438daf880, f=0x7f75e1e5e670) at /root/work/stonedb-dev-20230103/storage/tianmu/core/pack_str.cpp:671
#3  0x0000000003017e56 in Tianmu::core::PackStr::LoadDataFromFile (this=0x7f7438daf880, f=0x7f75e1e5e670) at /root/work/stonedb-dev-20230103/storage/tianmu/core/pack_str.cpp:108
#4  0x0000000003017838 in Tianmu::core::PackStr::PackStr (this=0x7f7438daf880, dpn=0x7f740490afa8, pc=..., col_share=0x7f7438a4fbe0)
    at /root/work/stonedb-dev-20230103/storage/tianmu/core/pack_str.cpp:57
#5  0x0000000002d34ca4 in __gnu_cxx::new_allocator<Tianmu::core::PackStr>::construct<Tianmu::core::PackStr<Tianmu::core::DPN*&, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty> const&, Tianmu::core::ColumnShare*&> > (this=0x7f75e1e5e87f, __p=0x7f7438daf880) at /usr/include/c++/8/ext/new_allocator.h:136
#6  0x0000000002d33cae in std::allocator_traits<std::allocator<Tianmu::core::PackStr> >::construct<Tianmu::core::PackStr<Tianmu::core::DPN*&, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty> const&, Tianmu::core::ColumnShare*&> > (__a=..., __p=0x7f7438daf880) at /usr/include/c++/8/bits/alloc_traits.h:475
#7  0x0000000002d31fbc in std::_Sp_counted_ptr_inplace<Tianmu::core::PackStr, std::allocator<Tianmu::core::PackStr>, (__gnu_cxx::_Lock_policy)2>::_Sp_counted_ptr_inplace<Tianmu::core::DPN*&, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty> const&, Tianmu::core::ColumnShare*&> (this=0x7f7438daf870, __a=...) at /usr/include/c++/8/bits/shared_ptr_base.h:545
#8  0x0000000002d2f999 in std::__shared_count<(__gnu_cxx::_Lock_policy)2>::__shared_count<Tianmu::core::PackStr, std::allocator<Tianmu::core::PackStr>, Tianmu::core::DPN*&, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty> const&, Tianmu::core::ColumnShare*&> (this=0x7f75e1e5ea78, __p=@0x7f75e1e5ea70: 0x0, __a=...)
    at /usr/include/c++/8/bits/shared_ptr_base.h:677
#9  0x0000000002d2d09e in std::__shared_ptr<Tianmu::core::PackStr, (__gnu_cxx::_Lock_policy)2>::__shared_ptr<std::allocator<Tianmu::core::PackStr>, Tianmu::core::DPN*&, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty> const&, Tianmu::core::ColumnShare*&> (this=0x7f75e1e5ea70, __tag=...) at /usr/include/c++/8/bits/shared_ptr_base.h:1342
#10 0x0000000002d2af0b in std::shared_ptr<Tianmu::core::PackStr>::shared_ptr<std::allocator<Tianmu::core::PackStr>, Tianmu::core::DPN*&, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty> const&, Tianmu::core::ColumnShare*&> (this=0x7f75e1e5ea70, __tag=...) at /usr/include/c++/8/bits/shared_ptr.h:359
#11 0x0000000002d291ed in std::allocate_shared<Tianmu::core::PackStr, std::allocator<Tianmu::core::PackStr>, Tianmu::core::DPN*&, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty> const&, Tianmu::core::ColumnShare*&> (__a=...) at /usr/include/c++/8/bits/shared_ptr.h:708
#12 0x0000000002d266d3 in std::make_shared<Tianmu::core::PackStr, Tianmu::core::DPN*&, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty> const&, Tianmu::core::ColumnShare*&> () at /usr/include/c++/8/bits/shared_ptr.h:724
#13 0x0000000002d1b497 in Tianmu::core::TianmuAttr::Fetch (this=0x7f7438a76a00, pc=...) at /root/work/stonedb-dev-20230103/storage/tianmu/core/tianmu_attr.cpp:847
#14 0x0000000002d26245 in Tianmu::core::DataCache::GetOrFetchObject<Tianmu::core::Pack, Tianmu::core::ObjectId<(Tianmu::core::COORD_TYPE)0, 3, Tianmu::core::object_id_helper::empty>, Tianmu::core::TianmuAttr> (this=0x49cfb40, coord_=..., fetcher_=0x7f7438a76a00) at /root/work/stonedb-dev-20230103/storage/tianmu/core/data_cache.h:234
#15 0x0000000002d1ab94 in Tianmu::core::TianmuAttr::LockPackForUse (this=0x7f7438a76a00, pn=0) at /root/work/stonedb-dev-20230103/storage/tianmu/core/tianmu_attr.cpp:783
#16 0x0000000002d51892 in Tianmu::core::TianmuTable::LockPackForUse (this=0x7f7438a74bc0, attr=1, pack_no=0) at /root/work/stonedb-dev-20230103/storage/tianmu/core/tianmu_table.cpp:212
#17 0x0000000002ff4467 in Tianmu::core::VCPackGuardian::LockPackrow (this=0x7f7438a52f48, mit=...) at /root/work/stonedb-dev-20230103/storage/tianmu/core/pack_guardian.cpp:77
#18 0x0000000002cac8c8 in Tianmu::vcolumn::VirtualColumn::LockSourcePacks (this=0x7f7438a52e70, mit=...) at /root/work/stonedb-dev-20230103/storage/tianmu/vc/virtual_column.h:45
#19 0x0000000002d00571 in Tianmu::core::TempTable::SendResult (this=0x7f7438a51ce0, limit=10, offset=0, sender=..., pagewise=false)
    at /root/work/stonedb-dev-20230103/storage/tianmu/core/temp_table_low.cpp:460
#20 0x0000000002cff817 in Tianmu::core::TempTable::FillMaterializedBuffers (this=0x7f7438a51ce0, local_limit=10, local_offset=0, sender=0x7f7438dad010, pagewise=false)
    at /root/work/stonedb-dev-20230103/storage/tianmu/core/temp_table_low.cpp:317
#21 0x0000000002ce9247 in Tianmu::core::TempTable::Materialize (this=0x7f7438a51ce0, in_subq=false, sender=0x7f7438dad010, lazy=false)
    at /root/work/stonedb-dev-20230103/storage/tianmu/core/temp_table.cpp:2045
#22 0x0000000002c74b53 in Tianmu::core::Engine::Execute (this=0x49cf7a0, thd=0x7f7438150850, lex=0x7f7438152b78, result_output=0x7f7438156320, unit_for_union=0x0)
    at /root/work/stonedb-dev-20230103/storage/tianmu/core/engine_execute.cpp:482
#23 0x0000000002c738be in Tianmu::core::Engine::HandleSelect (this=0x49cf7a0, thd=0x7f7438150850, lex=0x7f7438152b78, result=@0x7f75e1e5fdc8: 0x7f7438156320, setup_tables_done_option=0, 
    res=@0x7f75e1e5fdc4: 0, optimize_after_tianmu=@0x7f75e1e5fdbc: 1, tianmu_free_join=@0x7f75e1e5fdc0: 1, with_insert=0) at /root/work/stonedb-dev-20230103/storage/tianmu/core/engine_execute.cpp:238
#24 0x0000000002d76ae1 in Tianmu::handler::ha_my_tianmu_query (thd=0x7f7438150850, lex=0x7f7438152b78, result_output=@0x7f75e1e5fdc8: 0x7f7438156320, setup_tables_done_option=0, res=@0x7f75e1e5fdc4: 0, 
    optimize_after_tianmu=@0x7f75e1e5fdbc: 1, tianmu_free_join=@0x7f75e1e5fdc0: 1, with_insert=0) at /root/work/stonedb-dev-20230103/storage/tianmu/handler/ha_my_tianmu.cpp:88
--Type <RET> for more, q to quit, c to continue without paging--
#25 0x00000000023b4d19 in execute_sqlcom_select (thd=0x7f7438150850, all_tables=0x7f7438155d80) at /root/work/stonedb-dev-20230103/sql/sql_parse.cc:4875
#26 0x00000000023ae08f in mysql_execute_command (thd=0x7f7438150850, first_level=true) at /root/work/stonedb-dev-20230103/sql/sql_parse.cc:2675
#27 0x00000000023b5d7f in mysql_parse (thd=0x7f7438150850, parser_state=0x7f75e1e60f90) at /root/work/stonedb-dev-20230103/sql/sql_parse.cc:5279
#28 0x00000000023aad68 in dispatch_command (thd=0x7f7438150850, com_data=0x7f75e1e61730, command=COM_QUERY) at /root/work/stonedb-dev-20230103/sql/sql_parse.cc:1399
#29 0x00000000023a9ba9 in do_command (thd=0x7f7438150850) at /root/work/stonedb-dev-20230103/sql/sql_parse.cc:976
#30 0x00000000024db375 in handle_connection (arg=0x7d96430) at /root/work/stonedb-dev-20230103/sql/conn_handler/connection_handler_per_thread.cc:313
#31 0x0000000002bab156 in pfs_spawn_thread (arg=0x7868400) at /root/work/stonedb-dev-20230103/storage/perfschema/pfs.cc:2197
#32 0x00007f76320f41ca in start_thread () from /lib64/libpthread.so.0
#33 0x00007f762f244e73 in clone () from /lib64/libc.so.6
adofsauron commented 1 year ago

Caching of data


  template <typename T, typename U, typename V>
  std::shared_ptr<T> GetOrFetchObject(U const &coord_, V *fetcher_) {
    auto &c(cache<U>());
    auto &w(waitIO<U>());
    auto &cond(condition<U>());
    bool waited = false;
    /* a scope for mutex lock */
    {
      /* Lock is acquired inside */
      std::unique_lock<std::recursive_mutex> m_obj_guard(mm::TraceableObject::GetLockingMutex());

      std::scoped_lock lock(m_cache_mutex);

      auto it = c.find(coord_);

      if (it != c.end()) {
        if constexpr (U::ID == COORD_TYPE::PACK) {
          it->second->Lock();
          ++m_cacheHits;
          it->second->TrackAccess();
        }
        return std::static_pointer_cast<T>(it->second);
      }

      {
        if constexpr (U::ID == COORD_TYPE::PACK)
          m_cacheMisses++;
        auto rit = w.find(coord_);
        while (rit != w.end()) {
          m_readWaitInProgress++;
          if (waited)
            m_falseWakeup++;
          else
            m_readWait++;

          m_cache_mutex.unlock();
          cond.wait(m_obj_guard);
          m_cache_mutex.lock();

          waited = true;
          m_readWaitInProgress--;
          rit = w.find(coord_);
        }
        it = c.find(coord_);
        if (it != c.end()) {
          if constexpr (U::ID == COORD_TYPE::PACK) {
            it->second->Lock();
            it->second->TrackAccess();
          }
          return std::static_pointer_cast<T>(it->second);
        }
        // mm::TraceableObject::GetLockingMutex().Unlock();
        // if we get here the obj has been loaded, used, unlocked
        // and pushed out of memory before we should get to it after
        // waiting
      }
      w.insert(coord_);
      m_packLoadInProgress++;
    }

    std::shared_ptr<T> obj;
    try {
      obj = fetcher_->Fetch(coord_);
    } catch (...) {
      std::scoped_lock m_obj_guard(mm::TraceableObject::GetLockingMutex());
      std::scoped_lock lock(m_cache_mutex);
      m_loadErrors++;
      m_packLoadInProgress--;
      w.erase(coord_);
      cond.notify_all();
      throw;
    }

    {
      obj->SetOwner(this);
      std::scoped_lock m_obj_guard(mm::TraceableObject::GetLockingMutex());
      std::scoped_lock lock(m_cache_mutex);
      if constexpr (U::ID == COORD_TYPE::PACK) {
        m_packLoads++;
        obj->TrackAccess();
      }
      m_packLoadInProgress--;
      DEBUG_ASSERT(c.find(coord_) == c.end());
      c.insert(std::make_pair(coord_, obj));
      w.erase(coord_);
    }
    cond.notify_all();

    return obj;
  }
adofsauron commented 1 year ago

None of the cache-related data structures are thread-safe


  template <typename T>
  std::unordered_map<T, TraceableObjectPtr, T> &cache();

  template <typename T>
  std::unordered_set<T, T> &waitIO();

  template <typename T>
  std::condition_variable_any &condition();

  template <typename T>
  std::unordered_map<T, TraceableObjectPtr, T> const &cache() const;
adofsauron commented 1 year ago

The lockless queue of dpn


  // a tagged pointer, 16 bits as ref count.
  // Only read-only dpn uses it for ref counting; local dpn is managed only by
  // one write session
  std::atomic_ulong tagged_ptr;


  bool CAS(uint64_t &expected, uint64_t desired) { return tagged_ptr.compare_exchange_weak(expected, desired); }


  uint64_t GetPackPtr() const { return tagged_ptr.load(); }
  void SetPackPtr(uint64_t v) { tagged_ptr.store(v); }
adofsauron commented 1 year ago

Stack where an error occurs when performing multithreaded parallel aggregation:


(gdb) bt
#0  0x0000000002d225ca in Tianmu::mm::TraceableObject::IsLocked (this=0x0) at /root/work/stonedb-dev-20230103/storage/tianmu/mm/traceable_object.h:91
#1  0x0000000002d23644 in Tianmu::core::TianmuAttr::GetNotNullValueInt64 (this=0x7fd3309ee280, obj=497399) at /root/work/stonedb-dev-20230103/storage/tianmu/core/tianmu_attr.h:141
#2  0x0000000002e02551 in Tianmu::vcolumn::SingleColumn::GetNotNullValueInt64 (this=0x7fd330a08880, mit=...) at /root/work/stonedb-dev-20230103/storage/tianmu/vc/single_column.h:63
#3  0x0000000002f89741 in Tianmu::core::ColumnBinEncoder::EncoderInt::Encode (this=0x7fd328404b80, buf=0x7fd3280766aa "\022Brand#32\t", buf_sec=0x7fd328076660 "v\200\062\375\a", vc=0x7fd330a08880, 
    mit=..., update_stats=false) at /root/work/stonedb-dev-20230103/storage/tianmu/core/column_bin_encoder.cpp:326
#4  0x0000000002f88bf3 in Tianmu::core::ColumnBinEncoder::Encode (this=0x7fd328404b40, buf=0x7fd328076660 "v\200\062\375\a", mit=..., alternative_vc=0x0, update_stats=false)
    at /root/work/stonedb-dev-20230103/storage/tianmu/core/column_bin_encoder.cpp:174
#5  0x0000000002f7357b in Tianmu::core::GroupTable::PutGroupingValue (this=0x7fd3284041e8, col=2, mit=...) at /root/work/stonedb-dev-20230103/storage/tianmu/core/group_table.h:85
#6  0x0000000002f738ed in Tianmu::core::GroupByWrapper::PutGroupingValue (this=0x7fd328404120, gr_a=2, mit=...) at /root/work/stonedb-dev-20230103/storage/tianmu/core/groupby_wrapper.h:82
#7  0x0000000002f7069c in Tianmu::core::AggregationAlgorithm::AggregatePackrow (this=0x7fd51479a570, gbw=..., mit=0x7fd489fe9e20, cur_tuple=294414)
    at /root/work/stonedb-dev-20230103/storage/tianmu/core/aggregation_algorithm.cpp:584
#8  0x0000000002f71702 in Tianmu::core::AggregationWorkerEnt::TaskAggrePacks (this=0x7fd514799e20, taskIterator=0x7fd330dad038, dims=0x7fd514799bb0, mit=0x7fd514799e70, task=0x7fd330dab5f8, 
    gbw=0x7fd328404120, ci=0x7fd3309e9c80) at /root/work/stonedb-dev-20230103/storage/tianmu/core/aggregation_algorithm.cpp:868
#9  0x0000000002f81f8d in _ZSt13__invoke_implIvRMN6Tianmu4core20AggregationWorkerEntEFvPNS1_10MIIteratorEPNS1_15DimensionVectorES4_PNS1_5CTaskEPNS1_14GroupByWrapperEPNS1_11TransactionEERPS2_IRS4_RS6_SI_RS8_RSA_RSC_EET_St21__invoke_memfun_derefOT0_OT1_DpOT2_ (__f=
    @0x7fd330a27a58: (void (Tianmu::core::AggregationWorkerEnt::*)(Tianmu::core::AggregationWorkerEnt * const, Tianmu::core::MIIterator *, Tianmu::core::DimensionVector *, Tianmu::core::MIIterator *, Tianmu::core::CTask *, Tianmu::core::GroupByWrapper *, Tianmu::core::Transaction *)) 0x2f71614 <Tianmu::core::AggregationWorkerEnt::TaskAggrePacks(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>, __t=@0x7fd330a27a98: 0x7fd514799e20) at /usr/include/c++/8/bits/invoke.h:73
#10 0x0000000002f81b14 in std::__invoke<void (Tianmu::core::AggregationWorkerEnt::*&)(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*), Tianmu::core::AggregationWorkerEnt*&, Tianmu::core::MIIterator*&, Tianmu::core::DimensionVector*&, Tianmu::core::MIIterator*&, Tianmu::core::CTask*&, Tianmu::core::GroupByWrapper*&, Tianmu::core::Transaction*&> (__fn=
    @0x7fd330a27a58: (void (Tianmu::core::AggregationWorkerEnt::*)(Tianmu::core::AggregationWorkerEnt * const, Tianmu::core::MIIterator *, Tianmu::core::DimensionVector *, Tianmu::core::MIIterator *, Tianmu::core::CTask *, Tianmu::core::GroupByWrapper *, Tianmu::core::Transaction *)) 0x2f71614 <Tianmu::core::AggregationWorkerEnt::TaskAggrePacks(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>) at /usr/include/c++/8/bits/invoke.h:95
#11 0x0000000002f81557 in std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>::__call<void, , 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul>) (this=0x7fd330a27a58, 
    __args=empty std::tuple) at /usr/include/c++/8/functional:400
#12 0x0000000002f808d4 in std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>::operator()<, void>() (this=0x7fd330a27a58) at /usr/include/c++/8/functional:484
#13 0x0000000002f80087 in std::__invoke_impl<void, std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>&>(std::__invoke_other, std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#14 0x0000000002f7f982 in std::__invoke<std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>&>(std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#15 0x0000000002f7f301 in std::__future_base::_Task_state<std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}::operator()() const (this=0x7fd330a27a30)
    at /usr/include/c++/8/future:1421
--Type <RET> for more, q to quit, c to continue without paging--
#16 0x0000000002f8097b in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>::operator()() const (this=0x7fd489fea590) at /usr/include/c++/8/future:1362
#17 0x0000000002f80106 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/8/bits/std_function.h:283
#18 0x0000000002c417c9 in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (this=0x7fd489fea590)
    at /usr/include/c++/8/bits/std_function.h:687
#19 0x0000000002c3b3c1 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (
    this=0x7fd330a27a30, __f=0x7fd489fea590, __did_set=0x7fd489fea4f7) at /usr/include/c++/8/future:561
#20 0x0000000002c50ac3 in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2<std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>*> (__f=
    @0x7fd489fea510: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x2c3b39a <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __t=@0x7fd489fea508: 0x7fd330a27a30) at /usr/include/c++/8/bits/invoke.h:73
#21 0x0000000002c49033 in std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=
    @0x7fd489fea510: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x2c3b39a <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>) at /usr/include/c++/8/bits/invoke.h:95
#22 0x0000000002c413c6 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (this=0x7fd489fea480) at /usr/include/c++/8/mutex:672
#23 0x0000000002c41431 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (this=0x0) at /usr/include/c++/8/mutex:677
#24 0x0000000002c41442 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () at /usr/include/c++/8/mutex:677
#25 0x00007fd51f0cbe67 in __pthread_once_slow () from /lib64/libpthread.so.0
#26 0x0000000002c35ae5 in __gthread_once (__once=0x7fd330a27a48, __func=0x7fd51f39db90 <__once_proxy>) at /usr/include/c++/8/x86_64-redhat-linux/bits/gthr-default.h:699
#27 0x0000000002c414ec in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__once=..., __f=
    @0x7fd489fea510: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x2c3b39a <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool--Type <RET> for more, q to quit, c to continue without paging--
*)>) at /usr/include/c++/8/mutex:684
#28 0x0000000002c3b009 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (
    this=0x7fd330a27a30, __res=..., __ignore_failure=false) at /usr/include/c++/8/future:401
#29 0x0000000002f7f360 in std::__future_base::_Task_state<std::_Bind<void (Tianmu::core::AggregationWorkerEnt::*(Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*))(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*)>, std::allocator<int>, void ()>::_M_run() (this=0x7fd330a27a30) at /usr/include/c++/8/future:1423
#30 0x0000000002c4e07f in std::packaged_task<void ()>::operator()() (this=0x7fd330d7f380) at /usr/include/c++/8/future:1556
#31 0x0000000002f752b4 in Tianmu::utils::thread_pool::add_task<void (Tianmu::core::AggregationWorkerEnt::*)(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*), Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*&, Tianmu::core::Transaction*&>(void (Tianmu::core::AggregationWorkerEnt::*&&)(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*), Tianmu::core::AggregationWorkerEnt*&&, Tianmu::core::MIIterator*&&, Tianmu::core::DimensionVector*&&, Tianmu::core::MIIterator*&&, Tianmu::core::CTask*&&, Tianmu::core::GroupByWrapper*&, Tianmu::core::Transaction*&)::{lambda()#1}::operator()() const (this=0x7fd330d7ff80)
    at /root/work/stonedb-dev-20230103/storage/tianmu/util/thread_pool.h:94
#32 0x0000000002f7b255 in std::_Function_handler<void (), Tianmu::utils::thread_pool::add_task<void (Tianmu::core::AggregationWorkerEnt::*)(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*), Tianmu::core::AggregationWorkerEnt*, Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*&, Tianmu::core::Transaction*&>(void (Tianmu::core::AggregationWorkerEnt::*&&)(Tianmu::core::MIIterator*, Tianmu::core::DimensionVector*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::GroupByWrapper*, Tianmu::core::Transaction*), Tianmu::core::AggregationWorkerEnt*&&, Tianmu::core::MIIterator*&&, Tianmu::core::DimensionVector*&&, Tianmu::core::MIIterator*&&, Tianmu::core::CTask*&&, Tianmu::core::GroupByWrapper*&, Tianmu::core::Transaction*&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/8/bits/std_function.h:297
#33 0x0000000002c402ae in std::function<void ()>::operator()() const (this=0x7fd489fea6a0) at /usr/include/c++/8/bits/std_function.h:687
#34 0x0000000002c3b86b in Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}::operator()() const
    (__closure=0x6c48978) at /root/work/stonedb-dev-20230103/storage/tianmu/util/thread_pool.h:61
#35 0x0000000002c5f6bc in std::__invoke_impl<void, Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}>(std::__invoke_other, Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}&&) (__f=...)
    at /usr/include/c++/8/bits/invoke.h:60
#36 0x0000000002c58acb in std::__invoke<Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}>(std::__invoke_result&&, (Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}&&)...) (__fn=...)
    at /usr/include/c++/8/bits/invoke.h:95
#37 0x0000000002c6e024 in std::thread::_Invoker<std::tuple<Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x6c48978) at /usr/include/c++/8/thread:244
#38 0x0000000002c6d979 in std::thread::_Invoker<std::tuple<Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}> >::operator()() (this=0x6c48978) at /usr/include/c++/8/thread:253
#39 0x0000000002c6cab0 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}> > >::_M_run() (this=0x6c48970) at /usr/include/c++/8/thread:196
#40 0x00007fd51f39eb13 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#41 0x00007fd51f0c41ca in start_thread () from /lib64/libpthread.so.0
#42 0x00007fd51c214e73 in clone () from /lib64/libc.so.6
adofsauron commented 1 year ago

In a simpler scenario, for a nested loop where multiple threads are removed, the stack in question is as follows:


(gdb) bt
#0  0x00007f20882e7acf in raise () from /lib64/libc.so.6
#1  0x00007f20882baea5 in abort () from /lib64/libc.so.6
#2  0x00007f20882bad79 in __assert_fail_base.cold.0 () from /lib64/libc.so.6
#3  0x00007f20882e0426 in __assert_fail () from /lib64/libc.so.6
#4  0x000000000308e2f1 in Tianmu::core::IndexTable::Set64 (this=0x7f1e809d9ab0, n=2113, val=1379842) at /root/work/stonedb-dev-20230103/storage/tianmu/core/index_table.h:41
#5  0x000000000308d3f2 in Tianmu::core::IndexTable::SetByFilter (this=0x7f1e809d9ab0, f=0x7f1e80a09ec0, power=16) at /root/work/stonedb-dev-20230103/storage/tianmu/core/index_table.cpp:159
#6  0x00000000030ace99 in Tianmu::core::MINewContents::DisableOptimized (this=0x7f2080857370) at /root/work/stonedb-dev-20230103/storage/tianmu/core/mi_new_contents.cpp:247
#7  0x00000000030ad133 in Tianmu::core::MINewContents::CommitNewTableValues (this=0x7f2080857370) at /root/work/stonedb-dev-20230103/storage/tianmu/core/mi_new_contents.cpp:283
#8  0x00000000030919b5 in Tianmu::core::JoinerGeneral::ExecuteInnerJoinPackRow (this=0x7f1e809df0e0, mii=0x7f1ff5fe9cd0, task=0x7f1e80a0d4f8, cond=0x7f1ff5fe9db0, new_mind=0x7f2080857370, 
    all_dims=0x7f2080857550, _pack_desc_locked=0x7f1ff5fe9dd0, tuples_in_output=0x7f2080857368, limit=-1, count_only=false, stop_execution=0x7f20808571ff, rows_passed=0x7f2080857208, 
    rows_omitted=0x7f2080857200) at /root/work/stonedb-dev-20230103/storage/tianmu/core/joiner_general.cpp:154
#9  0x0000000003091c49 in Tianmu::core::JoinerGeneral::TaskInnerJoinPacks (this=0x7f1e809df0e0, taskIterator=0x7f1e80a0a468, task=0x7f1e80a0d4f8, cond=0x7f2080857680, new_mind=0x7f2080857370, 
    all_dims=0x7f2080857550, pack_desc_locked_p=0x7f2080857520, tuples_in_output=0x7f2080857368, limit=-1, count_only=false, stop_execution=0x7f20808571ff, rows_passed=0x7f2080857208, 
    rows_omitted=0x7f2080857200) at /root/work/stonedb-dev-20230103/storage/tianmu/core/joiner_general.cpp:195
#10 0x0000000003098ae9 in std::__invoke_impl<void, void (Tianmu::core::JoinerGeneral::*&)(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*), Tianmu::core::JoinerGeneral<Tianmu::core::MIIterator*&, Tianmu::core::CTask*&, Tianmu::core::Condition*&, Tianmu::core::MINewContents*&, Tianmu::core::DimensionVector*&, std::vector<bool, std::allocator<bool> >*&, long*&, long&, bool&, bool*&, bool&, bool&>*&> (__f=
    @0x7f1e80a0a738: (void (Tianmu::core::JoinerGeneral::*)(Tianmu::core::JoinerGeneral * const, Tianmu::core::MIIterator *, Tianmu::core::CTask *, Tianmu::core::Condition *, Tianmu::core::MINewContents *, Tianmu::core::DimensionVector *, std::vector<bool, std::allocator<bool> > *, long *, long, bool, bool *, long *, long *)) 0x3091ac6 <Tianmu::core::JoinerGeneral::TaskInnerJoinPacks(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>, __t=@0x7f1e80a0a7a8: 0x7f1e809df0e0) at /usr/include/c++/8/bits/invoke.h:73
#11 0x00000000030987e8 in std::__invoke<void (Tianmu::core::JoinerGeneral::*&)(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*), Tianmu::core::JoinerGeneral*&, Tianmu::core::MIIterator*&, Tianmu::core::CTask*&, Tianmu::core::Condition*&, Tianmu::core::MINewContents*&, Tianmu::core::DimensionVector*&, std::vector<bool, std::allocator<bool> >*&, long*&, long&, bool&, bool*&, long*&, long*&> (__fn=
    @0x7f1e80a0a738: (void (Tianmu::core::JoinerGeneral::*)(Tianmu::core::JoinerGeneral * const, Tianmu::core::MIIterator *, Tianmu::core::CTask *, Tianmu::core::Condition *, Tianmu::core::MINewContents *, Tianmu::core::DimensionVector *, std::vector<bool, std::allocator<bool> > *, long *, long, bool, bool *, long *, long *)) 0x3091ac6 <Tianmu::core::JoinerGeneral::TaskInnerJoinPacks(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>)
    at /usr/include/c++/8/bits/invoke.h:95
#12 0x000000000309838b in std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>::__call<void, , 0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul>(std::tuple<>&&, std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul, 5ul, 6ul, 7ul, 8ul, 9ul, 10ul, 11ul, 12ul>) (this=0x7f1e80a0a738, __args=empty std::tuple)
    at /usr/include/c++/8/functional:400
#13 0x0000000003097d84 in std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>::operator()<, void>() (this=0x7f1e80a0a738)
    at /usr/include/c++/8/functional:484
#14 0x0000000003097ab9 in std::__invoke_impl<void, std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>&>(std::__invoke_other, std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>&) (__f=...) at /usr/include/c++/8/bits/invoke.h:60
#15 0x0000000003097870 in std::__invoke<std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Cond--Type <RET> for more, q to quit, c to continue without paging--
ition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>&>(std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>&) (__fn=...) at /usr/include/c++/8/bits/invoke.h:95
#16 0x0000000003097637 in std::__future_base::_Task_state<std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}::operator()() const (this=0x7f1e80a0a710) at /usr/include/c++/8/future:1421
#17 0x0000000003097e2b in std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void>::operator()() const (this=0x7f1ff5fea590)
    at /usr/include/c++/8/future:1362
#18 0x0000000003097b38 in std::_Function_handler<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> (), std::__future_base::_Task_setter<std::unique_ptr<std::__future_base::_Result<void>, std::__future_base::_Result_base::_Deleter>, std::__future_base::_Task_state<std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>, std::allocator<int>, void ()>::_M_run()::{lambda()#1}, void> >::_M_invoke(std::_Any_data const&) (__functor=...) at /usr/include/c++/8/bits/std_function.h:283
#19 0x0000000002c417c9 in std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>::operator()() const (this=0x7f1ff5fea590)
    at /usr/include/c++/8/bits/std_function.h:687
#20 0x0000000002c3b3c1 in std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*) (
    this=0x7f1e80a0a710, __f=0x7f1ff5fea590, __did_set=0x7f1ff5fea4f7) at /usr/include/c++/8/future:561
#21 0x0000000002c50ac3 in std::__invoke_impl<void, void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2<std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>*> (__f=
    @0x7f1ff5fea510: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x2c3b39a <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>, __t=@0x7f1ff5fea508: 0x7f1e80a0a710) at /usr/include/c++/8/bits/invoke.h:73
#22 0x0000000002c49033 in std::__invoke<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__fn=
    @0x7f1ff5fea510: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x2c3b39a <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>) at /usr/include/c++/8/bits/invoke.h:95
#23 0x0000000002c413c6 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#1}::operator()() const (this=0x7f1ff5fea480) at /usr/include/c++/8/mutex:672
#24 0x0000000002c41431 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::operator()() const (this=0x0) at /usr/include/c++/8/mutex:677
#25 0x0000000002c41442 in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::uni--Type <RET> for more, q to quit, c to continue without paging--
que_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&)::{lambda()#2}::_FUN() () at /usr/include/c++/8/mutex:677
#26 0x00007f208b189e67 in __pthread_once_slow () from /lib64/libpthread.so.0
#27 0x0000000002c35ae5 in __gthread_once (__once=0x7f1e80a0a728, __func=0x7f208b45bb90 <__once_proxy>) at /usr/include/c++/8/x86_64-redhat-linux/bits/gthr-default.h:699
#28 0x0000000002c414ec in std::call_once<void (std::__future_base::_State_baseV2::*)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*>(std::once_flag&, void (std::__future_base::_State_baseV2::*&&)(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*), std::__future_base::_State_baseV2*&&, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*&&, bool*&&) (__once=..., __f=
    @0x7f1ff5fea510: (void (std::__future_base::_State_baseV2::*)(std::__future_base::_State_baseV2 * const, std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter>()> *, bool *)) 0x2c3b39a <std::__future_base::_State_baseV2::_M_do_set(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>*, bool*)>) at /usr/include/c++/8/mutex:684
#29 0x0000000002c3b009 in std::__future_base::_State_baseV2::_M_set_result(std::function<std::unique_ptr<std::__future_base::_Result_base, std::__future_base::_Result_base::_Deleter> ()>, bool) (
    this=0x7f1e80a0a710, __res=..., __ignore_failure=false) at /usr/include/c++/8/future:401
#30 0x0000000003097696 in std::__future_base::_Task_state<std::_Bind<void (Tianmu::core::JoinerGeneral::*(Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*))(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*)>, std::allocator<int>, void ()>::_M_run() (this=0x7f1e80a0a710) at /usr/include/c++/8/future:1423
#31 0x0000000002c4e07f in std::packaged_task<void ()>::operator()() (this=0x7f1e80a187c0) at /usr/include/c++/8/future:1556
#32 0x00000000030941a0 in Tianmu::utils::thread_pool::add_task<void (Tianmu::core::JoinerGeneral::*)(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*), Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long&, bool&, bool*, long*, long*>(void (Tianmu::core::JoinerGeneral::*&&)(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*), Tianmu::core::JoinerGeneral*&&, Tianmu::core::MIIterator*&&, Tianmu::core::CTask*&&, Tianmu::core::Condition*&&, Tianmu::core::MINewContents*&&, Tianmu::core::DimensionVector*&&, std::vector<bool, std::allocator<bool> >*&&, long*&&, long&, bool&, bool*&&, long*&&, long*&&)::{lambda()#1}::operator()() const (this=0x7f1e80a1aff0)
    at /root/work/stonedb-dev-20230103/storage/tianmu/util/thread_pool.h:94
#33 0x0000000003095423 in std::_Function_handler<void (), Tianmu::utils::thread_pool::add_task<void (Tianmu::core::JoinerGeneral::*)(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*), Tianmu::core::JoinerGeneral*, Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long&, bool&, bool*, long*, long*>(void (Tianmu::core::JoinerGeneral::*&&)(Tianmu::core::MIIterator*, Tianmu::core::CTask*, Tianmu::core::Condition*, Tianmu::core::MINewContents*, Tianmu::core::DimensionVector*, std::vector<bool, std::allocator<bool> >*, long*, long, bool, bool*, long*, long*), Tianmu::core::JoinerGeneral*&&, Tianmu::core::MIIterator*&&, Tianmu::core::CTask*&&, Tianmu::core::Condition*&&, Tianmu::core::MINewContents*&&, Tianmu::core::DimensionVector*&&, std::vector<bool, std::allocator<bool> >*&&, long*&&, long&, bool&, bool*&&, long*&&, long*&&)::{lambda()#1}>::_M_invoke(std::_Any_data const&) (__functor=...)
    at /usr/include/c++/8/bits/std_function.h:297
#34 0x0000000002c402ae in std::function<void ()>::operator()() const (this=0x7f1ff5fea6a0) at /usr/include/c++/8/bits/std_function.h:687
#35 0x0000000002c3b86b in Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}::operator()() const
    (__closure=0x7c9a968) at /root/work/stonedb-dev-20230103/storage/tianmu/util/thread_pool.h:61
#36 0x0000000002c5f6bc in std::__invoke_impl<void, Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}>(std::__invoke_other, Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}&&) (__f=...)
    at /usr/include/c++/8/bits/invoke.h:60
#37 0x0000000002c58acb in std::__invoke<Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}>(std::__invoke_result&&, (Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}&&)...) (__fn=...)
    at /usr/include/c++/8/bits/invoke.h:95
#38 0x0000000002c6e024 in std::thread::_Invoker<std::tuple<Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}> >::_M_invoke<0ul>(std::_Index_tuple<0ul>) (this=0x7c9a968) at /usr/include/c++/8/thread:244
#39 0x0000000002c6d979 in std::thread::_Invoker<std::tuple<Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, unsigned long)::{lambda()#1}> >::operator()() (this=0x7c9a968) at /usr/include/c++/8/thread:253
#40 0x0000000002c6cab0 in std::thread::_State_impl<std::thread::_Invoker<std::tuple<Tianmu::utils::thread_pool::thread_pool(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >--Type <RET> for more, q to quit, c to continue without paging--
 const&, unsigned long)::{lambda()#1}> > >::_M_run() (this=0x7c9a960) at /usr/include/c++/8/thread:196
#41 0x00007f208b45cb13 in execute_native_thread_routine () from /lib64/libstdc++.so.6
#42 0x00007f208b1821ca in start_thread () from /lib64/libpthread.so.0
#43 0x00007f20882d2e73 in clone () from /lib64/libc.so.6
adofsauron commented 1 year ago

PACK with index 0 is not read


[2023-01-05 08:12:45.479481] [95440] [INFO] [aggregation_algorithm.cpp:897] MSG: TaskAggrePacks LockPack gr_a: 0 cur_pack: 0
[2023-01-05 08:12:45.479487] [95440] [INFO] [dpn.h:85] MSG: dpn::CAS packPtr: -1 dataAddress: 0
[2023-01-05 08:12:45.485302] [95440] [INFO] [aggregation_algorithm.cpp:897] MSG: TaskAggrePacks LockPack gr_a: 0 cur_pack: 1
[2023-01-05 08:12:45.485332] [95440] [INFO] [dpn.h:85] MSG: dpn::CAS packPtr: 0 dataAddress: 0
[2023-01-05 08:12:45.485339] [95440] [INFO] [dpn.h:85] MSG: dpn::CAS packPtr: -1 dataAddress: 46262
[2023-01-05 08:12:45.485344] [95440] [INFO] [dpn.h:85] MSG: dpn::CAS packPtr: 2013273776 dataAddress: 46262
[2023-01-05 08:12:45.485348] [95440] [INFO] [tianmu_attr.cpp:800] MSG: TianmuAttr::LockPackForUse PACK_INDEX: 1 dpn::packPtr: 0x17f9478001eb0 dpn::dataAddress: 46262


(gdb) p task[0]
$5 = {
  dwTaskId = 0, 
  dwPackNum = 7, 
  dwEndPackno = 6, 
  dwTuple = 0, 
  dwStartPackno = 0, 
  dwStartTuple = 0, 
  dwEndTuple = 393215, 
  dwPack2cur = 0x7f9635e5eb30
}
adofsauron commented 1 year ago

But when read, the thread's dpn still loses the pointer to pack


(gdb) p dpn.tagged_ptr
$5 = {
  <std::__atomic_base<unsigned long>> = {
    static _S_alignment = 8, 
    _M_i = 0
  }, 
  members of std::atomic<unsigned long>: 
  static is_always_lock_free = true
}


[2023-01-05 08:27:15.593718] [98186] [INFO] [aggregation_algorithm.cpp:898] MSG: TaskAggrePacks LockPack gr_a: 0 cur_pack: 0
[2023-01-05 08:27:15.593725] [98186] [INFO] [dpn.h:85] MSG: dpn::CAS packPtr: -1 dataAddress: 0
[2023-01-05 08:27:15.599016] [98186] [INFO] [dpn.h:85] MSG: dpn::CAS packPtr: -1 dataAddress: 0
[2023-01-05 08:27:15.599062] [98186] [INFO] [dpn.h:85] MSG: dpn::CAS packPtr: -1073734160 dataAddress: 0
[2023-01-05 08:27:15.599070] [98186] [INFO] [tianmu_attr.cpp:800] MSG: TianmuAttr::LockPackForUse PACK_INDEX: 0 dpn::packPtr: 0x17f06c0001df0 dpn::dataAddress: 0
adofsauron commented 1 year ago

The reason is that lock and unlock for DPN have problems with threading, and unlock for dpn is refactoring for parallel processing

adofsauron commented 1 year ago

Single thread aggregation time:


[2023-01-06 03:43:11.033576] [109123] [INFO] [aggregation_algorithm.cpp:326] MSG: AggregatePackrow sin thread spend: 2.220218

Multithreaded aggregation takes time:


[2023-01-06 03:36:45.706681] [106499] [INFO] [aggregation_algorithm.cpp:904] MSG: TaskAggrePacks start: 0 end: 23
[2023-01-06 03:36:45.706829] [106508] [INFO] [aggregation_algorithm.cpp:904] MSG: TaskAggrePacks start: 96 end: 123
[2023-01-06 03:36:45.707114] [106498] [INFO] [aggregation_algorithm.cpp:904] MSG: TaskAggrePacks start: 24 end: 47
[2023-01-06 03:36:45.707368] [106511] [INFO] [aggregation_algorithm.cpp:904] MSG: TaskAggrePacks start: 48 end: 71
[2023-01-06 03:36:45.707550] [106506] [INFO] [aggregation_algorithm.cpp:904] MSG: TaskAggrePacks start: 72 end: 95
[2023-01-06 03:36:46.291094] [106598] [INFO] [aggregation_algorithm.cpp:1052] MSG: DistributeAggreTaskAverage spend: 0.619852
adofsauron commented 1 year ago

Contrast multithreaded aggregation with single-threaded aggregation

cf48bb891ad575b16a27167b0b0c533a


    select
        c_name,
        c_custkey,
        o_orderkey,
        o_orderdate,
        o_totalprice,
        sum(l_quantity)
    from
        customer,
        orders,
        lineitem
    where
        c_custkey = o_custkey
        and o_orderkey = l_orderkey
    group by
        c_name,
        c_custkey,
        o_orderkey,
        o_orderdate,
        o_totalprice
    order by
        o_totalprice desc,
        o_orderdate
    limit 10;


[2023-01-06 05:41:43.751620] [144166] [INFO] [aggregation_algorithm.cpp:329] MSG: AggregatePackrow thread_type: multi spend: 33.770813


[2023-01-06 05:48:01.563733] [147230] [INFO] [aggregation_algorithm.cpp:329] MSG: AggregatePackrow thread_type: sin spend: 78.494194


mysql> select
    -> c_name,
    -> c_custkey,
    -> o_orderkey,
    -> o_orderdate,
    -> o_totalprice,
    -> sum(l_quantity)
    -> from
    -> customer,
    -> orders,
    -> lineitem
    -> where
    -> c_custkey = o_custkey
    -> and o_orderkey = l_orderkey
    -> group by
    -> c_name,
    -> c_custkey,
    -> o_orderkey,
    -> o_orderdate,
    -> o_totalprice
    -> order by
    -> o_totalprice desc,
    -> o_orderdate
    -> limit 10;
+--------------------+-----------+------------+-------------+--------------+-----------------+
| c_name             | c_custkey | o_orderkey | o_orderdate | o_totalprice | sum(l_quantity) |
+--------------------+-----------+------------+-------------+--------------+-----------------+
| Customer#000513169 |    513169 |   39394405 | 1992-01-04  |    558822.56 |          297.00 |
| Customer#001287812 |   1287812 |   42290181 | 1997-11-26  |    558289.17 |          318.00 |
| Customer#001172513 |   1172513 |   36667107 | 1997-06-06  |    550142.18 |          322.00 |
| Customer#000399481 |    399481 |   43906817 | 1995-04-06  |    549431.65 |          312.00 |
| Customer#000984557 |    984557 |   49667013 | 1995-12-05  |    542546.72 |          296.00 |
| Customer#000630869 |    630869 |   30671170 | 1993-09-30  |    541620.62 |          297.00 |
| Customer#001492954 |   1492954 |   30332516 | 1996-03-10  |    541181.80 |          310.00 |
| Customer#001082018 |   1082018 |   31018979 | 1995-12-06  |    537993.05 |          304.00 |
| Customer#001114039 |   1114039 |   30417318 | 1995-10-31  |    536420.39 |          305.00 |
| Customer#001398991 |   1398991 |   57874788 | 1994-01-19  |    536235.24 |          295.00 |
+--------------------+-----------+------------+-------------+--------------+-----------------+
10 rows in set (2 min 23.10 sec)


mysql> select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem where c_custkey = o_custkey and o_orderkey = l_orderkey group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate limit 10;
ERROR 2006 (HY000): MySQL server has gone away
No connection. Trying to reconnect...
Connection id:    3
Current database: tpch

+--------------------+-----------+------------+-------------+--------------+-----------------+
| c_name             | c_custkey | o_orderkey | o_orderdate | o_totalprice | sum(l_quantity) |
+--------------------+-----------+------------+-------------+--------------+-----------------+
| Customer#000513169 |    513169 |   39394405 | 1992-01-04  |    558822.56 |          297.00 |
| Customer#001287812 |   1287812 |   42290181 | 1997-11-26  |    558289.17 |          318.00 |
| Customer#001172513 |   1172513 |   36667107 | 1997-06-06  |    550142.18 |          322.00 |
| Customer#000399481 |    399481 |   43906817 | 1995-04-06  |    549431.65 |          312.00 |
| Customer#000984557 |    984557 |   49667013 | 1995-12-05  |    542546.72 |          296.00 |
| Customer#000630869 |    630869 |   30671170 | 1993-09-30  |    541620.62 |          297.00 |
| Customer#001492954 |   1492954 |   30332516 | 1996-03-10  |    541181.80 |          310.00 |
| Customer#001082018 |   1082018 |   31018979 | 1995-12-06  |    537993.05 |          304.00 |
| Customer#001114039 |   1114039 |   30417318 | 1995-10-31  |    536420.39 |          305.00 |
| Customer#001398991 |   1398991 |   57874788 | 1994-01-19  |    536235.24 |          295.00 |
+--------------------+-----------+------------+-------------+--------------+-----------------+
10 rows in set (4 min 4.74 sec)
adofsauron commented 1 year ago

A new thread switch frees the 101 indexed pack


[2023-01-09 01:36:20.189828] [12072] [DEBUG] [aggregation_algorithm.cpp:606] MSG: AggregatePackrow PutGroupingValue gr_a: 0 cur_pack: 12
[2023-01-09 01:36:20.189834] [12072] [DEBUG] [aggregation_algorithm.cpp:606] MSG: AggregatePackrow PutGroupingValue gr_a: 1 cur_pack: 128
[2023-01-09 01:36:20.189839] [12072] [DEBUG] [aggregation_algorithm.cpp:606] MSG: AggregatePackrow PutGroupingValue gr_a: 2 cur_pack: 101
[2023-01-09 01:36:20.189843] [12072] [DEBUG] [aggregation_algorithm.cpp:606] MSG: AggregatePackrow PutGroupingValue gr_a: 3 cur_pack: 0
[2023-01-09 01:36:20.189847] [12072] [DEBUG] [aggregation_algorithm.cpp:606] MSG: AggregatePackrow PutGroupingValue gr_a: 4 cur_pack: 288
[2023-01-09 01:36:20.189852] [12072] [DEBUG] [aggregation_algorithm.cpp:642] MSG: AggregatePackrow PutAggregatedValue gr_a: 5 cur_pack: 0
[2023-01-09 01:36:20.189858] [12072] [DEBUG] [aggregation_algorithm.cpp:606] MSG: AggregatePackrow PutGroupingValue gr_a: 0 cur_pack: 12
[2023-01-09 01:36:20.189862] [12072] [DEBUG] [aggregation_algorithm.cpp:606] MSG: AggregatePackrow PutGroupingValue gr_a: 1 cur_pack: 128
[2023-01-09 01:36:20.189928] [12150] [DEBUG] [tianmu_attr.cpp:833] MSG: TianmuAttr::UnlockPackFromUse newv is zero PACK_INDEX: 12 dpn::packPtr: 0x17f74f02d8ab0 dpn::dataAddress: 14156076
[2023-01-09 01:36:20.189969] [12150] [DEBUG] [dpn.h:84] MSG: dpn::CAS packPtr: 0 dataAddress: 14156076
[2023-01-09 01:36:20.189983] [12150] [DEBUG] [tianmu_attr.cpp:833] MSG: TianmuAttr::UnlockPackFromUse newv is zero PACK_INDEX: 12 dpn::packPtr: 0x17f74f02d8df0 dpn::dataAddress: 384
[2023-01-09 01:36:20.189988] [12150] [DEBUG] [dpn.h:84] MSG: dpn::CAS packPtr: 0 dataAddress: 384
[2023-01-09 01:36:20.189996] [12150] [DEBUG] [tianmu_attr.cpp:833] MSG: TianmuAttr::UnlockPackFromUse newv is zero PACK_INDEX: 128 dpn::packPtr: 0x17f74f02d8fb0 dpn::dataAddress: 1444864
[2023-01-09 01:36:20.190010] [12150] [DEBUG] [dpn.h:84] MSG: dpn::CAS packPtr: 0 dataAddress: 1444864
[2023-01-09 01:36:20.190018] [12150] [DEBUG] [tianmu_attr.cpp:833] MSG: TianmuAttr::UnlockPackFromUse newv is zero PACK_INDEX: 128 dpn::packPtr: 0x17f74f02d91b0 dpn::dataAddress: 13638242
[2023-01-09 01:36:20.190022] [12150] [DEBUG] [dpn.h:84] MSG: dpn::CAS packPtr: 0 dataAddress: 13638242
[2023-01-09 01:36:20.190027] [12150] [DEBUG] [tianmu_attr.cpp:833] MSG: TianmuAttr::UnlockPackFromUse newv is zero PACK_INDEX: 128 dpn::packPtr: 0x17f74f02d9350 dpn::dataAddress: 27696931
[2023-01-09 01:36:20.190031] [12150] [DEBUG] [dpn.h:84] MSG: dpn::CAS packPtr: 0 dataAddress: 27696931
[2023-01-09 01:36:20.190038] [12150] [DEBUG] [tianmu_attr.cpp:833] MSG: TianmuAttr::UnlockPackFromUse newv is zero PACK_INDEX: 101 dpn::packPtr: 0x17f74f02d9610 dpn::dataAddress: 4993066
[2023-01-09 01:36:20.190042] [12150] [DEBUG] [dpn.h:84] MSG: dpn::CAS packPtr: 0 dataAddress: 4993066
[2023-01-09 01:36:20.190065] [12072] [DEBUG] [aggregation_algorithm.cpp:606] MSG: AggregatePackrow PutGroupingValue gr_a: 2 cur_pack: 101
adofsauron commented 1 year ago

The current pack elimination policy is LOCK_ONE, which causes the pack address to be lost during multithreading. A new LOCK_ALL policy is designed to ensure that multithreaded aggregation works properly. Design idea:

  1. During multithreaded aggregation operation, the held pack is not eliminated
  2. Keep the original logic for reading and releasing packs image
adofsauron commented 1 year ago

Single-thread traversal computation time


[2023-01-12 05:47:55.951686] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 0 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:47:57.345679] [168551] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 1 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:47:58.573738] [168548] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 2 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:47:59.979975] [168547] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 3 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:01.302163] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 4 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:02.617330] [168552] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 5 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:03.946821] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 6 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:05.313356] [168560] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 7 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:06.580501] [168559] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 8 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:08.035818] [168552] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 9 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:09.433728] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 10 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:10.784532] [168553] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 11 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:12.026133] [168557] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 12 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:13.482009] [168556] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 13 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:14.873938] [168554] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 14 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:16.248055] [168546] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 15 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:17.672838] [168547] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 16 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:19.114565] [168556] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 17 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:20.556304] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 18 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:22.119459] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 19 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:23.519644] [168546] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 20 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:24.978950] [168559] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 21 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:26.394068] [168554] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 22 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:27.833264] [168554] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 23 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:29.230550] [168554] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 24 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:30.670639] [168550] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 25 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:32.110078] [168559] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 26 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:33.479022] [168560] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 27 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:34.967799] [168556] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 28 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:36.473401] [168554] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 29 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:37.811346] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 30 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:39.267895] [168550] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 31 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:40.686730] [168557] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 32 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:42.076043] [168553] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 33 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:43.512703] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 34 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:44.918907] [168546] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 35 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:46.484076] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 36 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:47.917339] [168556] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 37 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:49.360511] [168556] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 38 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:50.886774] [168553] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 39 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:52.416647] [168550] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 40 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:53.993155] [168556] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 41 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:55.410135] [168553] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 42 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:57.049073] [168551] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 43 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:48:58.633501] [168548] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 44 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:00.036655] [168548] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 45 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:01.571126] [168552] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 46 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:03.167577] [168550] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 47 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:04.791314] [168551] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 48 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:06.241731] [168548] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 49 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:07.799417] [168546] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 50 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:09.277515] [168554] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 51 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:10.925557] [168553] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 52 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:12.432667] [168547] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 53 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:14.018427] [168555] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 54 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:15.609985] [168553] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 55 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:17.063569] [168549] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 56 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:18.622441] [168556] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 57 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:20.152561] [168547] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 58 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:21.721630] [168548] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 59 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:23.277473] [168554] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 60 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:24.772742] [168560] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 61 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:26.346577] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 62 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:27.786916] [168557] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 63 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:29.325500] [168554] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 64 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:30.915921] [168550] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 65 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:32.514797] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 66 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:34.069973] [168559] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 67 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:35.474975] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 68 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:37.040043] [168560] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 69 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:38.616808] [168549] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 70 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:40.119750] [168559] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 71 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:41.666460] [168560] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 72 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:43.271759] [168559] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 73 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:44.841819] [168547] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 74 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:46.370030] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 75 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:48.010857] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 76 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:49.536753] [168547] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 77 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:51.057163] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 78 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:52.743181] [168551] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 79 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:54.319031] [168551] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 80 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:55.833623] [168553] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 81 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:57.407706] [168561] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 82 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:49:58.902540] [168553] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 83 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:50:00.338193] [168546] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 84 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:50:01.993250] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 85 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:50:03.463402] [168549] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 86 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:50:04.950526] [168549] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 87 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:50:06.478262] [168559] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 88 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:50:08.233417] [168560] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 89 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:50:09.892527] [168552] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 90 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
[2023-01-12 05:50:11.558812] [168558] [DEBUG] [pack_guardian.cpp:210] MSG: LockPackrowOnLockAll cur_dim: 0 cur_pack: 91 col_index: 0 field_name: l_orderkey table_name: ./tpch/lineitem table_type: 0
adofsauron commented 1 year ago

Problems occur when iterators encounter predicate validation logic


#0  0x0000000002c95e2a in Tianmu::core::FilterOnesIterator::GetPackSizeLeft (this=0x7f3eac501750) at /root/work/stonedb-dev-20230110/storage/tianmu/core/filter_iterators.cpp:179
#1  0x000000000309822e in Tianmu::core::DimensionGroupFilter::DGFilterIterator::GetPackSizeLeft (this=0x7f3eac501740) at /root/work/stonedb-dev-20230110/storage/tianmu/core/dimension_group.h:192
#2  0x0000000002cf5f36 in Tianmu::core::MIIterator::InitNextPackrow (this=0x7f4161fe9110) at /root/work/stonedb-dev-20230110/storage/tianmu/core/mi_iterator.h:288
#3  0x0000000002fea559 in Tianmu::core::MIIterator::NextPackrow (this=0x7f4161fe9110) at /root/work/stonedb-dev-20230110/storage/tianmu/core/mi_iterator.cpp:386
#4  0x000000000302ffe3 in Tianmu::core::ParameterizedFilter::RoughUpdateMultiIndex (this=0x7f3ef0929080) at /root/work/stonedb-dev-20230110/storage/tianmu/core/parameterized_filter.cpp:385
#5  0x000000000303489c in Tianmu::core::ParameterizedFilter::UpdateMultiIndex (this=0x7f3ef0929080, count_only=false, limit=1)
    at /root/work/stonedb-dev-20230110/storage/tianmu/core/parameterized_filter.cpp:1232
#6  0x0000000002cf0d23 in Tianmu::core::TempTable::ProcessParameters (this=0x7f3ef0928f30, mit=..., alias=-2) at /root/work/stonedb-dev-20230110/storage/tianmu/core/temp_table.cpp:1775
#7  0x0000000002e0fe04 in Tianmu::vcolumn::SubSelectColumn::PrepareSubqResult (this=0x7f3ef08facb0, mit=..., exists_only=true)
    at /root/work/stonedb-dev-20230110/storage/tianmu/vc/subselect_column.cpp:388
#8  0x0000000002e106ac in Tianmu::vcolumn::SubSelectColumn::CheckExists (this=0x7f3ef08facb0, mit=...) at /root/work/stonedb-dev-20230110/storage/tianmu/vc/subselect_column.cpp:497
#9  0x0000000002fb6792 in Tianmu::core::Descriptor::CheckCondition (this=0x7f3eac4fbe50, mit=...) at /root/work/stonedb-dev-20230110/storage/tianmu/core/descriptor.cpp:1080


bool ParameterizedFilter::RoughUpdateMultiIndex() {
  MEASURE_FET("ParameterizedFilter::RoughUpdateMultiIndex(...)");

  bool is_nonempty = true;
  bool false_desc = false;

  for (uint i = 0; i < descriptors_.Size(); i++) {
    if (!descriptors_[i].done && descriptors_[i].IsInner() && (descriptors_[i].IsTrue()))
      descriptors_[i].done = true;

    if (descriptors_[i].IsFalse() && descriptors_[i].IsInner()) {
      rough_mind_->MakeDimensionEmpty();
      false_desc = true;
    }
  }

  // one-dimensional conditions
  if (!false_desc) {
    // init by previous values of mind_ (if any nontrivial)
    for (int i = 0; i < mind_->NumOfDimensions(); i++) {
      Filter *loc_f = mind_->GetFilter(i);
      rough_mind_->UpdateGlobalRoughFilter(i, loc_f);  // if the filter is nontrivial, then copy pack status
    }

    for (uint i = 0; i < descriptors_.Size(); i++) {
      if (descriptors_[i].done || descriptors_[i].IsDelayed() || !descriptors_[i].IsInner() ||
          descriptors_[i].GetJoinType() != DescriptorJoinType::DT_NON_JOIN)
        continue;

      DimensionVector dims(mind_->NumOfDimensions());
      descriptors_[i].DimensionUsed(dims);
      int dim = dims.GetOneDim();
      if (dim == -1)
        continue;
      common::RoughSetValue *rf = rough_mind_->GetLocalDescFilter(dim, i);  // rough filter for a descriptor
      descriptors_[i].ClearRoughValues();                                   // clear accumulated rough values
                                                                            // for descriptor
      MIIterator mit(mind_, dim, true);
      while (mit.IsValid()) {
        int p = mit.GetCurPackrow(dim);
        if (p >= 0 && rf[p] != common::RoughSetValue::RS_NONE)
          rf[p] = descriptors_[i].EvaluateRoughlyPack(mit);  // rough values are also accumulated inside

        mit.NextPackrow();

        if (mind_->m_conn->Killed())
          throw common::KilledException();
      }

      bool this_nonempty = rough_mind_->UpdateGlobalRoughFilter(dim, i);  // update the filter using local information
      is_nonempty = (is_nonempty && this_nonempty);
      descriptors_[i].UpdateVCStatistics();
      descriptors_[i].SimplifyAfterRoughAccumulate();  // simplify tree if there is a
                                                       // roughly trivial leaf
    }
  }

  // Recalculate all multidimensional dependencies only if there are 1-dim
  // descriptors_ which can benefit from the projection
  if (DimsWith1dimFilters())
    RoughMakeProjections();

  // Displaying statistics
  if (tianmu_control_.isOn() || mind_->m_conn->Explain()) {
    int pack_full = 0, pack_some = 0, pack_all = 0;
    tianmu_control_.lock(mind_->m_conn->GetThreadID()) << "Packs/packrows after KN evaluation:" << system::unlock;

    for (int dim = 0; dim < rough_mind_->NumOfDimensions(); dim++) {
      std::stringstream ss;
      pack_full = 0;
      pack_some = 0;
      pack_all = rough_mind_->NumOfPacks(dim);

      for (int b = 0; b < pack_all; b++) {
        if (rough_mind_->GetPackStatus(dim, b) == common::RoughSetValue::RS_ALL)
          pack_full++;
        else if (rough_mind_->GetPackStatus(dim, b) != common::RoughSetValue::RS_NONE)
          pack_some++;
      }

      ss << "(t" << dim << ") Pckrows: " << pack_all << ", susp. " << pack_some << " ("
         << pack_all - (pack_full + pack_some) << " empty " << pack_full
         << " full). Conditions: " << rough_mind_->NumOfConditions(dim);

      if (mind_->m_conn->Explain()) {
        mind_->m_conn->SetExplainMsg(ss.str());
      } else {
        tianmu_control_.lock(mind_->m_conn->GetThreadID()) << ss.str() << system::unlock;
      }
    }
  }

  return is_nonempty;
}


        if (types::RequiresUTFConversions((*cond)[i].GetCollation())) {
          if ((*cond)[i].CheckCondition_UTF(*mii) == false)
            loc_result = false;
        } else {
          if ((*cond)[i].CheckCondition(*mii) == false)
            loc_result = false;
        }
adofsauron commented 1 year ago

With a small amount of data, the total cost of multithreaded aggregation is higher than that of a single thread


MSG: AggregatePackrow thread_type: sin spend: 0.037590


MSG: AggregatePackrow thread_type: multi spend: 0.074052
adofsauron commented 1 year ago

Functional test:

  1. Check whether the results of single-threaded aggregation are consistent with those of multithreaded aggregation
  2. Pay attention! Only consistency between single-threaded aggregation and multithreaded aggregation is guaranteed, but consistency between multithreaded aggregation and innodb is not guaranteed

Performance test:

  1. In the log, check whether the data comparison between single thread and multithread of AggregatePackrow thread_type: %s spend: %f is consistent with the target
  2. When testing performance, ensure that the memory is sufficient and use only MEM instead of SWAP during running. Otherwise, the test result will be greatly affected
  3. Pay attention! There is no guarantee of any TPCH rate of performance improvement for a single SQL. Any SQL has multiple component units, and no commitment to any overall testing is made until all involved units have been processed
adofsauron commented 1 year ago

Overall workload:

  1. Add the configuration parameters for enabling multithreaded aggregation. You can enable or disable multithreaded aggregation by modifying the configuration parameters

  2. The aggregation table pack is divided as a whole

  3. Prepare data for an independent thread

  4. Separation of a single thread pack[start, end]

  5. Aggregate local data within a single thread

  6. Merge the aggregation results of multiple threads

  7. Dynamically calculate the cost of table sweeping and loss cost of multithreaded aggregation, and dynamically determine the threshold of multithreaded aggregation

  8. pack cache policy lock_all

  9. pack cache policy lock_lru

  10. Independent lru control unit

a. Provide mechanisms rather than policies. Analyze the users of pack, but do not make presets or fixed policies

b. Provide a preheating mechanism

c. Provide a mechanism to adjust element activity after access

d. Provide elimination strategies. The number of eliminations at a time can be controlled

e. Ensure thread-safe and can be accessed by multiple threads in parallel. Consider the parallel-Hashmap data structure

adofsauron commented 1 year ago

During the aggregation of multithreaded table scanning, a crash occurred when the LOCK_ONE policy caused the pack of other threads to be released during thread switching. And with LOCK_ONE, each new pack read releases the old pack held, which is also bad performance.

Consider designing the release of pack as an asynchronous strategy, with a separate thread doing the release of pack.

Core ideas:

  1. Transfer the release of column attributes, the synchronous release of pack by the TianmuAttr module, to a separate thread for asynchronous operation

  2. Read pack in TianmuAttr module is still synchronous operation

  3. In the DPN module, the address count of the existing atomic variable tagged_ptr is used as a rule to determine whether it is no longer used

  4. The release pack interface of the TianmuAttr module only gives the release address usage count to the tagged_ptr of DPN

  5. Create a separate pack release module named column_unlock for releasing unwanted packs in a held column

  6. column_unlock is a storage unit of the container with the column of tianmu_table. The life cycle of the storage unit is the same as that of column_share

  7. The column_unlock module obtains all information about the column, including DPN, from the column_share

  8. The life cycle of the column_unlock module is the same as that of the database service

a. Created when the database service is created and destroyed when the database service is destroyed

b. This is independent of the life cycle of an independent query thread

  1. After the column_unlock module is created, an independent internal thread pool is enabled. All DPNS of the column are traversed from the columns held in the internal container

a. The column_unlock module is read-only for DPN

b. column_unlock releases the corresponding pack only when it identifies that tagged_use in DPN is NOT_USE and the use count in tagged_ptr is 0

  1. Hold members in VCPackGuardian to record a accessed pack on this thread as a user's policy for controlling pack usage
adofsauron commented 1 year ago

Abstract:

Describes the configurable parameters provided by the aggregate multithreaded module to the user.

tianmu_groupby_parallel_degree

Explanation:

The parallelism degree of multithreaded aggregation, specifically, the number of threads executing group by multithreaded aggregation

Data type:

An unsigned positive integer, uint_t

Assignment interval:

[0,+uint_t)

Explanation:

  1. Assign value < =0: group by is executed using a single thread

  2. The upper limit is 2 times the number of physical cpu cores. If the upper limit exceeds this value, the upper limit is 2 times the number of physical cpu cores.

Scope scope:

Dynamically set in a client session:

  1. What is the global situation

  2. This parameter takes effect only for dynamically configured queries

Static configuration in my.cnf configuration file:

  1. Restart the mysqld service

tianmu_groupby_parallel_rows_minimum

Explanation:

Threshold for the minimum number of rows that can start executing a multithreaded group by thread. Only when the value is greater than this value and the number of threads after the tianmu_groupby_parallel_degree parameter is computed > The multithreaded group by 1 is executed.

Note: See the use of tianmu_groupby_parallel_degree

Data type:

It is an unsigned 64-bit positive integer uint64_t

Assignment interval:

[655360,+UINT64_MAX)

Explanation:

  1. The minimum value is 655360. If it is smaller than this value, set it to 655360

  2. The maximum value is an unsigned 64-bit positive integer

  3. If this value exceeds the number of rows in the group by table, it indicates that multithreaded aggregation is never executed and only single-thread aggregation is used. It is recommended that you use it based on the amount of data in your table

Scope scope:

Dynamically set in a client session:

  1. What is the global situation

  2. This parameter takes effect only for dynamically configured queries

Static configuration in my.cnf configuration file:

  1. Restart the mysqld service
adofsauron commented 1 year ago

Aggregate multithreaded table scanning

mysql

adofsauron commented 1 year ago

63DF5B9B-2855-42da-B8A0-2FB043A63ED1

adofsauron commented 1 year ago

Improve pack_guard's data protection with thread-local variables

https://www.akkadia.org/drepper/tls.pdf

https://en.wikipedia.org/wiki/Thread-local_storage

adofsauron commented 1 year ago

ABI changes to thread-local variables

d811355d87bc8d590236172522d78655

725e3dfe929993244c1c483c5eb6e679

adofsauron commented 1 year ago

It can only be used to modify the POD type, not the class type, because constructors and destructors cannot be called automatically. __thread can be used to modify global variables, static variables within a function, but not local variables of a function, or ordinary member variables of a class. In addition, variables can only be initialized with compile-time constants

adofsauron commented 1 year ago 
   The pthread_key_create() function shall fail if:

   EAGAIN The system lacked the necessary resources to create another thread-specific data key, or the system-imposed limit on the total number of keys  per  process  {PTHREAD_KEYS_MAX}  has
          been exceeded.

   ENOMEM Insufficient memory exists to create the key.

   The pthread_key_create() function shall not return an error code of [EINTR].
adofsauron commented 1 year ago

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=55800

adofsauron commented 1 year ago

on ubuntu20.04

5c10af8dd09ae811d59935b5895eeacb

adofsauron commented 1 year ago

6705ac7e5f267f10109a852a89741b30

adofsauron commented 1 year ago

https://gcc.gnu.org/onlinedocs/gcc/Optimize-Options.html

adofsauron commented 1 year ago

-O3 Optimize yet more. -O3 turns on all optimizations specified by -O2 and also turns on the following optimization flags:

-fgcse-after-reload -fipa-cp-clone -floop-interchange -floop-unroll-and-jam -fpeel-loops -fpredictive-commoning -fsplit-loops -fsplit-paths -ftree-loop-distribution -ftree-partial-pre -funswitch-loops -fvect-cost-model=dynamic -fversion-loops-for-strides

adofsauron commented 1 year ago

In general, MySQL optimizer has the following problems:

Optimization is local and can only be done on a query block by query block, although there is subquery -> semi-join/derived table is a cross-block transformation strategy, but it is fully solidified and only targeted at the simplest case (SPJ subquery). In addition, every block must be joined -&gt. group by -> order by such an optimization order

join ordering only supports left-deep tree, not bushy join, and greedy search algorithm is used, so the optimal plan cannot be guaranteed

The processing of predicates is imperfect, AND in some cases the extraction of common expr cannot be supported, such as (A AND B) OR (A AND C) => A AND (B OR C), may lead to poor execution efficiency

aggregation < -> The group by the join placement cannot be pushed to the lower part of the join

Filter cannot be pushed below group by (derived table can, subquery cannot)

adofsauron commented 1 year ago

The ascension of polymerization enters the third stage

  1. No longer dependent on existing access to data, with the right design goals to drive the right performance and results
  2. group and aggregation operations are separated to facilitate the use of a more streamlined data structure to carry the group, and facilitate the slicing of data
  3. The selection of group's data structure
  4. The selection of aggregated data structure
  5. Separation of threads and selection of critical sections
  6. The cache management of columns, as well as the cache of data switching, requires finer control for more refined parallel shards
adofsauron commented 1 year ago

creates a selection-vector, filled with positions of tuples that match our predicate. Then the Project operator is executed to calculate expressions needed for the final aggregation. Note that ”discount” and ”extendedprice” columns are not modified during selection. Instead, the selection-vector is taken into account by map-primitives to perform calculations only for relevant tuples, writing results at the same positions in the output vector as they were in the input one. This behavior requires propagating of the selection-vector to the final Aggr. There, for each tuple its position in the hash table is calculated, and then, using this data, aggregate results are updated. Additionally, for the new elements in the hash table, values of the grouping attribute are saved. The contents of the hash-table becomes available as the query result as soon as the underlying operators become exhausted and cannot produce more vectors.

adofsauron commented 1 year ago

1caaf885927a94d84024eb468e909d2c

adofsauron commented 1 year ago

a Table is a materialized relation, whereas a Dataflow just consists of tuples flowing through a pipeline

adofsauron commented 1 year ago

The primary reason for using the column-wise vector layout is not to optimize memory layout in the cache

adofsauron commented 1 year ago

In a vertically fragmented data model, the execution primitives only know about the columns they operate on without having to know about the overall table layout (e.g. record offsets).

adofsauron commented 1 year ago

For an aggregated group, this is a silly way to set up a hash. The group only generates intermediate passes and materializes the data based on projections, which leads to a copy of the data

f204d58a0ea7102264c9d7e517102cd8

adofsauron commented 1 year ago

But the bigger problem is the design of the API interface for accessing the stored data. To get the current use row must use a MIIterator's global structure


  void GetNotNullValueString(types::BString &s, const core::MIIterator &mit) override {
    col_->GetNotNullValueString(mit[dim_], s);
  }
adofsauron commented 1 year ago 

|    335 | C_7=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 11:int, 32:int);                                                                        |
|    651 | C_8=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 13:int, 32:int);                                                                        |
|   1255 | C_9=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 0:int, 32:int);                                                                         |
|    185 | X_10=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 27:int, 32:int);                                             |
|   1108 | X_12=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 30:int, 32:int);                                             |
|   1596 | C_13=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 4:int, 32:int);                                                                        |
|   1391 | C_14=[1874568]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 31:int, 32:int);                                                                       |
|    123 | C_11=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 27:int, 32:int);                                                                       |
|   2118 | X_15=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 6:int, 32:int);                                              |
|   2347 | X_16=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 5:int, 32:int);                                              |
|   2419 | C_17=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 9:int, 32:int);                                                                        |
|   2142 | C_18=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 29:int, 32:int);                                                                       |
|   1087 | X_19=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 25:int, 32:int);                                             |
|   2679 | C_20=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 8:int, 32:int);                                                                        |
|   1508 | C_21=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 26:int, 32:int);                                                                       |
|   2194 | C_22=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 28:int, 32:int);                                                                       |
|   3027 | C_23=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 12:int, 32:int);                                                                       |
|   3300 | C_24=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 7:int, 32:int);                                                                        |
|    629 | C_25=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 22:int, 32:int);                                                                       |
|   3407 | C_26=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 5:int, 32:int);                                                                        |
|   3400 | C_27=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 10:int, 32:int);                                                                       |
|   3472 | X_28=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 8:int, 32:int);                                              |
|   1515 | X_29=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 24:int, 32:int);                                             |
|     72 | C_30=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 18:int, 32:int);                                                                       |
|   3667 | X_31=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 11:int, 32:int);                                             |
|   3850 | X_32=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 29:int, 32:int);                                             |
|   4268 | C_33=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 2:int, 32:int);                                                                        |
|   4025 | X_34=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 3:int, 32:int);                                              |
|   1888 | C_35=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 24:int, 32:int);                                                                       |
|   1044 | C_36=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 19:int, 32:int);                                                                       |
|   4265 | C_37=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 30:int, 32:int);                                                                       |
|    496 | X_38=[1874564]:bat[:lng] := algebra.projection(C_18=[1874564]:bat[:oid], X_32=[1874564]:bat[:lng]);                                                              |
|   2084 | C_39=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 23:int, 32:int);                                                                       |
|   1643 | X_40=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 19:int, 32:int);                                             |
|   3216 | X_41=[1874564]:bat[:lng] := algebra.projection(C_11=[1874564]:bat[:oid], X_10=[1874564]:bat[:lng]);                                                              |
|     84 | X_42=[1874564]:bat[:lng] := algebra.projection(C_36=[1874564]:bat[:oid], X_40=[1874564]:bat[:lng]);                                                              |
|   3450 | C_43=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 25:int, 32:int);                                                                       |
|   3923 | X_44=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 26:int, 32:int);                                             |
|   2427 | C_45=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 21:int, 32:int);                                                                       |
|   5030 | X_46=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 28:int, 32:int);                                             |
|   5429 | X_47=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 17:int, 32:int);                                             |
|   5679 | C_48=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 6:int, 32:int);                                                                        |
|   1683 | C_49=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 16:int, 32:int);                                                                       |
|   5896 | X_50=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 10:int, 32:int);                                             |
|   2236 | C_51=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 17:int, 32:int);                                                                       |
|    115 | X_52=[1874564]:bat[:lng] := algebra.projection(C_27=[1874564]:bat[:oid], X_50=[1874564]:bat[:lng]);                                                              |
|   6018 | X_53=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 16:int, 32:int);                                             |
|   2820 | X_54=[1874564]:bat[:lng] := algebra.projection(C_26=[1874564]:bat[:oid], X_16=[1874564]:bat[:lng]);                                                              |
|     64 | C_55=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 3:int, 32:int);                                                                        |
|   2702 | X_56=[1874564]:bat[:lng] := algebra.projection(C_20=[1874564]:bat[:oid], X_28=[1874564]:bat[:lng]);                                                              |
|   2028 | C_57=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 15:int, 32:int);                                                                       |
|   3336 | C_59=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 20:int, 32:int);                                                                       |
|   6497 | X_58=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 9:int, 32:int);                                              |
|   6822 | X_60=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 23:int, 32:int);                                             |
|   3428 | X_61=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 18:int, 32:int);                                             |
|   2824 | X_62=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 15:int, 32:int);                                             |
|    125 | X_63=[1874564]:bat[:lng] := algebra.projection(C_30=[1874564]:bat[:oid], X_61=[1874564]:bat[:lng]);                                                              |
|   2577 | X_64=[1874564]:bat[:lng] := algebra.projection(C_37=[1874564]:bat[:oid], X_12=[1874564]:bat[:lng]);                                                              |
|   2153 | X_65=[1874564]:bat[:lng] := algebra.projection(C_43=[1874564]:bat[:oid], X_19=[1874564]:bat[:lng]);                                                              |
|   1939 | X_66=[1874564]:bat[:lng] := algebra.projection(C_22=[1874564]:bat[:oid], X_46=[1874564]:bat[:lng]);                                                              |
|   4978 | X_67=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 22:int, 32:int);                                             |
|     89 | X_68=[1874564]:bat[:lng] := algebra.projection(C_25=[1874564]:bat[:oid], X_67=[1874564]:bat[:lng]);                                                              |
|   7844 | X_69=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 7:int, 32:int);                                              |
|    102 | X_70=[1874564]:bat[:lng] := algebra.projection(C_24=[1874564]:bat[:oid], X_69=[1874564]:bat[:lng]);                                                              |
|   6621 | X_71=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 12:int, 32:int);                                             |
|   2208 | C_72=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 14:int, 32:int);                                                                       |
|   1525 | X_73=[1874564]:bat[:lng] := algebra.projection(C_17=[1874564]:bat[:oid], X_58=[1874564]:bat[:lng]);                                                              |
|   1179 | X_74=[1874564]:bat[:lng] := algebra.projection(C_39=[1874564]:bat[:oid], X_60=[1874564]:bat[:lng]);                                                              |
|   8447 | X_75=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 4:int, 32:int);                                              |
|    133 | X_86=[1874564]:bat[:lng] := algebra.projection(C_13=[1874564]:bat[:oid], X_75=[1874564]:bat[:lng]);                                                              |
|   4331 | X_76=[1874564]:bat[:lng] := algebra.projection(C_35=[1874564]:bat[:oid], X_29=[1874564]:bat[:lng]);                                                              |
|   5507 | X_77=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 20:int, 32:int);                                             |
|   3626 | X_88=[1874564]:bat[:lng] := algebra.projection(C_48=[1874564]:bat[:oid], X_15=[1874564]:bat[:lng]);                                                              |
|   4152 | X_79=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 14:int, 32:int);                                             |
|   8516 | X_80=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 0:int, 32:int);                                              |
|    849 | X_81=[1874564]:bat[:lng] := algebra.projection(C_23=[1874564]:bat[:oid], X_71=[1874564]:bat[:lng]);                                                              |
|   2744 | X_82=[1874564]:bat[:lng] := algebra.projection(C_55=[1874564]:bat[:oid], X_34=[1874564]:bat[:lng]);                                                              |
|    239 | X_89=[1874564]:bat[:lng] := algebra.projection(C_59=[1874564]:bat[:oid], X_77=[1874564]:bat[:lng]);                                                              |
|   4021 | X_84=[1874564]:bat[:lng] := algebra.projection(C_21=[1874564]:bat[:oid], X_44=[1874564]:bat[:lng]);                                                              |
|   3390 | X_85=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 13:int, 32:int);                                             |
|   6531 | X_87=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 21:int, 32:int);                                             |
|   1640 | X_78=[1874564]:bat[:lng] := algebra.projection(C_57=[1874564]:bat[:oid], X_62=[1874564]:bat[:lng]);                                                              |
|   5294 | X_83=[1874564]:bat[:lng] := algebra.projection(C_7=[1874564]:bat[:oid], X_31=[1874564]:bat[:lng]);                                                               |
|    283 | X_90=[1874564]:bat[:lng] := algebra.projection(C_9=[1874564]:bat[:oid], X_80=[1874564]:bat[:lng]);                                                               |
|    127 | X_91=[1874564]:bat[:lng] := algebra.projection(C_45=[1874564]:bat[:oid], X_87=[1874564]:bat[:lng]);                                                              |
|    628 | X_92=[1874564]:bat[:lng] := algebra.projection(C_72=[1874564]:bat[:oid], X_79=[1874564]:bat[:lng]);                                                              |
|   2679 | X_93=[1874564]:bat[:lng] := algebra.projection(C_8=[1874564]:bat[:oid], X_85=[1874564]:bat[:lng]);                                                               |
|   7192 | X_94=[1874564]:bat[:bte] := algebra.project(X_92=[1874564]:bat[:lng], 1:bte);                                                                                    |
|   9662 | X_95=[1874564]:bat[:bte] := algebra.project(X_91=[1874564]:bat[:lng], 1:bte);                                                                                    |
|  10272 | X_96=[1874564]:bat[:bte] := algebra.project(X_81=[1874564]:bat[:lng], 1:bte);                                                                                    |
|   9795 | X_97=[1874564]:bat[:bte] := algebra.project(X_90=[1874564]:bat[:lng], 1:bte);                                                                                    |
|  10689 | X_98=[1874564]:bat[:bte] := algebra.project(X_82=[1874564]:bat[:lng], 1:bte);                                                                                    |
|    208 | C_99=[1874564]:bat[:oid] := sql.tid(X_2=0:int, "sys":str, "lineitem":str, 1:int, 32:int);                                                                        |
|    200 | X_100=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 1:int, 32:int);                                             |
|     84 | X_101=[1874564]:bat[:lng] := algebra.projection(C_99=[1874564]:bat[:oid], X_100=[1874564]:bat[:lng]);                                                            |
|  39324 | X_102=[1874564]:bat[:bte] := algebra.project(X_70=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  42428 | X_103=[1874564]:bat[:bte] := algebra.project(X_42=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  10736 | X_104=[1874564]:bat[:bte] := algebra.project(X_83=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  23150 | X_105=[1874564]:bat[:bte] := algebra.project(X_86=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  41657 | X_106=[1874564]:bat[:bte] := algebra.project(X_74=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  24698 | X_107=[1874564]:bat[:bte] := algebra.project(X_78=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  46618 | X_108=[1874564]:bat[:bte] := algebra.project(X_38=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  47915 | X_109=[1874564]:bat[:bte] := algebra.project(X_56=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  46437 | X_110=[1874564]:bat[:bte] := algebra.project(X_54=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  15868 | X_111=[1874564]:bat[:bte] := algebra.project(X_101=[1874564]:bat[:lng], 1:bte);                                                                                  |
|  46718 | X_112=[1874564]:bat[:bte] := algebra.project(X_66=[1874564]:bat[:lng], 1:bte);                                                                                   |
|    174 | X_113=[1874564]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 2:int, 32:int);                                             |
|    160 | X_114=[1874568]:bat[:lng] := sql.bind(X_2=0:int, "sys":str, "lineitem":str, "l_quantity":str, 0:int, 31:int, 32:int);                                            |
|     77 | X_115=[1874568]:bat[:lng] := algebra.projection(C_14=[1874568]:bat[:oid], X_114=[1874568]:bat[:lng]);                                                            |
|  61283 | X_116=[1874564]:bat[:bte] := algebra.project(X_63=[1874564]:bat[:lng], 1:bte);                                                                                   |
|    129 | X_117=[1874564]:bat[:lng] := algebra.projection(C_33=[1874564]:bat[:oid], X_113=[1874564]:bat[:lng]);                                                            |
|  42229 | X_118=[1874564]:bat[:bte] := algebra.project(X_93=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  41410 | X_119=[1874564]:bat[:bte] := algebra.project(X_76=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  64023 | X_120=[1874564]:bat[:bte] := algebra.project(X_73=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  47007 | X_121=[1874564]:bat[:bte] := algebra.project(X_88=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  70840 | X_122=[1874564]:bat[:bte] := algebra.project(X_64=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  71231 | X_123=[1874564]:bat[:bte] := algebra.project(X_65=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  53664 | X_124=[1874564]:bat[:bte] := algebra.project(X_84=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  77667 | X_125=[1874564]:bat[:bte] := algebra.project(X_41=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  13629 | X_126=[1874564]:bat[:bte] := algebra.project(X_117=[1874564]:bat[:lng], 1:bte);                                                                                  |
|  77028 | X_127=[1874564]:bat[:bte] := algebra.project(X_68=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  81403 | X_128=[1874564]:bat[:bte] := algebra.project(X_52=[1874564]:bat[:lng], 1:bte);                                                                                   |
|    257 | X_129=[1874564]:bat[:lng] := algebra.projection(C_51=[1874564]:bat[:oid], X_47=[1874564]:bat[:lng]);                                                             |
|  61878 | X_130=[1874564]:bat[:bte] := algebra.project(X_89=[1874564]:bat[:lng], 1:bte);                                                                                   |
|  21551 | X_131=[1874568]:bat[:bte] := algebra.project(X_115=[1874568]:bat[:lng], 1:bte);                                                                                  |
|   4596 | X_132=[1874564]:bat[:bte] := algebra.project(X_129=[1874564]:bat[:lng], 1:bte);                                                                                  |
|    144 | X_133=[1874564]:bat[:lng] := algebra.projection(C_49=[1874564]:bat[:oid], X_53=[1874564]:bat[:lng]);                                                             |
|   4987 | X_134=[1874564]:bat[:bte] := algebra.project(X_133=[1874564]:bat[:lng], 1:bte);                                                                                  |
| 340766 | (X_135=[1874564]:bat[:oid], C_136=[50]:bat[:oid]) := group.groupdone(X_92=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    173 | X_137=[50]:bat[:lng] := algebra.projection(C_136=[50]:bat[:oid], X_92=[1874564]:bat[:lng]); # project1_lng                                                       |
| 342860 | (X_138=[1874564]:bat[:oid], C_139=[50]:bat[:oid]) := group.groupdone(X_88=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    112 | X_140=[50]:bat[:lng] := algebra.projection(C_139=[50]:bat[:oid], X_88=[1874564]:bat[:lng]); # project1_lng                                                       |
|     19 | X_141=0@0:void := language.pass(X_92=[1874564]:bat[:lng]);                                                                                                       |
| 358619 | (X_142=[1874564]:bat[:oid], C_143=[50]:bat[:oid]) := group.groupdone(X_101=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                          |
|    154 | X_144=[50]:bat[:lng] := algebra.projection(C_143=[50]:bat[:oid], X_101=[1874564]:bat[:lng]); # project1_lng                                                      |
|     37 | X_145=0@0:void := language.pass(X_101=[1874564]:bat[:lng]);                                                                                                      |
| 380366 | (X_146=[1874564]:bat[:oid], C_147=[50]:bat[:oid]) := group.groupdone(X_91=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
| 411452 | (X_148=[1874564]:bat[:oid], C_149=[50]:bat[:oid]) := group.groupdone(X_56=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    140 | X_150=[50]:bat[:lng] := algebra.projection(C_147=[50]:bat[:oid], X_91=[1874564]:bat[:lng]); # project1_lng                                                       |
|    110 | X_151=[50]:bat[:lng] := algebra.projection(C_149=[50]:bat[:oid], X_56=[1874564]:bat[:lng]); # project1_lng                                                       |
|     25 | X_152=0@0:void := language.pass(X_88=[1874564]:bat[:lng]);                                                                                                       |
|     64 | X_153=0@0:void := language.pass(X_91=[1874564]:bat[:lng]);                                                                                                       |
| 412133 | (X_154=[1874564]:bat[:oid], C_155=[50]:bat[:oid]) := group.groupdone(X_54=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    109 | X_156=[50]:bat[:lng] := algebra.projection(C_155=[50]:bat[:oid], X_54=[1874564]:bat[:lng]); # project1_lng                                                       |
|     23 | X_157=0@0:void := language.pass(X_56=[1874564]:bat[:lng]);                                                                                                       |
|  47271 | X_158=[50]:bat[:lng] := aggr.subcount(X_94=[1874564]:bat[:bte], X_135=[1874564]:bat[:oid], C_136=[50]:bat[:oid], true:bit);                                      |
|     47 | X_159=0@0:void := language.pass(C_136=[50]:bat[:oid]);                                                                                                           |
| 360640 | (X_160=[1874568]:bat[:oid], C_161=[50]:bat[:oid]) := group.groupdone(X_115=[1874568]:bat[:lng]); # GRP_create_partial_hash_table, dense                          |
|    127 | X_162=[50]:bat[:lng] := algebra.projection(C_161=[50]:bat[:oid], X_115=[1874568]:bat[:lng]); # project1_lng                                                      |
|     25 | X_163=0@0:void := language.pass(X_54=[1874564]:bat[:lng]);                                                                                                       |
| 385898 | (X_164=[1874564]:bat[:oid], C_165=[50]:bat[:oid]) := group.groupdone(X_89=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    123 | X_166=[50]:bat[:lng] := algebra.projection(C_165=[50]:bat[:oid], X_89=[1874564]:bat[:lng]); # project1_lng                                                       |
| 384597 | (X_167=[1874564]:bat[:oid], C_168=[50]:bat[:oid]) := group.groupdone(X_84=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|     87 | X_169=[50]:bat[:lng] := algebra.projection(C_168=[50]:bat[:oid], X_84=[1874564]:bat[:lng]); # project1_lng                                                       |
|     58 | X_170=0@0:void := language.pass(X_115=[1874568]:bat[:lng]);                                                                                                      |
|     65 | X_171=0@0:void := language.pass(X_84=[1874564]:bat[:lng]);                                                                                                       |
|    193 | X_172=0@0:void := language.pass(X_89=[1874564]:bat[:lng]);                                                                                                       |
| 384955 | (X_173=[1874564]:bat[:oid], C_174=[50]:bat[:oid]) := group.groupdone(X_76=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|     97 | X_175=[50]:bat[:lng] := algebra.projection(C_174=[50]:bat[:oid], X_76=[1874564]:bat[:lng]); # project1_lng                                                       |
|     70 | X_176=0@0:void := language.pass(X_76=[1874564]:bat[:lng]);                                                                                                       |
| 356940 | (X_177=[1874564]:bat[:oid], C_178=[50]:bat[:oid]) := group.groupdone(X_63=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    122 | X_179=[50]:bat[:lng] := algebra.projection(C_178=[50]:bat[:oid], X_63=[1874564]:bat[:lng]); # project1_lng                                                       |
|     47 | X_180=0@0:void := language.pass(X_63=[1874564]:bat[:lng]);                                                                                                       |
|  47881 | X_181=[50]:bat[:lng] := aggr.subcount(X_121=[1874564]:bat[:bte], X_138=[1874564]:bat[:oid], C_139=[50]:bat[:oid], true:bit);                                     |
|     53 | X_182=0@0:void := language.pass(C_139=[50]:bat[:oid]);                                                                                                           |
| 380257 | (X_183=[1874564]:bat[:oid], C_184=[50]:bat[:oid]) := group.groupdone(X_117=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                          |
|    165 | X_185=[50]:bat[:lng] := algebra.projection(C_184=[50]:bat[:oid], X_117=[1874564]:bat[:lng]); # project1_lng                                                      |
|    119 | X_186=0@0:void := language.pass(X_117=[1874564]:bat[:lng]);                                                                                                      |
| 370617 | (X_187=[1874564]:bat[:oid], C_188=[50]:bat[:oid]) := group.groupdone(X_73=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|     98 | X_189=[50]:bat[:lng] := algebra.projection(C_188=[50]:bat[:oid], X_73=[1874564]:bat[:lng]); # project1_lng                                                       |
| 374248 | (X_190=[1874564]:bat[:oid], C_191=[50]:bat[:oid]) := group.groupdone(X_42=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    132 | X_192=[50]:bat[:lng] := algebra.projection(C_191=[50]:bat[:oid], X_42=[1874564]:bat[:lng]); # project1_lng                                                       |
|     69 | X_193=0@0:void := language.pass(X_73=[1874564]:bat[:lng]);                                                                                                       |
|     69 | X_194=0@0:void := language.pass(X_42=[1874564]:bat[:lng]);                                                                                                       |
| 455019 | (X_195=[1874564]:bat[:oid], C_196=[50]:bat[:oid]) := group.groupdone(X_38=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    113 | X_197=[50]:bat[:lng] := algebra.projection(C_196=[50]:bat[:oid], X_38=[1874564]:bat[:lng]); # project1_lng                                                       |
| 383036 | (X_198=[1874564]:bat[:oid], C_199=[50]:bat[:oid]) := group.groupdone(X_68=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    114 | X_200=[50]:bat[:lng] := algebra.projection(C_199=[50]:bat[:oid], X_68=[1874564]:bat[:lng]); # project1_lng                                                       |
| 381651 | (X_201=[1874564]:bat[:oid], C_202=[50]:bat[:oid]) := group.groupdone(X_70=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|     90 | X_203=0@0:void := language.pass(X_38=[1874564]:bat[:lng]);                                                                                                       |
|    152 | X_204=[50]:bat[:lng] := algebra.projection(C_202=[50]:bat[:oid], X_70=[1874564]:bat[:lng]); # project1_lng                                                       |
|     26 | X_205=0@0:void := language.pass(X_68=[1874564]:bat[:lng]);                                                                                                       |
|     83 | X_206=0@0:void := language.pass(X_70=[1874564]:bat[:lng]);                                                                                                       |
| 416045 | (X_207=[1874564]:bat[:oid], C_208=[50]:bat[:oid]) := group.groupdone(X_82=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    184 | X_209=[50]:bat[:lng] := algebra.projection(C_208=[50]:bat[:oid], X_82=[1874564]:bat[:lng]); # project1_lng                                                       |
|     70 | X_210=0@0:void := language.pass(X_82=[1874564]:bat[:lng]);                                                                                                       |
| 432649 | (X_211=[1874564]:bat[:oid], C_212=[50]:bat[:oid]) := group.groupdone(X_90=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
| 396271 | (X_213=[1874564]:bat[:oid], C_214=[50]:bat[:oid]) := group.groupdone(X_65=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    144 | X_215=[50]:bat[:lng] := algebra.projection(C_212=[50]:bat[:oid], X_90=[1874564]:bat[:lng]); # project1_lng                                                       |
|    100 | X_216=[50]:bat[:lng] := algebra.projection(C_214=[50]:bat[:oid], X_65=[1874564]:bat[:lng]); # project1_lng                                                       |
|    212 | X_217=[50]:bat[:lng] := mat.packIncrement(X_215=[50]:bat[:lng], 32:int);                                                                                         |
|     69 | X_218=[100]:bat[:lng] := mat.packIncrement(X_217=[100]:bat[:lng], X_144=[50]:bat[:lng]);                                                                         |
|     41 | X_219=[150]:bat[:lng] := mat.packIncrement(X_218=[150]:bat[:lng], X_185=[50]:bat[:lng]);                                                                         |
| 423580 | (X_220=[1874564]:bat[:oid], C_221=[50]:bat[:oid]) := group.groupdone(X_83=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    108 | X_222=[50]:bat[:lng] := algebra.projection(C_221=[50]:bat[:oid], X_83=[1874564]:bat[:lng]); # project1_lng                                                       |
|    101 | X_223=0@0:void := language.pass(X_90=[1874564]:bat[:lng]);                                                                                                       |
|    220 | X_224=0@0:void := language.pass(X_65=[1874564]:bat[:lng]);                                                                                                       |
|    281 | X_225=[200]:bat[:lng] := mat.packIncrement(X_219=[200]:bat[:lng], X_209=[50]:bat[:lng]);                                                                         |
|     92 | X_226=0@0:void := language.pass(X_83=[1874564]:bat[:lng]);                                                                                                       |
|  61396 | X_227=[50]:bat[:lng] := aggr.subcount(X_95=[1874564]:bat[:bte], X_146=[1874564]:bat[:oid], C_147=[50]:bat[:oid], true:bit);                                      |
| 414175 | (X_228=[1874564]:bat[:oid], C_229=[50]:bat[:oid]) := group.groupdone(X_86=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    160 | X_230=[50]:bat[:lng] := algebra.projection(C_229=[50]:bat[:oid], X_86=[1874564]:bat[:lng]); # project1_lng                                                       |
|     80 | X_231=[250]:bat[:lng] := mat.packIncrement(X_225=[250]:bat[:lng], X_230=[50]:bat[:lng]);                                                                         |
|  75101 | X_232=[50]:bat[:lng] := aggr.subcount(X_111=[1874564]:bat[:bte], X_142=[1874564]:bat[:oid], C_143=[50]:bat[:oid], true:bit);                                     |
|     65 | X_233=[300]:bat[:lng] := mat.packIncrement(X_231=[300]:bat[:lng], X_156=[50]:bat[:lng]);                                                                         |
|     97 | X_234=[350]:bat[:lng] := mat.packIncrement(X_233=[350]:bat[:lng], X_140=[50]:bat[:lng]);                                                                         |
|     55 | X_235=[400]:bat[:lng] := mat.packIncrement(X_234=[400]:bat[:lng], X_204=[50]:bat[:lng]);                                                                         |
|     87 | X_236=[450]:bat[:lng] := mat.packIncrement(X_235=[450]:bat[:lng], X_151=[50]:bat[:lng]);                                                                         |
| 409987 | (X_237=[1874564]:bat[:oid], C_238=[50]:bat[:oid]) := group.groupdone(X_74=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    111 | X_239=[500]:bat[:lng] := mat.packIncrement(X_236=[500]:bat[:lng], X_189=[50]:bat[:lng]);                                                                         |
|    153 | X_240=[50]:bat[:lng] := algebra.projection(C_238=[50]:bat[:oid], X_74=[1874564]:bat[:lng]); # project1_lng                                                       |
|     64 | X_241=0@0:void := language.pass(X_86=[1874564]:bat[:lng]);                                                                                                       |
|    107 | X_242=0@0:void := language.pass(X_74=[1874564]:bat[:lng]);                                                                                                       |
|     52 | X_243=0@0:void := language.pass(C_147=[50]:bat[:oid]);                                                                                                           |
|     76 | X_244=0@0:void := language.pass(C_143=[50]:bat[:oid]);                                                                                                           |
| 387782 | (X_245=[1874564]:bat[:oid], C_246=[50]:bat[:oid]) := group.groupdone(X_133=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                          |
|    169 | X_247=[50]:bat[:lng] := algebra.projection(C_246=[50]:bat[:oid], X_133=[1874564]:bat[:lng]); # project1_lng                                                      |
|    166 | X_248=0@0:void := language.pass(X_133=[1874564]:bat[:lng]);                                                                                                      |
| 482551 | (X_249=[1874564]:bat[:oid], C_250=[50]:bat[:oid]) := group.groupdone(X_41=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    198 | X_251=[50]:bat[:lng] := algebra.projection(C_250=[50]:bat[:oid], X_41=[1874564]:bat[:lng]); # project1_lng                                                       |
|     30 | X_252=0@0:void := language.pass(X_41=[1874564]:bat[:lng]);                                                                                                       |
| 453841 | (X_253=[1874564]:bat[:oid], C_254=[50]:bat[:oid]) := group.groupdone(X_93=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
| 400564 | (X_255=[1874564]:bat[:oid], C_256=[50]:bat[:oid]) := group.groupdone(X_129=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                          |
| 400312 | (X_257=[1874564]:bat[:oid], C_258=[50]:bat[:oid]) := group.groupdone(X_52=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
| 443966 | (X_259=[1874564]:bat[:oid], C_260=[50]:bat[:oid]) := group.groupdone(X_81=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    242 | X_264=[50]:bat[:lng] := algebra.projection(C_254=[50]:bat[:oid], X_93=[1874564]:bat[:lng]); # project1_lng                                                       |
|  56087 | X_261=[50]:bat[:lng] := aggr.subcount(X_124=[1874564]:bat[:bte], X_167=[1874564]:bat[:oid], C_168=[50]:bat[:oid], true:bit);                                     |
| 486838 | (X_262=[1874564]:bat[:oid], C_263=[50]:bat[:oid]) := group.groupdone(X_64=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|  52578 | X_265=[50]:bat[:lng] := aggr.subcount(X_116=[1874564]:bat[:bte], X_177=[1874564]:bat[:oid], C_178=[50]:bat[:oid], true:bit);                                     |
|    379 | X_266=[50]:bat[:lng] := algebra.projection(C_256=[50]:bat[:oid], X_129=[1874564]:bat[:lng]); # project1_lng                                                      |
|     89 | X_267=0@0:void := language.pass(X_93=[1874564]:bat[:lng]);                                                                                                       |
|    263 | X_268=[50]:bat[:lng] := algebra.projection(C_263=[50]:bat[:oid], X_64=[1874564]:bat[:lng]); # project1_lng                                                       |
|    201 | X_269=0@0:void := language.pass(X_129=[1874564]:bat[:lng]);                                                                                                      |
|     45 | X_270=0@0:void := language.pass(X_64=[1874564]:bat[:lng]);                                                                                                       |
|    631 | X_271=[50]:bat[:lng] := algebra.projection(C_260=[50]:bat[:oid], X_81=[1874564]:bat[:lng]); # project1_lng                                                       |
|     31 | X_272=0@0:void := language.pass(X_81=[1874564]:bat[:lng]);                                                                                                       |
|    948 | X_273=[50]:bat[:lng] := algebra.projection(C_258=[50]:bat[:oid], X_52=[1874564]:bat[:lng]); # project1_lng                                                       |
|    135 | X_274=0@0:void := language.pass(X_52=[1874564]:bat[:lng]);                                                                                                       |
|  77951 | X_275=[50]:bat[:lng] := aggr.subcount(X_109=[1874564]:bat[:bte], X_148=[1874564]:bat[:oid], C_149=[50]:bat[:oid], true:bit);                                     |
|     92 | X_276=0@0:void := language.pass(C_168=[50]:bat[:oid]);                                                                                                           |
|     60 | X_277=0@0:void := language.pass(C_149=[50]:bat[:oid]);                                                                                                           |
|  72183 | X_278=[50]:bat[:lng] := aggr.subcount(X_110=[1874564]:bat[:bte], X_154=[1874564]:bat[:oid], C_155=[50]:bat[:oid], true:bit);                                     |
|    116 | X_279=[550]:bat[:lng] := mat.packIncrement(X_239=[550]:bat[:lng], X_273=[50]:bat[:lng]);                                                                         |
|     63 | X_280=[600]:bat[:lng] := mat.packIncrement(X_279=[600]:bat[:lng], X_222=[50]:bat[:lng]);                                                                         |
|     54 | X_281=[650]:bat[:lng] := mat.packIncrement(X_280=[650]:bat[:lng], X_271=[50]:bat[:lng]);                                                                         |
|     52 | X_282=[700]:bat[:lng] := mat.packIncrement(X_281=[700]:bat[:lng], X_264=[50]:bat[:lng]);                                                                         |
| 450427 | (X_283=[1874564]:bat[:oid], C_284=[50]:bat[:oid]) := group.groupdone(X_78=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    138 | X_285=[50]:bat[:lng] := algebra.projection(C_284=[50]:bat[:oid], X_78=[1874564]:bat[:lng]); # project1_lng                                                       |
|    210 | X_286=[750]:bat[:lng] := mat.packIncrement(X_282=[750]:bat[:lng], X_137=[50]:bat[:lng]);                                                                         |
|    136 | X_287=0@0:void := language.pass(X_78=[1874564]:bat[:lng]);                                                                                                       |
|     72 | X_288=[800]:bat[:lng] := mat.packIncrement(X_286=[800]:bat[:lng], X_285=[50]:bat[:lng]);                                                                         |
|    121 | X_289=0@0:void := language.pass(C_178=[50]:bat[:oid]);                                                                                                           |
|     65 | X_290=[850]:bat[:lng] := mat.packIncrement(X_288=[850]:bat[:lng], X_247=[50]:bat[:lng]);                                                                         |
|     59 | X_291=[900]:bat[:lng] := mat.packIncrement(X_290=[900]:bat[:lng], X_266=[50]:bat[:lng]);                                                                         |
|     67 | X_292=[950]:bat[:lng] := mat.packIncrement(X_291=[950]:bat[:lng], X_179=[50]:bat[:lng]);                                                                         |
|     68 | X_293=[1000]:bat[:lng] := mat.packIncrement(X_292=[1000]:bat[:lng], X_192=[50]:bat[:lng]);                                                                       |
|     53 | X_294=[1050]:bat[:lng] := mat.packIncrement(X_293=[1050]:bat[:lng], X_166=[50]:bat[:lng]);                                                                       |
|     57 | X_295=[1100]:bat[:lng] := mat.packIncrement(X_294=[1100]:bat[:lng], X_150=[50]:bat[:lng]);                                                                       |
|  67713 | X_296=[50]:bat[:lng] := aggr.subcount(X_131=[1874568]:bat[:bte], X_160=[1874568]:bat[:oid], C_161=[50]:bat[:oid], true:bit);                                     |
|     56 | X_297=[1150]:bat[:lng] := mat.packIncrement(X_295=[1150]:bat[:lng], X_200=[50]:bat[:lng]);                                                                       |
|    115 | X_298=[1200]:bat[:lng] := mat.packIncrement(X_297=[1200]:bat[:lng], X_240=[50]:bat[:lng]);                                                                       |
|     57 | X_299=[1250]:bat[:lng] := mat.packIncrement(X_298=[1250]:bat[:lng], X_175=[50]:bat[:lng]);                                                                       |
|     57 | X_300=[1300]:bat[:lng] := mat.packIncrement(X_299=[1300]:bat[:lng], X_216=[50]:bat[:lng]);                                                                       |
|     56 | X_301=[1350]:bat[:lng] := mat.packIncrement(X_300=[1350]:bat[:lng], X_169=[50]:bat[:lng]);                                                                       |
|     51 | X_302=0@0:void := language.pass(C_155=[50]:bat[:oid]);                                                                                                           |
|  48193 | X_303=[50]:bat[:lng] := aggr.subcount(X_103=[1874564]:bat[:bte], X_190=[1874564]:bat[:oid], C_191=[50]:bat[:oid], true:bit);                                     |
|    226 | X_304=[1400]:bat[:lng] := mat.packIncrement(X_301=[1400]:bat[:lng], X_251=[50]:bat[:lng]);                                                                       |
|     45 | X_305=0@0:void := language.pass(C_161=[50]:bat[:oid]);                                                                                                           |
|  76531 | X_306=[50]:bat[:lng] := aggr.subcount(X_130=[1874564]:bat[:bte], X_164=[1874564]:bat[:oid], C_165=[50]:bat[:oid], true:bit);                                     |
| 437390 | (X_307=[1874564]:bat[:oid], C_308=[50]:bat[:oid]) := group.groupdone(X_66=[1874564]:bat[:lng]); # GRP_create_partial_hash_table, dense                           |
|    159 | X_309=0@0:void := language.pass(C_191=[50]:bat[:oid]);                                                                                                           |
|    209 | X_310=[50]:bat[:lng] := algebra.projection(C_308=[50]:bat[:oid], X_66=[1874564]:bat[:lng]); # project1_lng                                                       |
|    122 | X_311=[1450]:bat[:lng] := mat.packIncrement(X_304=[1450]:bat[:lng], X_310=[50]:bat[:lng]);                                                                       |
|     58 | X_312=[1500]:bat[:lng] := mat.packIncrement(X_311=[1500]:bat[:lng], X_197=[50]:bat[:lng]);                                                                       |
|     88 | X_313=[1550]:bat[:lng] := mat.packIncrement(X_312=[1550]:bat[:lng], X_268=[50]:bat[:lng]);                                                                       |
|    149 | X_314=0@0:void := language.pass(X_66=[1874564]:bat[:lng]);                                                                                                       |
|     60 | X_315=[1600]:bat[:lng] := mat.packIncrement(X_313=[1600]:bat[:lng], X_162=[50]:bat[:lng]);                                                                       |
|  70639 | X_316=[50]:bat[:lng] := aggr.subcount(X_119=[1874564]:bat[:bte], X_173=[1874564]:bat[:oid], C_174=[50]:bat[:oid], true:bit);                                     |
|    610 | (X_317=[1600]:bat[:oid], C_318=[50]:bat[:oid]) := group.groupdone(X_315=[1600]:bat[:lng]); # GRP_create_partial_hash_table, dense                                |
|     45 | X_319=0@0:void := language.pass(C_165=[50]:bat[:oid]);                                                                                                           |
|  58053 | X_320=[50]:bat[:lng] := aggr.subcount(X_127=[1874564]:bat[:bte], X_198=[1874564]:bat[:oid], C_199=[50]:bat[:oid], true:bit);                                     |
|     61 | X_321=0@0:void := language.pass(C_174=[50]:bat[:oid]);                                                                                                           |
|  64200 | X_322=[50]:bat[:lng] := aggr.subcount(X_120=[1874564]:bat[:bte], X_187=[1874564]:bat[:oid], C_188=[50]:bat[:oid], true:bit);                                     |
|  71763 | X_323=[50]:bat[:lng] := aggr.subcount(X_126=[1874564]:bat[:bte], X_183=[1874564]:bat[:oid], C_184=[50]:bat[:oid], true:bit);                                     |
|     83 | X_324=0@0:void := language.pass(C_199=[50]:bat[:oid]);                                                                                                           |
|     68 | X_325=0@0:void := language.pass(C_188=[50]:bat[:oid]);                                                                                                           |
|     36 | X_326=0@0:void := language.pass(C_184=[50]:bat[:oid]);                                                                                                           |
|  61573 | X_327=[50]:bat[:lng] := aggr.subcount(X_97=[1874564]:bat[:bte], X_211=[1874564]:bat[:oid], C_212=[50]:bat[:oid], true:bit);                                      |
|  69910 | X_328=[50]:bat[:lng] := aggr.subcount(X_98=[1874564]:bat[:bte], X_207=[1874564]:bat[:oid], C_208=[50]:bat[:oid], true:bit);                                      |
|  71305 | X_329=[50]:bat[:lng] := aggr.subcount(X_102=[1874564]:bat[:bte], X_201=[1874564]:bat[:oid], C_202=[50]:bat[:oid], true:bit);                                     |
|  65283 | X_330=[50]:bat[:lng] := aggr.subcount(X_104=[1874564]:bat[:bte], X_220=[1874564]:bat[:oid], C_221=[50]:bat[:oid], true:bit);                                     |
|     50 | X_331=0@0:void := language.pass(C_212=[50]:bat[:oid]);                                                                                                           |
|    194 | X_332=[50]:bat[:lng] := mat.packIncrement(X_327=[50]:bat[:lng], 32:int);                                                                                         |
|     68 | X_333=[100]:bat[:lng] := mat.packIncrement(X_332=[100]:bat[:lng], X_232=[50]:bat[:lng]);                                                                         |
|     55 | X_334=[150]:bat[:lng] := mat.packIncrement(X_333=[150]:bat[:lng], X_323=[50]:bat[:lng]);                                                                         |
|  77340 | X_335=[50]:bat[:lng] := aggr.subcount(X_108=[1874564]:bat[:bte], X_195=[1874564]:bat[:oid], C_196=[50]:bat[:oid], true:bit);                                     |
|     38 | X_336=0@0:void := language.pass(C_208=[50]:bat[:oid]);                                                                                                           |
|     92 | X_337=[200]:bat[:lng] := mat.packIncrement(X_334=[200]:bat[:lng], X_328=[50]:bat[:lng]);                                                                         |
|     89 | X_338=0@0:void := language.pass(C_202=[50]:bat[:oid]);                                                                                                           |
|     66 | X_339=0@0:void := language.pass(C_221=[50]:bat[:oid]);                                                                                                           |
|     47 | X_340=0@0:void := language.pass(C_196=[50]:bat[:oid]);                                                                                                           |
|  73230 | X_341=[50]:bat[:lng] := aggr.subcount(X_123=[1874564]:bat[:bte], X_213=[1874564]:bat[:oid], C_214=[50]:bat[:oid], true:bit);                                     |
|  50437 | X_342=[50]:bat[:lng] := aggr.subcount(X_118=[1874564]:bat[:bte], X_253=[1874564]:bat[:oid], C_254=[50]:bat[:oid], true:bit);                                     |
|  51738 | X_343=[50]:bat[:lng] := aggr.subcount(X_122=[1874564]:bat[:bte], X_262=[1874564]:bat[:oid], C_263=[50]:bat[:oid], true:bit);                                     |
|  52322 | X_344=[50]:bat[:lng] := aggr.subcount(X_132=[1874564]:bat[:bte], X_255=[1874564]:bat[:oid], C_256=[50]:bat[:oid], true:bit);                                     |
|  54871 | X_345=[50]:bat[:lng] := aggr.subcount(X_96=[1874564]:bat[:bte], X_259=[1874564]:bat[:oid], C_260=[50]:bat[:oid], true:bit);                                      |
|     53 | X_346=0@0:void := language.pass(C_214=[50]:bat[:oid]);                                                                                                           |
|     53 | X_347=0@0:void := language.pass(C_254=[50]:bat[:oid]);                                                                                                           |
|     48 | X_348=0@0:void := language.pass(C_263=[50]:bat[:oid]);                                                                                                           |
|     38 | X_349=0@0:void := language.pass(C_256=[50]:bat[:oid]);                                                                                                           |
|  73145 | X_350=[50]:bat[:lng] := aggr.subcount(X_105=[1874564]:bat[:bte], X_228=[1874564]:bat[:oid], C_229=[50]:bat[:oid], true:bit);                                     |
|  60789 | X_351=[50]:bat[:lng] := aggr.subcount(X_128=[1874564]:bat[:bte], X_257=[1874564]:bat[:oid], C_258=[50]:bat[:oid], true:bit);                                     |
|  76017 | X_352=[50]:bat[:lng] := aggr.subcount(X_106=[1874564]:bat[:bte], X_237=[1874564]:bat[:oid], C_238=[50]:bat[:oid], true:bit);                                     |
|  71501 | X_353=[50]:bat[:lng] := aggr.subcount(X_134=[1874564]:bat[:bte], X_245=[1874564]:bat[:oid], C_246=[50]:bat[:oid], true:bit);                                     |
|  72126 | X_354=[50]:bat[:lng] := aggr.subcount(X_125=[1874564]:bat[:bte], X_249=[1874564]:bat[:oid], C_250=[50]:bat[:oid], true:bit);                                     |
|    136 | X_355=0@0:void := language.pass(C_229=[50]:bat[:oid]);                                                                                                           |
|    221 | X_356=[250]:bat[:lng] := mat.packIncrement(X_337=[250]:bat[:lng], X_350=[50]:bat[:lng]);                                                                         |
|     60 | X_357=0@0:void := language.pass(C_258=[50]:bat[:oid]);                                                                                                           |
|    182 | X_358=[300]:bat[:lng] := mat.packIncrement(X_356=[300]:bat[:lng], X_278=[50]:bat[:lng]);                                                                         |
|     46 | X_359=0@0:void := language.pass(C_260=[50]:bat[:oid]);                                                                                                           |
|     60 | X_360=[350]:bat[:lng] := mat.packIncrement(X_358=[350]:bat[:lng], X_181=[50]:bat[:lng]);                                                                         |
|     52 | X_361=[400]:bat[:lng] := mat.packIncrement(X_360=[400]:bat[:lng], X_329=[50]:bat[:lng]);                                                                         |
|     51 | X_362=[450]:bat[:lng] := mat.packIncrement(X_361=[450]:bat[:lng], X_275=[50]:bat[:lng]);                                                                         |
|     50 | X_363=[500]:bat[:lng] := mat.packIncrement(X_362=[500]:bat[:lng], X_322=[50]:bat[:lng]);                                                                         |
|     58 | X_364=[550]:bat[:lng] := mat.packIncrement(X_363=[550]:bat[:lng], X_351=[50]:bat[:lng]);                                                                         |
|     52 | X_365=[600]:bat[:lng] := mat.packIncrement(X_364=[600]:bat[:lng], X_330=[50]:bat[:lng]);                                                                         |
|     49 | X_366=[650]:bat[:lng] := mat.packIncrement(X_365=[650]:bat[:lng], X_345=[50]:bat[:lng]);                                                                         |
|     51 | X_367=[700]:bat[:lng] := mat.packIncrement(X_366=[700]:bat[:lng], X_342=[50]:bat[:lng]);                                                                         |
|     46 | X_368=0@0:void := language.pass(C_238=[50]:bat[:oid]);                                                                                                           |
|     62 | X_369=0@0:void := language.pass(C_246=[50]:bat[:oid]);                                                                                                           |
|     40 | X_370=0@0:void := language.pass(C_250=[50]:bat[:oid]);                                                                                                           |
|    162 | X_371=[750]:bat[:lng] := mat.packIncrement(X_367=[750]:bat[:lng], X_158=[50]:bat[:lng]);                                                                         |
|  72614 | X_372=[50]:bat[:lng] := aggr.subcount(X_107=[1874564]:bat[:bte], X_283=[1874564]:bat[:oid], C_284=[50]:bat[:oid], true:bit);                                     |
|  60407 | X_373=[50]:bat[:lng] := aggr.subcount(X_112=[1874564]:bat[:bte], X_307=[1874564]:bat[:oid], C_308=[50]:bat[:oid], true:bit);                                     |
|     41 | X_374=0@0:void := language.pass(C_284=[50]:bat[:oid]);                                                                                                           |
|     98 | X_375=[800]:bat[:lng] := mat.packIncrement(X_371=[800]:bat[:lng], X_372=[50]:bat[:lng]);                                                                         |
|     50 | X_376=[850]:bat[:lng] := mat.packIncrement(X_375=[850]:bat[:lng], X_353=[50]:bat[:lng]);                                                                         |
|     40 | X_377=[900]:bat[:lng] := mat.packIncrement(X_376=[900]:bat[:lng], X_344=[50]:bat[:lng]);                                                                         |
|     47 | X_378=[950]:bat[:lng] := mat.packIncrement(X_377=[950]:bat[:lng], X_265=[50]:bat[:lng]);                                                                         |
|     56 | X_379=0@0:void := language.pass(C_308=[50]:bat[:oid]);                                                                                                           |
|     68 | X_380=[1000]:bat[:lng] := mat.packIncrement(X_378=[1000]:bat[:lng], X_303=[50]:bat[:lng]);                                                                       |
|     42 | X_381=[1050]:bat[:lng] := mat.packIncrement(X_380=[1050]:bat[:lng], X_306=[50]:bat[:lng]);                                                                       |
|     38 | X_382=[1100]:bat[:lng] := mat.packIncrement(X_381=[1100]:bat[:lng], X_227=[50]:bat[:lng]);                                                                       |
|     65 | X_383=[1150]:bat[:lng] := mat.packIncrement(X_382=[1150]:bat[:lng], X_320=[50]:bat[:lng]);                                                                       |
|     41 | X_384=[1200]:bat[:lng] := mat.packIncrement(X_383=[1200]:bat[:lng], X_352=[50]:bat[:lng]);                                                                       |
|     45 | X_385=[1250]:bat[:lng] := mat.packIncrement(X_384=[1250]:bat[:lng], X_316=[50]:bat[:lng]);                                                                       |
|     39 | X_386=[1300]:bat[:lng] := mat.packIncrement(X_385=[1300]:bat[:lng], X_341=[50]:bat[:lng]);                                                                       |
|     63 | X_387=[1350]:bat[:lng] := mat.packIncrement(X_386=[1350]:bat[:lng], X_261=[50]:bat[:lng]);                                                                       |
|     41 | X_388=[1400]:bat[:lng] := mat.packIncrement(X_387=[1400]:bat[:lng], X_354=[50]:bat[:lng]);                                                                       |
|     84 | X_389=[1450]:bat[:lng] := mat.packIncrement(X_388=[1450]:bat[:lng], X_373=[50]:bat[:lng]);                                                                       |
|     43 | X_390=[1500]:bat[:lng] := mat.packIncrement(X_389=[1500]:bat[:lng], X_335=[50]:bat[:lng]);                                                                       |
|     38 | X_391=[1550]:bat[:lng] := mat.packIncrement(X_390=[1550]:bat[:lng], X_343=[50]:bat[:lng]);                                                                       |
|     40 | X_392=[1600]:bat[:lng] := mat.packIncrement(X_391=[1600]:bat[:lng], X_296=[50]:bat[:lng]);                                                                       |
|    325 | X_393=[50]:bat[:lng] := aggr.subsum(X_392=[1600]:bat[:lng], X_317=[1600]:bat[:oid], C_318=[50]:bat[:oid], true:bit, true:bit); # sum: no candidates, with groups |
|     71 | C_394=[10]:bat[:oid] := algebra.subslice(X_393=[50]:bat[:lng], 0:lng, 9:lng);                                                                                    |
|    104 | X_395=[10]:bat[:lng] := algebra.projection(C_394=[10]:bat[:oid], X_393=[50]:bat[:lng]);                                                                          |
|     26 | X_396=0@0:void := language.pass(X_393=[50]:bat[:lng]);                                                                                                           |
| 607665 | barrier X_397=false:bit := language.dataflow();                                                                                                                  |
|    589 | X_398=2:int := sql.resultSet(X_5=[1]:bat[:str], X_4=[1]:bat[:str], X_6=[1]:bat[:str], X_1=[1]:bat[:int], X_3=[1]:bat[:int], X_395=[10]:bat[:lng]);               |
adofsauron commented 1 year ago 

 {
      "join_optimization": {
        "select#": 1,
        "steps": [
          {
            "substitute_generated_columns": {
            }
          },
          {
            "table_dependencies": [
              {
                "table": "`lineitem`",
                "row_may_be_null": false,
                "map_bit": 0,
                "depends_on_map_bits": [
                ]
              }
            ]
          },
          {
            "rows_estimation": [
              {
                "table": "`lineitem`",
                "table_scan": {
                  "rows": 32986053,
                  "cost": 1649312
                }
              }
            ]
          },
          {
            "considered_execution_plans": [
              {
                "plan_prefix": [
                ],
                "table": "`lineitem`",
                "best_access_path": {
                  "considered_access_paths": [
                    {
                      "rows_to_scan": 32986053,
                      "access_type": "scan",
                      "resulting_rows": 3.3e7,
                      "cost": 8.25e6,
                      "chosen": true
                    }
                  ]
                },
                "condition_filtering_pct": 100,
                "rows_for_plan": 3.3e7,
                "cost_for_plan": 8.25e6,
                "chosen": true
              }
            ]
          },
          {
            "attaching_conditions_to_tables": {
              "original_condition": null,
              "attached_conditions_computation": [
              ],
              "attached_conditions_summary": [
                {
                  "table": "`lineitem`",
                  "attached": null
                }
              ]
            }
          },
          {
            "clause_processing": {
              "clause": "GROUP BY",
              "original_clause": "`lineitem`.`l_quantity`",
              "items": [
                {
                  "item": "`lineitem`.`l_quantity`"
                }
              ],
              "resulting_clause_is_simple": true,
              "resulting_clause": "`lineitem`.`l_quantity`"
            }
          },
          {
            "refine_plan": [
              {
                "table": "`lineitem`"
              }
            ]
          }
        ]
      }
    },
    {
      "join_explain": {
        "select#": 1,
        "steps": [
        ]
      }
    }
  ]
adofsauron commented 1 year ago

The way column data is accessed makes it difficult to be thread-safe when multithreaded slicing

Specifies the key of the hash table used for aggregation of the grouping Grouping

The classic volcano model is used

adofsauron commented 1 year ago

The difficulty of slicing column data horizontally prevents thread parallelism

The burst is manifested in

Thread-safe, poorly defined critical sections lead to errors when accessing data

Some data types, such as decimal, are evaluated at access time, causing multiple threads to modify the same piece of data at the same time

The most prominent problem with an aggregated hahs table using the value of the grouping column as the key for grouping is performance and memory usage

The volcanic model leads to vectorization problems

The iterator of the volcano model results in non-vectorization of data access

Bad for CPU cache

Generating an aggregation hash during access to the volcano model results in the multi-pipeline branch prediction taking effect

adofsauron commented 1 year ago

The data in the column file was not reprocessed

The data in the column file is not sorted

adofsauron commented 1 year ago

Use batch traversal to optimize cpu cache

adofsauron commented 1 year ago

One of the dangers of such an interpreter, especially if the granularity of interpretation is a tuple, is that the cost of the “real work” (i.e. executing the expressions found in the query) is only a tiny fraction of total query execution cost.

adofsauron commented 1 year ago

Other factors, such as locking overhead (pthread mutex unlock, mutex test and set) or buffer page allocation (buf frame align) seem to play only a minor role in this decision support query

adofsauron commented 1 year ago

8dc6ec62247c6ae8d3e3dd72ef96826b

adofsauron commented 1 year ago

As the routine called by MySQL only computes one addition per call, instead of an array of additions, the compiler cannot perform loop pipelining.