adofsauron commented 1 year ago

Abstract:

TIANMU Engine - Aggregate operation intermediate results cache disk files to avoid OOM- requirements analysis

Note of context:

The results of the current aggregate operation are cached in the in-memory HASH. Once the amount of data exceeds the RAM, the OOM occurs.

If the data volume exceeds RAM, convert it to disk file storage to avoid OOM

Functional Requirements:

1. The HASH result of the aggregation operation is cached in the disk file and properly participates in the operation

When the intermediate result exceeds RAM

Performance requirements:

1. The compression ratio of the cache file size to the original data content, and the ratio of disk space to memory

Influence compression algorithm

2. Disk cache file write speed and read speed

Standard disk hardware

It affects the speed of aggregation operations and the rules for writing and reading disk cache files

3. The impact on the performance of aggregation operations

Development cycle:

TODO:

adofsauron commented 1 year ago

Aggregate Cache disks - Development process

Funny layer

Step 1 Include

Idiots who can't read code
A variety of power jokers with managerial titles

Step 2: Strategy

Straight to death
Not qualified to participate in any development process
Requirement layer

Step 1 Include

Product side
Test side

Step 2: Strategy

Rigorous needs analysis to define boundaries
Leave clear evidence of any needs
In case of transgression
Kill immediately
Leave clear evidence to inform all
Remember the past as a guide to the future
Working layer

Step 1 Include

Someone who can read code
Someone who can go beyond existing code

Step 2: Strategy

Demonstrate the rationality of each detail design in detail
Cross validation
Development planning
Demand determination and counter-propaganda
One to two days
Programme pre-study
One to two days
Detailed scheme design
Two to three days
Code development
Three to five days
Write unit tests /MTR
Two to three days

adofsauron commented 1 year ago

Aggregate cache hard drives - Requirements analysis

Functional requirements
If the aggregate cache exceeds RAM, use hard disk files
Write the disk only when the RAM exceeds
Limit the size of cache files
The aggregation result must be the same before and after the change
Ensure that the query results are consistent only before and after the modification
INNODB does not guarantee the consistency of query results
You can set parameters to disable the feature dynamically
The aggregation operation in progress is not affected
Users decide whether to enable this feature
After the query is complete, the disk cache file is automatically cleared
Consider asynchrony
Disk cache clearing does not affect aggregate queries
Performance requirements
Disk data starts to be written after the RAM usage reaches the threshold
80%
Reserve memory for other modules to work
Intermediate result Indicates the disk cache file write speed threshold
Compression ratio of the cache file to the original data
Compression speed of the cache file compression algorithm
Write disk I/OS synchronously or asynchronously
Check whether the block cache needs to flush disk I/OS in batches to improve performance
Indicates the speed threshold for reading cache files
The decompressing speed of the cache file compression algorithm
What can I do if the memory is insufficient when I read the cached file
Whether to use elimination and replacement mechanism
If there is no replacement
Terminate the aggregate query
Give a clear error message
Clear the disk cache
Impact of cache file reading on aggregate scanning
Aggregate scan sequential traversal
Append the intermediate result
Writing to the cached file will involve disk I/O
Determine whether to create a resident memory pool for cached files
Replace the BLOCK with LRU
Local data caching reduces repeated I/OS
Impact on aggregate query
Intermediate results are all in memory
Some intermediate results are cached on the disk
30%
50%
80%
All intermediate results are cached on disk
Stability needs
mysqld dump occurs when data is being written to the disk cache
How do I clean residual files after mysqld restarts
Disk I/OS are abnormal when data is being written to the disk cache
The write fails
How do I verify the correctness of cached files
CRC
Other verification algorithms
Read data fails
Terminate the aggregate query
An error message is displayed
Clear the disk cache
Impact on other modules
Aggregate the consistency of computing module interfaces
Interface compatibility of the read/write module for the aggregated cache result
Pluggable and replaceable policy interfaces
Learn from the filesort module

adofsauron commented 1 year ago

Abstract:

Cache the disk file for the intermediate results of the aggregate operation to avoid the OOM summary design as a starting point for the next detailed design.

And play a role in communicating design ideas to other developers only.

Requirement analysis: 2022-11-17 mysql column storage Engine - Aggregate operation intermediate results cache disk files to avoid OOM- Requirement Analysis _ Zunwu World's blog -CSDN blog

Cache RAM analysis of current aggregated intermediate results:

Static structure:

Dynamic structure:

adofsauron commented 1 year ago

Summary design for adding disk cache:

Design Idea:

Maintain compatibility with the upper layer module interfaces of existing aggregation operations

Avoid excessive disk I/OS without causing OOM

Design Strategy:

Add DiskCache instead of BlockedRowMemStorage, but keep the interface consistent

DiskCache uses LRU internally to cache disk blocks

When the LRU reaches the upper limit to eliminate blocks, the Block must be flushed to disk and the occupied RAM space must be released

adofsauron commented 1 year ago

Architecture Design:

Static structure:

Dynamic structure:

adofsauron commented 1 year ago

Refer to the AIO used by mysql for asynchronous IO

https://dev.mysql.com/doc/refman/5.7/en/innodb-linux-native-aio.html

adofsauron commented 1 year ago

The data flow that interacts with the file system

adofsauron commented 1 year ago

More flexible memory control and disk swapping mechanisms are needed to make more efficient use of memory

adofsauron commented 1 year ago

buffer Settings for mysql5.7.36


mysql> show variables like  '%buffer%';
+-------------------------------------+----------------+
| Variable_name                       | Value          |
+-------------------------------------+----------------+
| bulk_insert_buffer_size             | 8388608        |
| innodb_buffer_pool_chunk_size       | 134217728      |
| innodb_buffer_pool_dump_at_shutdown | ON             |
| innodb_buffer_pool_dump_now         | OFF            |
| innodb_buffer_pool_dump_pct         | 40             |
| innodb_buffer_pool_filename         | ib_buffer_pool |
| innodb_buffer_pool_instances        | 1              |
| innodb_buffer_pool_load_abort       | OFF            |
| innodb_buffer_pool_load_at_startup  | ON             |
| innodb_buffer_pool_load_now         | OFF            |
| innodb_buffer_pool_size             | 536870912      |
| innodb_change_buffer_max_size       | 25             |
| innodb_change_buffering             | all            |
| innodb_log_buffer_size              | 1048576        |
| innodb_sort_buffer_size             | 1048576        |
| join_buffer_size                    | 262144         |
| key_buffer_size                     | 536870912      |
| myisam_sort_buffer_size             | 8388608        |
| net_buffer_length                   | 16384          |
| preload_buffer_size                 | 32768          |
| read_buffer_size                    | 4194304        |
| read_rnd_buffer_size                | 16777216       |
| sort_buffer_size                    | 4194304        |
| sql_buffer_result                   | OFF            |
| tianmu_insert_buffer_size           | 512            |
| tianmu_insert_max_buffered          | 65536          |
| tianmu_sync_buffers                 | 0              |
+-------------------------------------+----------------+
27 rows in set (0.00 sec)


mysql> show global status like '%innodb_buffer_pool%';
+---------------------------------------+--------------------------------------------------+
| Variable_name                         | Value                                            |
+---------------------------------------+--------------------------------------------------+
| Innodb_buffer_pool_dump_status        | Dumping of buffer pool not started               |
| Innodb_buffer_pool_load_status        | Buffer pool(s) load completed at 230208  7:42:46 |
| Innodb_buffer_pool_resize_status      |                                                  |
| Innodb_buffer_pool_pages_data         | 252                                              |
| Innodb_buffer_pool_bytes_data         | 4128768                                          |
| Innodb_buffer_pool_pages_dirty        | 0                                                |
| Innodb_buffer_pool_bytes_dirty        | 0                                                |
| Innodb_buffer_pool_pages_flushed      | 36                                               |
| Innodb_buffer_pool_pages_free         | 32512                                            |
| Innodb_buffer_pool_pages_misc         | 0                                                |
| Innodb_buffer_pool_pages_total        | 32764                                            |
| Innodb_buffer_pool_read_ahead_rnd     | 0                                                |
| Innodb_buffer_pool_read_ahead         | 0                                                |
| Innodb_buffer_pool_read_ahead_evicted | 0                                                |
| Innodb_buffer_pool_read_requests      | 1055                                             |
| Innodb_buffer_pool_reads              | 219                                              |
| Innodb_buffer_pool_wait_free          | 0                                                |
| Innodb_buffer_pool_write_requests     | 325                                              |
+---------------------------------------+--------------------------------------------------+
18 rows in set (0.00 sec)

adofsauron commented 1 year ago

Buffer Pool

When reading data, it will first check whether the data page exists in the cache. If it does not exist, it will search on disk and then cache it into innodb_buffer_pool. Similarly, the insertion, modification, or deletion of data in the cache is done first, and then updated to the disk at a certain frequency. This mechanism is called Checkpoint


The last 3/8 area of the LRU list is used to store cold data.

The midpoint of LRU list is the boundary where hot data tail and cold data head intersect.

The accessed cold data is moved from the cold data list to the hot data list.

If the data in the hot data list is not accessed for a long time, it will gradually move to the cold data list.

Cold data is not accessed for a long time, and the LRU list is full, then the cold data at the end of the LRU list will be eliminated.

Preread data will only be inserted into the LRU list and will not be moved to the hot data list.

adofsauron commented 1 year ago

Multiple Buffer Pool Instances


For systems with buffer pools in the multi-gigabyte range, dividing the buffer pool into separate instances can improve concurrency, by reducing contention as different threads read and write to cached pages. This feature is typically intended for systems with a [buffer pool](https://dev.mysql.com/doc/refman/8.0/en/glossary.html#glos_buffer_pool) size in the multi-gigabyte range. Multiple buffer pool instances are configured using the [innodb_buffer_pool_instances](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_buffer_pool_instances) configuration option, and you might also adjust the [innodb_buffer_pool_size](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_buffer_pool_size) value.

When the InnoDB buffer pool is large, many data requests can be satisfied by retrieving from memory. You might encounter bottlenecks from multiple threads trying to access the buffer pool at once. You can enable multiple buffer pools to minimize this contention. Each page that is stored in or read from the buffer pool is assigned to one of the buffer pools randomly, using a hashing function. Each buffer pool manages its own free lists, flush lists, LRUs, and all other data structures connected to a buffer pool. Prior to MySQL 8.0, each buffer pool was protected by its own buffer pool mutex. In MySQL 8.0 and later, the buffer pool mutex was replaced by several list and hash protecting mutexes, to reduce contention.

To enable multiple buffer pool instances, set the innodb_buffer_pool_instances configuration option to a value greater than 1 (the default) up to 64 (the maximum). This option takes effect only when you set innodb_buffer_pool_size to a size of 1GB or more. The total size you specify is divided among all the buffer pools. For best efficiency, specify a combination of [innodb_buffer_pool_instances](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_buffer_pool_instances) and [innodb_buffer_pool_size](https://dev.mysql.com/doc/refman/8.0/en/innodb-parameters.html#sysvar_innodb_buffer_pool_size) so that each buffer pool instance is at least 1GB.

adofsauron commented 1 year ago

Design objectives:

No matter how large the data volume is, no OOM is displayed

The upper limit of the disk usage is not considered

There is no noticeable performance degradation until memory is exhausted

adofsauron commented 1 year ago

Static memory control:

Simple control strategy

Unable to use memory efficiently

adofsauron commented 1 year ago

Dynamic memory control:

Granularity control of the minimum allocation unit block

Meta-information management for aggregation blocks

Identify whether the block is in memory or on disk

Location information on the disk

The meta information must all be in memory

The unity of the aggregation block: to read and write memory | disk interface

Unified upper-layer interfaces facilitate unified control

Multithreaded concurrent access control

Concurrent access by multiple query threads

A single query thread but concurrent access by multiple aggregation threads

adofsauron commented 1 year ago

A preaggregation operation of an aggregator in which internal data is stored through a hash table whose key is a "grouping key" value (for example, if group by b is used in an sql statement, the key of the hash table is all the different values of b in the table). The hash table is dynamic, and as the number of keys increases, ClickHouse switches it to a two-level hash table to improve performance; In addition, for different key types, ClickHouse provides many specialties to optimize for specific types.

For a single level hash table, the block that aggregator converts is the single_level block, for a double level hash table, the block that aggregator converts is the two_level block, two_level block will have a block_num, You can think of block_num as the key at the first level of the two-tier hash table. There are two benefits to using two_level block:

Blocks of the same block_num of multiple nodes performing pre-aggregation can be combined so that different combinations can perform the merge operation in parallel

If you restrict the nodes that produce two_level blocks to the order in which block_num increments, you can reduce memory usage because the data that needs to be merged must be in the same combination, and when you see a new block_num, it means that all previous merging operations have been completed.

In fact, the branch above that writes data to a disk file does just that. In GroupingAggregatedTransform node, it will be single_level block into two_level block, and carried out in accordance with the block_num combination, Then to MergingAggregatedBucketTransform merge, because MergingAggregatedBucketTransform there can be multiple, so combining phase can also be parallel. Finally to SortingAggregatedTransform nodes according to block_num sort

adofsauron commented 1 year ago

GROUP BY in External Memory You can enable dumping temporary data to the disk to restrict memory usage during GROUP BY. The max_bytes_before_external_group_by setting determines the threshold RAM consumption for dumping GROUP BY temporary data to the file system. If set to 0 (the default), it is disabled.

When using max_bytes_before_external_group_by, we recommend that you set max_memory_usage about twice as high. This is necessary because there are two stages to aggregation: reading the data and forming intermediate data (1) and merging the intermediate data (2). Dumping data to the file system can only occur during stage 1. If the temporary data wasn’t dumped, then stage 2 might require up to the same amount of memory as in stage 1.

For example, if max_memory_usage was set to 10000000000 and you want to use external aggregation, it makes sense to set max_bytes_before_external_group_by to 10000000000, and max_memory_usage to 20000000000. When external aggregation is triggered (if there was at least one dump of temporary data), maximum consumption of RAM is only slightly more than max_bytes_before_external_group_by.

With distributed query processing, external aggregation is performed on remote servers. In order for the requester server to use only a small amount of RAM, set distributed_aggregation_memory_efficient to 1.

When merging data flushed to the disk, as well as when merging results from remote servers when the distributed_aggregation_memory_efficient setting is enabled, consumes up to 1/256 * the_number_of_threads from the total amount of RAM.

When external aggregation is enabled, if there was less than max_bytes_before_external_group_by of data (i.e. data was not flushed), the query runs just as fast as without external aggregation. If any temporary data was flushed, the run time will be several times longer (approximately three times).

If you have an ORDER BY with a LIMIT after GROUP BY, then the amount of used RAM depends on the amount of data in LIMIT, not in the whole table. But if the ORDER BY does not have LIMIT, do not forget to enable external sorting (max_bytes_before_external_sort).

adofsauron commented 1 year ago


template <typename Method>
void Aggregator::writeToTemporaryFileImpl(
    AggregatedDataVariants & data_variants,
    Method & method,
    TemporaryFileStream & out) const
{
    size_t max_temporary_block_size_rows = 0;
    size_t max_temporary_block_size_bytes = 0;

    auto update_max_sizes = [&](const Block & block)
    {
        size_t block_size_rows = block.rows();
        size_t block_size_bytes = block.bytes();

        if (block_size_rows > max_temporary_block_size_rows)
            max_temporary_block_size_rows = block_size_rows;
        if (block_size_bytes > max_temporary_block_size_bytes)
            max_temporary_block_size_bytes = block_size_bytes;
    };

    for (UInt32 bucket = 0; bucket < Method::Data::NUM_BUCKETS; ++bucket)
    {
        Block block = convertOneBucketToBlock(data_variants, method, data_variants.aggregates_pool, false, bucket);
        out.write(block);
        update_max_sizes(block);
    }

    if (params.overflow_row)
    {
        Block block = prepareBlockAndFillWithoutKey(data_variants, false, true);
        out.write(block);
        update_max_sizes(block);
    }

    /// Pass ownership of the aggregate functions states:
    /// `data_variants` will not destroy them in the destructor, they are now owned by ColumnAggregateFunction objects.
    data_variants.aggregator = nullptr;

    LOG_DEBUG(log, "Max size of temporary block: {} rows, {}.", max_temporary_block_size_rows, ReadableSize(max_temporary_block_size_bytes));
}

adofsauron commented 1 year ago

Data write file


/// This class helps with the handling of temporary files or directories.
/// A unique name for the temporary file or directory is automatically chosen based on a specified prefix.
/// Create a directory in the constructor.
/// The destructor always removes the temporary file or directory with all contained files.
class TemporaryFileOnDisk
{
public:
    explicit TemporaryFileOnDisk(const DiskPtr & disk_);
    explicit TemporaryFileOnDisk(const DiskPtr & disk_, CurrentMetrics::Value metric_scope);
    explicit TemporaryFileOnDisk(const DiskPtr & disk_, const String & prefix);

    ~TemporaryFileOnDisk();

    DiskPtr getDisk() const { return disk; }
    String getPath() const;

private:
    DiskPtr disk;

    /// Relative path in disk to the temporary file or directory
    String relative_path;

    CurrentMetrics::Increment metric_increment;

    /// Specified if we know what for file is used (sort/aggregate/join).
    std::optional<CurrentMetrics::Increment> sub_metric_increment = {};
};


/*
 * Data can be written into this stream and then read.
 * After finish writing, call `finishWriting` and then `read` to read the data.
 * Account amount of data written to disk in parent scope.
 */
class TemporaryFileStream : boost::noncopyable
{
public:
    struct Stat
    {
        /// Statistics for file
        /// Non-atomic because we don't allow to `read` or `write` into single file from multiple threads
        size_t compressed_size = 0;
        size_t uncompressed_size = 0;
        size_t num_rows = 0;
    };

    TemporaryFileStream(TemporaryFileOnDiskHolder file_, const Block & header_, TemporaryDataOnDisk * parent_);
    TemporaryFileStream(FileSegmentsHolder && segments_, const Block & header_, TemporaryDataOnDisk * parent_);

    size_t write(const Block & block);
    void flush();

    Stat finishWriting();
    bool isWriteFinished() const;

    Block read();

    String getPath() const;

    Block getHeader() const { return header; }

    /// Read finished and file released
    bool isEof() const;

    ~TemporaryFileStream();

private:
    void updateAllocAndCheck();

    /// Release everything, close reader and writer, delete file
    void release();

    TemporaryDataOnDisk * parent;

    Block header;

    /// Data can be stored in file directly or in the cache
    TemporaryFileOnDiskHolder file;
    FileSegmentsHolder segment_holder;

    Stat stat;

    struct OutputWriter;
    std::unique_ptr<OutputWriter> out_writer;

    struct InputReader;
    std::unique_ptr<InputReader> in_reader;
};

adofsauron commented 1 year ago

Data segmentation for parallel aggregation


                    auto many_data = std::make_shared<ManyAggregatedData>(streams);
                    for (size_t j = 0; j < streams; ++j)
                    {
                        auto aggregation_for_set = std::make_shared<AggregatingTransform>(input_header, transform_params_for_set, many_data, j, merge_threads, temporary_data_merge_threads);
                        // For each input stream we have `grouping_sets_size` copies, so port index
                        // for transform #j should skip ports of first (j-1) streams.
                        connect(*ports[i + grouping_sets_size * j], aggregation_for_set->getInputs().front());
                        ports[i + grouping_sets_size * j] = &aggregation_for_set->getOutputs().front();
                        processors.push_back(aggregation_for_set);
                    }

adofsauron commented 1 year ago


2023.02.10 20:37:31.132939 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> AggregatingTransform: Aggregating
2023.02.10 20:37:31.133521 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Aggregation method: serialized
2023.02.10 20:37:31.145087 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> AggregatingTransform: Aggregating
2023.02.10 20:37:31.146234 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Aggregation method: serialized
2023.02.10 20:37:31.212701 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> AggregatingTransform: Aggregating
2023.02.10 20:37:31.212986 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Aggregation method: serialized
2023.02.10 20:37:31.235648 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> AggregatingTransform: Aggregating
2023.02.10 20:37:31.236181 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Aggregation method: serialized
2023.02.10 20:37:31.255100 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> AggregatingTransform: Aggregating
2023.02.10 20:37:31.255397 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Aggregation method: serialized
2023.02.10 20:37:31.257787 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> AggregatingTransform: Aggregating
2023.02.10 20:37:31.257909 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Aggregation method: serialized
2023.02.10 20:37:31.262983 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> AggregatingTransform: Aggregating
2023.02.10 20:37:31.263184 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Aggregation method: serialized
2023.02.10 20:37:31.427304 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> MemoryTracker: Current memory usage (for query): 3.00 GiB.
2023.02.10 20:37:32.005273 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.48 GiB, peak 4.48 GiB, free memory in arenas 78.00 MiB, will set to 4.35 GiB (RSS), difference: -136.76 MiB
2023.02.10 20:37:32.775464 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Converting aggregation data to two-level.
2023.02.10 20:37:32.779167 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Converting aggregation data to two-level.
2023.02.10 20:37:32.810370 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Converting aggregation data to two-level.
2023.02.10 20:37:32.829024 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Converting aggregation data to two-level.
2023.02.10 20:37:32.840999 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Converting aggregation data to two-level.
2023.02.10 20:37:32.872675 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Converting aggregation data to two-level.
2023.02.10 20:37:32.904684 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> Aggregator: Converting aggregation data to two-level.
2023.02.10 20:37:33.002875 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.32 GiB, peak 4.48 GiB, free memory in arenas 132.90 MiB, will set to 4.56 GiB (RSS), difference: 245.42 MiB
2023.02.10 20:37:34.001386 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.25 GiB on local disk `_tmp_default`, having unreserved 39.26 GiB.
2023.02.10 20:37:34.002735 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613aaaaaa
2023.02.10 20:37:34.003989 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.70 GiB, peak 4.73 GiB, free memory in arenas 140.14 MiB, will set to 4.71 GiB (RSS), difference: 5.61 MiB
2023.02.10 20:37:34.060384 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.26 GiB on local disk `_tmp_default`, having unreserved 39.26 GiB.
2023.02.10 20:37:34.062218 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613baaaaa
2023.02.10 20:37:34.065246 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.26 GiB on local disk `_tmp_default`, having unreserved 39.26 GiB.
2023.02.10 20:37:34.066696 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613caaaaa
2023.02.10 20:37:34.105046 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.27 GiB on local disk `_tmp_default`, having unreserved 39.26 GiB.
2023.02.10 20:37:34.107155 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613daaaaa
2023.02.10 20:37:34.137376 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.24 GiB on local disk `_tmp_default`, having unreserved 39.26 GiB.
2023.02.10 20:37:34.138497 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613eaaaaa
2023.02.10 20:37:34.150694 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.23 GiB on local disk `_tmp_default`, having unreserved 39.26 GiB.
2023.02.10 20:37:34.152425 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613faaaaa
2023.02.10 20:37:34.171662 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.23 GiB on local disk `_tmp_default`, having unreserved 39.26 GiB.
2023.02.10 20:37:34.172282 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613gaaaaa
2023.02.10 20:37:35.003550 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.46 GiB, peak 4.73 GiB, free memory in arenas 141.16 MiB, will set to 4.43 GiB (RSS), difference: -34.19 MiB
2023.02.10 20:37:35.623713 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 964 rows, 49.89 KiB.
2023.02.10 20:37:35.635407 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.634 sec., 225234 rows, 11.42 MiB uncompressed, 5.33 MiB compressed, 53.186 uncompressed bytes per row, 24.800 compressed bytes per row, compression rate: 2.145 (137808.606 rows/sec., 6.99 MiB/sec. uncompressed, 3.26 MiB/sec. compressed)
2023.02.10 20:37:35.672945 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 947 rows, 49.01 KiB.
2023.02.10 20:37:35.676381 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 952 rows, 49.27 KiB.
2023.02.10 20:37:35.686735 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.622 sec., 225577 rows, 11.44 MiB uncompressed, 5.40 MiB compressed, 53.186 uncompressed bytes per row, 25.098 compressed bytes per row, compression rate: 2.119 (139053.417 rows/sec., 7.05 MiB/sec. uncompressed, 3.33 MiB/sec. compressed)
2023.02.10 20:37:35.690445 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 966 rows, 50.00 KiB.
2023.02.10 20:37:35.695309 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.635 sec., 225149 rows, 11.42 MiB uncompressed, 5.42 MiB compressed, 53.186 uncompressed bytes per row, 25.249 compressed bytes per row, compression rate: 2.106 (137689.501 rows/sec., 6.98 MiB/sec. uncompressed, 3.32 MiB/sec. compressed)
2023.02.10 20:37:35.697783 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 955 rows, 49.43 KiB.
2023.02.10 20:37:35.700710 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.564 sec., 224876 rows, 11.41 MiB uncompressed, 5.40 MiB compressed, 53.187 uncompressed bytes per row, 25.160 compressed bytes per row, compression rate: 2.114 (143824.545 rows/sec., 7.30 MiB/sec. uncompressed, 3.45 MiB/sec. compressed)
2023.02.10 20:37:35.710832 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.606 sec., 225478 rows, 11.44 MiB uncompressed, 5.39 MiB compressed, 53.186 uncompressed bytes per row, 25.065 compressed bytes per row, compression rate: 2.122 (140396.883 rows/sec., 7.12 MiB/sec. uncompressed, 3.36 MiB/sec. compressed)
2023.02.10 20:37:35.716700 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 957 rows, 49.53 KiB.
2023.02.10 20:37:35.726570 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.576 sec., 224881 rows, 11.41 MiB uncompressed, 5.39 MiB compressed, 53.187 uncompressed bytes per row, 25.144 compressed bytes per row, compression rate: 2.115 (142677.853 rows/sec., 7.24 MiB/sec. uncompressed, 3.42 MiB/sec. compressed)
2023.02.10 20:37:35.737304 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 961 rows, 49.74 KiB.
2023.02.10 20:37:35.746642 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.575 sec., 226129 rows, 11.47 MiB uncompressed, 5.38 MiB compressed, 53.186 uncompressed bytes per row, 24.964 compressed bytes per row, compression rate: 2.131 (143555.250 rows/sec., 7.28 MiB/sec. uncompressed, 3.42 MiB/sec. compressed)
2023.02.10 20:37:36.002963 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 3.75 GiB, peak 4.73 GiB, free memory in arenas 127.23 MiB, will set to 3.91 GiB (RSS), difference: 157.30 MiB
2023.02.10 20:37:37.003561 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.07 GiB, peak 4.73 GiB, free memory in arenas 127.21 MiB, will set to 4.03 GiB (RSS), difference: -44.86 MiB
2023.02.10 20:37:38.004000 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.24 GiB, peak 4.73 GiB, free memory in arenas 127.21 MiB, will set to 4.22 GiB (RSS), difference: -25.07 MiB
2023.02.10 20:37:39.006610 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.50 GiB, peak 4.73 GiB, free memory in arenas 100.41 MiB, will set to 4.45 GiB (RSS), difference: -51.03 MiB
2023.02.10 20:37:39.163363 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.14 GiB on local disk `_tmp_default`, having unreserved 39.22 GiB.
2023.02.10 20:37:39.163826 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613haaaaa
2023.02.10 20:37:39.166541 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.14 GiB on local disk `_tmp_default`, having unreserved 39.22 GiB.
2023.02.10 20:37:39.168046 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613iaaaaa
2023.02.10 20:37:39.185181 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.14 GiB on local disk `_tmp_default`, having unreserved 39.22 GiB.
2023.02.10 20:37:39.186032 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613jaaaaa
2023.02.10 20:37:39.199760 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.14 GiB on local disk `_tmp_default`, having unreserved 39.22 GiB.
2023.02.10 20:37:39.202190 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613kaaaaa
2023.02.10 20:37:39.218797 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.14 GiB on local disk `_tmp_default`, having unreserved 39.22 GiB.
2023.02.10 20:37:39.220288 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613laaaaa
2023.02.10 20:37:39.234810 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.13 GiB on local disk `_tmp_default`, having unreserved 39.22 GiB.
2023.02.10 20:37:39.236573 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613maaaaa
2023.02.10 20:37:39.244683 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.13 GiB on local disk `_tmp_default`, having unreserved 39.22 GiB.
2023.02.10 20:37:39.245990 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613naaaaa
2023.02.10 20:37:40.002968 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.37 GiB, peak 4.73 GiB, free memory in arenas 136.52 MiB, will set to 4.38 GiB (RSS), difference: 9.56 MiB
2023.02.10 20:37:40.670828 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 782 rows, 40.47 KiB.
2023.02.10 20:37:40.677275 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.511 sec., 179166 rows, 9.10 MiB uncompressed, 4.27 MiB compressed, 53.234 uncompressed bytes per row, 24.987 compressed bytes per row, compression rate: 2.130 (118588.282 rows/sec., 6.02 MiB/sec. uncompressed, 2.83 MiB/sec. compressed)
2023.02.10 20:37:40.689326 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 778 rows, 40.27 KiB.
2023.02.10 20:37:40.698772 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.536 sec., 181235 rows, 9.20 MiB uncompressed, 4.32 MiB compressed, 53.232 uncompressed bytes per row, 24.969 compressed bytes per row, compression rate: 2.132 (118008.022 rows/sec., 5.99 MiB/sec. uncompressed, 2.81 MiB/sec. compressed)
2023.02.10 20:37:40.747567 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 783 rows, 40.53 KiB.
2023.02.10 20:37:40.755982 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 778 rows, 40.27 KiB.
2023.02.10 20:37:40.756288 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.571 sec., 180632 rows, 9.17 MiB uncompressed, 4.30 MiB compressed, 53.232 uncompressed bytes per row, 24.965 compressed bytes per row, compression rate: 2.132 (114952.107 rows/sec., 5.84 MiB/sec. uncompressed, 2.74 MiB/sec. compressed)
2023.02.10 20:37:40.762947 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.544 sec., 180871 rows, 9.18 MiB uncompressed, 4.31 MiB compressed, 53.232 uncompressed bytes per row, 24.962 compressed bytes per row, compression rate: 2.133 (117117.113 rows/sec., 5.95 MiB/sec. uncompressed, 2.79 MiB/sec. compressed)
2023.02.10 20:37:40.783162 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 779 rows, 40.32 KiB.
2023.02.10 20:37:40.791678 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.557 sec., 181056 rows, 9.19 MiB uncompressed, 4.31 MiB compressed, 53.232 uncompressed bytes per row, 24.990 compressed bytes per row, compression rate: 2.130 (116272.947 rows/sec., 5.90 MiB/sec. uncompressed, 2.77 MiB/sec. compressed)
2023.02.10 20:37:40.794420 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 784 rows, 40.58 KiB.
2023.02.10 20:37:40.801395 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.602 sec., 180245 rows, 9.15 MiB uncompressed, 4.29 MiB compressed, 53.233 uncompressed bytes per row, 24.975 compressed bytes per row, compression rate: 2.131 (112525.423 rows/sec., 5.71 MiB/sec. uncompressed, 2.68 MiB/sec. compressed)
2023.02.10 20:37:40.808499 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 792 rows, 40.99 KiB.
2023.02.10 20:37:40.814891 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.570 sec., 179589 rows, 9.12 MiB uncompressed, 4.28 MiB compressed, 53.234 uncompressed bytes per row, 25.002 compressed bytes per row, compression rate: 2.129 (114361.607 rows/sec., 5.81 MiB/sec. uncompressed, 2.73 MiB/sec. compressed)
2023.02.10 20:37:41.004178 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 3.80 GiB, peak 4.73 GiB, free memory in arenas 124.26 MiB, will set to 3.90 GiB (RSS), difference: 108.72 MiB
2023.02.10 20:37:42.003161 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.07 GiB, peak 4.73 GiB, free memory in arenas 124.24 MiB, will set to 4.01 GiB (RSS), difference: -54.51 MiB
2023.02.10 20:37:43.003969 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.22 GiB, peak 4.73 GiB, free memory in arenas 118.21 MiB, will set to 4.19 GiB (RSS), difference: -30.12 MiB
2023.02.10 20:37:44.004587 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.47 GiB, peak 4.73 GiB, free memory in arenas 93.00 MiB, will set to 4.42 GiB (RSS), difference: -47.73 MiB
2023.02.10 20:37:44.232463 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.14 GiB on local disk `_tmp_default`, having unreserved 39.19 GiB.
2023.02.10 20:37:44.236322 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613oaaaaa
2023.02.10 20:37:44.286675 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.14 GiB on local disk `_tmp_default`, having unreserved 39.19 GiB.
2023.02.10 20:37:44.287196 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613paaaaa
2023.02.10 20:37:44.315295 [ 1797 ] {} <Debug> DNSResolver: Updating DNS cache
2023.02.10 20:37:44.315497 [ 1797 ] {} <Debug> DNSResolver: Updated DNS cache
2023.02.10 20:37:44.340855 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.14 GiB on local disk `_tmp_default`, having unreserved 39.19 GiB.
2023.02.10 20:37:44.341405 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613qaaaaa
2023.02.10 20:37:44.395594 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.13 GiB on local disk `_tmp_default`, having unreserved 39.19 GiB.
2023.02.10 20:37:44.396113 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613raaaaa
2023.02.10 20:37:44.404822 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.13 GiB on local disk `_tmp_default`, having unreserved 39.19 GiB.
2023.02.10 20:37:44.407145 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613saaaaa
2023.02.10 20:37:44.416001 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.12 GiB on local disk `_tmp_default`, having unreserved 39.19 GiB.
2023.02.10 20:37:44.418103 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613taaaaa
2023.02.10 20:37:44.471882 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.12 GiB on local disk `_tmp_default`, having unreserved 39.19 GiB.
2023.02.10 20:37:44.472391 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613uaaaaa
2023.02.10 20:37:45.004051 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.37 GiB, peak 4.73 GiB, free memory in arenas 136.04 MiB, will set to 4.39 GiB (RSS), difference: 23.79 MiB
2023.02.10 20:37:45.648999 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 776 rows, 40.16 KiB.
2023.02.10 20:37:45.656111 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.424 sec., 180015 rows, 9.14 MiB uncompressed, 4.29 MiB compressed, 53.233 uncompressed bytes per row, 24.984 compressed bytes per row, compression rate: 2.131 (126429.931 rows/sec., 6.42 MiB/sec. uncompressed, 3.01 MiB/sec. compressed)
2023.02.10 20:37:45.787908 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 768 rows, 39.75 KiB.
2023.02.10 20:37:45.795987 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.509 sec., 180286 rows, 9.15 MiB uncompressed, 4.35 MiB compressed, 53.233 uncompressed bytes per row, 25.299 compressed bytes per row, compression rate: 2.104 (119435.798 rows/sec., 6.06 MiB/sec. uncompressed, 2.88 MiB/sec. compressed)
2023.02.10 20:37:45.858919 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 785 rows, 40.63 KiB.
2023.02.10 20:37:45.865490 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.525 sec., 179875 rows, 9.13 MiB uncompressed, 4.28 MiB compressed, 53.233 uncompressed bytes per row, 24.976 compressed bytes per row, compression rate: 2.131 (117960.245 rows/sec., 5.99 MiB/sec. uncompressed, 2.81 MiB/sec. compressed)
2023.02.10 20:37:45.906580 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 775 rows, 40.11 KiB.
2023.02.10 20:37:45.912077 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.517 sec., 179376 rows, 9.11 MiB uncompressed, 4.27 MiB compressed, 53.234 uncompressed bytes per row, 24.966 compressed bytes per row, compression rate: 2.132 (118268.088 rows/sec., 6.00 MiB/sec. uncompressed, 2.82 MiB/sec. compressed)
2023.02.10 20:37:45.970291 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 767 rows, 39.70 KiB.
2023.02.10 20:37:45.976009 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 800 rows, 41.41 KiB.
2023.02.10 20:37:45.976910 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.572 sec., 180338 rows, 9.16 MiB uncompressed, 4.33 MiB compressed, 53.233 uncompressed bytes per row, 25.153 compressed bytes per row, compression rate: 2.116 (114696.222 rows/sec., 5.82 MiB/sec. uncompressed, 2.75 MiB/sec. compressed)
2023.02.10 20:37:45.984986 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.569 sec., 180442 rows, 9.16 MiB uncompressed, 4.30 MiB compressed, 53.233 uncompressed bytes per row, 25.008 compressed bytes per row, compression rate: 2.129 (114977.647 rows/sec., 5.84 MiB/sec. uncompressed, 2.74 MiB/sec. compressed)
2023.02.10 20:37:46.003618 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 3.87 GiB, peak 4.73 GiB, free memory in arenas 125.11 MiB, will set to 3.95 GiB (RSS), difference: 88.39 MiB
2023.02.10 20:37:46.004664 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 785 rows, 40.63 KiB.
2023.02.10 20:37:46.016651 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.545 sec., 180275 rows, 9.15 MiB uncompressed, 4.29 MiB compressed, 53.233 uncompressed bytes per row, 24.966 compressed bytes per row, compression rate: 2.132 (116686.123 rows/sec., 5.92 MiB/sec. uncompressed, 2.78 MiB/sec. compressed)
2023.02.10 20:37:47.003911 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.00 GiB, peak 4.73 GiB, free memory in arenas 123.44 MiB, will set to 3.96 GiB (RSS), difference: -35.08 MiB
2023.02.10 20:37:48.003410 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.10 GiB, peak 4.73 GiB, free memory in arenas 117.40 MiB, will set to 4.09 GiB (RSS), difference: -19.71 MiB
2023.02.10 20:37:48.996628 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 2.99 GiB on local disk `_tmp_default`, having unreserved 39.16 GiB.
2023.02.10 20:37:48.999196 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613vaaaaa
2023.02.10 20:37:49.005089 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.36 GiB, peak 4.73 GiB, free memory in arenas 99.82 MiB, will set to 4.34 GiB (RSS), difference: -19.97 MiB
2023.02.10 20:37:49.573831 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.12 GiB on local disk `_tmp_default`, having unreserved 39.16 GiB.
2023.02.10 20:37:49.575041 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613waaaaa
2023.02.10 20:37:49.810581 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.12 GiB on local disk `_tmp_default`, having unreserved 39.16 GiB.
2023.02.10 20:37:49.811255 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613xaaaaa
2023.02.10 20:37:49.853701 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.12 GiB on local disk `_tmp_default`, having unreserved 39.16 GiB.
2023.02.10 20:37:49.855350 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613yaaaaa
2023.02.10 20:37:49.882184 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.12 GiB on local disk `_tmp_default`, having unreserved 39.16 GiB.
2023.02.10 20:37:49.883280 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.11 GiB on local disk `_tmp_default`, having unreserved 39.16 GiB.
2023.02.10 20:37:49.884476 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613zaaaaa
2023.02.10 20:37:49.885926 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613abaaaa
2023.02.10 20:37:49.919542 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.11 GiB on local disk `_tmp_default`, having unreserved 39.16 GiB.
2023.02.10 20:37:49.920430 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613bbaaaa
2023.02.10 20:37:50.004006 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.45 GiB, peak 4.73 GiB, free memory in arenas 136.23 MiB, will set to 4.44 GiB (RSS), difference: -9.73 MiB
2023.02.10 20:37:50.573485 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 783 rows, 40.53 KiB.
2023.02.10 20:37:50.580516 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.584 sec., 180225 rows, 9.15 MiB uncompressed, 4.29 MiB compressed, 53.233 uncompressed bytes per row, 24.974 compressed bytes per row, compression rate: 2.132 (113754.239 rows/sec., 5.77 MiB/sec. uncompressed, 2.71 MiB/sec. compressed)
2023.02.10 20:37:51.003710 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.24 GiB, peak 4.73 GiB, free memory in arenas 134.98 MiB, will set to 4.26 GiB (RSS), difference: 15.66 MiB
2023.02.10 20:37:51.055345 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 783 rows, 40.53 KiB.
2023.02.10 20:37:51.065132 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.491 sec., 180996 rows, 9.19 MiB uncompressed, 4.31 MiB compressed, 53.232 uncompressed bytes per row, 24.969 compressed bytes per row, compression rate: 2.132 (121353.009 rows/sec., 6.16 MiB/sec. uncompressed, 2.89 MiB/sec. compressed)
2023.02.10 20:37:51.288408 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 784 rows, 40.58 KiB.
2023.02.10 20:37:51.295473 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.485 sec., 179344 rows, 9.10 MiB uncompressed, 4.33 MiB compressed, 53.234 uncompressed bytes per row, 25.313 compressed bytes per row, compression rate: 2.103 (120765.782 rows/sec., 6.13 MiB/sec. uncompressed, 2.92 MiB/sec. compressed)
2023.02.10 20:37:51.377213 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 784 rows, 40.58 KiB.
2023.02.10 20:37:51.383646 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 774 rows, 40.06 KiB.
2023.02.10 20:37:51.386705 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.505 sec., 180690 rows, 9.17 MiB uncompressed, 4.30 MiB compressed, 53.232 uncompressed bytes per row, 24.958 compressed bytes per row, compression rate: 2.133 (120077.044 rows/sec., 6.10 MiB/sec. uncompressed, 2.86 MiB/sec. compressed)
2023.02.10 20:37:51.391441 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.538 sec., 179228 rows, 9.10 MiB uncompressed, 4.30 MiB compressed, 53.234 uncompressed bytes per row, 25.162 compressed bytes per row, compression rate: 2.116 (116531.085 rows/sec., 5.92 MiB/sec. uncompressed, 2.80 MiB/sec. compressed)
2023.02.10 20:37:51.401773 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 782 rows, 40.47 KiB.
2023.02.10 20:37:51.408625 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.489 sec., 180680 rows, 9.17 MiB uncompressed, 4.31 MiB compressed, 53.232 uncompressed bytes per row, 24.995 compressed bytes per row, compression rate: 2.130 (121311.116 rows/sec., 6.16 MiB/sec. uncompressed, 2.89 MiB/sec. compressed)
2023.02.10 20:37:51.416294 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 764 rows, 39.54 KiB.
2023.02.10 20:37:51.422094 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.539 sec., 179923 rows, 9.13 MiB uncompressed, 4.32 MiB compressed, 53.233 uncompressed bytes per row, 25.181 compressed bytes per row, compression rate: 2.114 (116899.583 rows/sec., 5.93 MiB/sec. uncompressed, 2.81 MiB/sec. compressed)
2023.02.10 20:37:52.003859 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 3.86 GiB, peak 4.73 GiB, free memory in arenas 116.44 MiB, will set to 3.93 GiB (RSS), difference: 67.98 MiB
2023.02.10 20:37:53.002999 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.11 GiB, peak 4.73 GiB, free memory in arenas 110.43 MiB, will set to 4.06 GiB (RSS), difference: -48.56 MiB
2023.02.10 20:37:54.003883 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.28 GiB, peak 4.73 GiB, free memory in arenas 16.57 MiB, will set to 4.27 GiB (RSS), difference: -17.29 MiB
2023.02.10 20:37:54.044228 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 2.93 GiB on local disk `_tmp_default`, having unreserved 39.13 GiB.
2023.02.10 20:37:54.045063 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613cbaaaa
2023.02.10 20:37:54.757869 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.02 GiB on local disk `_tmp_default`, having unreserved 39.13 GiB.
2023.02.10 20:37:54.759505 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613dbaaaa
2023.02.10 20:37:55.004887 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.42 GiB, peak 4.73 GiB, free memory in arenas 24.93 MiB, will set to 4.39 GiB (RSS), difference: -29.13 MiB
2023.02.10 20:37:55.162643 [ 1615 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Information> TCPHandler: Query was cancelled.
2023.02.10 20:37:55.322442 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.10 GiB on local disk `_tmp_default`, having unreserved 39.13 GiB.
2023.02.10 20:37:55.323246 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613ebaaaa
2023.02.10 20:37:55.692368 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.09 GiB on local disk `_tmp_default`, having unreserved 39.12 GiB.
2023.02.10 20:37:55.692916 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613fbaaaa
2023.02.10 20:37:55.696002 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.09 GiB on local disk `_tmp_default`, having unreserved 39.12 GiB.
2023.02.10 20:37:55.697863 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613gbaaaa
2023.02.10 20:37:55.704975 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.09 GiB on local disk `_tmp_default`, having unreserved 39.12 GiB.
2023.02.10 20:37:55.705756 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613hbaaaa
2023.02.10 20:37:55.709936 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Trace> DiskLocal: Reserved 3.09 GiB on local disk `_tmp_default`, having unreserved 39.12 GiB.
2023.02.10 20:37:55.711498 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Writing part of aggregation data into temporary file ./tmp/1613ibaaaa
2023.02.10 20:37:55.825763 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 794 rows, 41.10 KiB.
2023.02.10 20:37:55.831723 [ 1831 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.788 sec., 180747 rows, 9.18 MiB uncompressed, 4.31 MiB compressed, 53.232 uncompressed bytes per row, 24.992 compressed bytes per row, compression rate: 2.130 (101099.002 rows/sec., 5.13 MiB/sec. uncompressed, 2.41 MiB/sec. compressed)
2023.02.10 20:37:56.003683 [ 1827 ] {} <Trace> AsynchronousMetrics: MemoryTracking: was 4.33 GiB, peak 4.73 GiB, free memory in arenas 53.52 MiB, will set to 4.33 GiB (RSS), difference: 2.70 MiB
2023.02.10 20:37:56.286569 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 796 rows, 41.20 KiB.
2023.02.10 20:37:56.292846 [ 1809 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.535 sec., 179932 rows, 9.13 MiB uncompressed, 4.29 MiB compressed, 53.233 uncompressed bytes per row, 24.983 compressed bytes per row, compression rate: 2.131 (117210.093 rows/sec., 5.95 MiB/sec. uncompressed, 2.79 MiB/sec. compressed)
2023.02.10 20:37:56.560494 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 771 rows, 39.91 KiB.
2023.02.10 20:37:56.565874 [ 1806 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.244 sec., 179893 rows, 9.13 MiB uncompressed, 4.29 MiB compressed, 53.233 uncompressed bytes per row, 24.979 compressed bytes per row, compression rate: 2.131 (144631.558 rows/sec., 7.34 MiB/sec. uncompressed, 3.45 MiB/sec. compressed)
2023.02.10 20:37:56.806713 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 775 rows, 40.11 KiB.
2023.02.10 20:37:56.820729 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 795 rows, 41.15 KiB.
2023.02.10 20:37:56.827589 [ 1820 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.136 sec., 179598 rows, 9.12 MiB uncompressed, 4.28 MiB compressed, 53.234 uncompressed bytes per row, 24.996 compressed bytes per row, compression rate: 2.130 (158159.948 rows/sec., 8.03 MiB/sec. uncompressed, 3.77 MiB/sec. compressed)
2023.02.10 20:37:56.837133 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 776 rows, 40.16 KiB.
2023.02.10 20:37:56.840970 [ 1823 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.131 sec., 180683 rows, 9.17 MiB uncompressed, 4.30 MiB compressed, 53.232 uncompressed bytes per row, 24.971 compressed bytes per row, compression rate: 2.132 (159740.939 rows/sec., 8.11 MiB/sec. uncompressed, 3.80 MiB/sec. compressed)
2023.02.10 20:37:56.841142 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Max size of temporary block: 804 rows, 41.61 KiB.
2023.02.10 20:37:56.842784 [ 1812 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.147 sec., 180033 rows, 9.14 MiB uncompressed, 4.29 MiB compressed, 53.233 uncompressed bytes per row, 24.980 compressed bytes per row, compression rate: 2.131 (156947.772 rows/sec., 7.97 MiB/sec. uncompressed, 3.74 MiB/sec. compressed)
2023.02.10 20:37:56.846135 [ 1836 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Debug> Aggregator: Written part in 1.141 sec., 180762 rows, 9.18 MiB uncompressed, 4.30 MiB compressed, 53.232 uncompressed bytes per row, 24.960 compressed bytes per row, compression rate: 2.133 (158368.440 rows/sec., 8.04 MiB/sec. uncompressed, 3.77 MiB/sec. compressed)
2023.02.10 20:37:56.966232 [ 1615 ] {ab3ed4f5-ef6b-4d48-88ea-61515f4fb3a9} <Error> executeQuery: Code: 210. DB::NetException: I/O error: Broken pipe, while writing to socket ([::1]:42420). (NETWORK_ERROR) (version 23.2.1.1) (from [::1]:42420) (in query: select c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice, sum(l_quantity) from customer, orders, lineitem where c_custkey = o_custkey and o_orderkey = l_orderkey group by c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice order by o_totalprice desc, o_orderdate limit 10;), Stack trace (when copying this message, always include the lines below):

adofsauron commented 1 year ago

ExpressionTransform


│ ExpressionTransform                                                │
│   (Limit)                                                          │
│   Limit                                                            │
│     (Sorting)                                                      │
│     MergingSortedTransform 16 → 1                                  │
│       MergeSortingTransform × 16                                   │
│         LimitsCheckingTransform × 16                               │
│           PartialSortingTransform × 16                             │
│             (Expression)                                           │
│             ExpressionTransform × 16                               │
│               (Aggregating)                                        │
│               Resize 8 → 16                                        │
│                 AggregatingTransform × 8                           │
│                   StrictResize 8 → 8                               │
│                     (Expression)                                   │
│                     ExpressionTransform × 8                        │
│                       (Filter)                                     │
│                       FilterTransform × 8                          │
│                         (Join)                                     │
│                         JoiningTransform × 8 2 → 1                 │
│                           (Filter)                                 │
│                           FilterTransform × 8                      │
│                             (Filter)                               │
│                             FilterTransform × 8                    │
│                               (Join)                               │
│                               JoiningTransform × 8 2 → 1           │
│                                 (Expression)                       │
│                                 ExpressionTransform × 8            │
│                                   (ReadFromMergeTree)              │
│                                   MergeTreeThread × 8 0 → 1        │
│                                 (Expression)                       │
│                                 Resize 1 → 8                       │
│                                   FillingRightJoinSide             │
│                                     Resize 16 → 1                  │
│                                       ExpressionTransform × 16     │
│                                         (ReadFromMergeTree)        │
│                                         MergeTreeThread × 16 0 → 1 │
│                           (Expression)                             │
│                           Resize 1 → 8                             │
│                             FillingRightJoinSide                   │
│                               Resize 16 → 1                        │
│                                 ExpressionTransform × 16           │
│                                   (ReadFromMergeTree)              │
│                                   MergeTreeThread × 16 0 → 1       │

adofsauron commented 1 year ago

execute


│ SelectWithUnionQuery (children 1)           │
│  ExpressionList (children 1)                │
│   SelectQuery (children 6)                  │
│    ExpressionList (children 6)              │
│     Identifier c_name                       │
│     Identifier c_custkey                    │
│     Identifier o_orderkey                   │
│     Identifier o_orderdate                  │
│     Identifier o_totalprice                 │
│     Function sum (children 1)               │
│      ExpressionList (children 1)            │
│       Identifier l_quantity                 │
│    TablesInSelectQuery (children 3)         │
│     TablesInSelectQueryElement (children 1) │
│      TableExpression (children 1)           │
│       TableIdentifier customer              │
│     TablesInSelectQueryElement (children 2) │
│      TableExpression (children 1)           │
│       TableIdentifier orders                │
│      TableJoin                              │
│     TablesInSelectQueryElement (children 2) │
│      TableExpression (children 1)           │
│       TableIdentifier lineitem              │
│      TableJoin                              │
│    Function and (children 1)                │
│     ExpressionList (children 2)             │
│      Function equals (children 1)           │
│       ExpressionList (children 2)           │
│        Identifier c_custkey                 │
│        Identifier o_custkey                 │
│      Function equals (children 1)           │
│       ExpressionList (children 2)           │
│        Identifier o_orderkey                │
│        Identifier l_orderkey                │
│    ExpressionList (children 5)              │
│     Identifier c_name                       │
│     Identifier c_custkey                    │
│     Identifier o_orderkey                   │
│     Identifier o_orderdate                  │
│     Identifier o_totalprice                 │
│    ExpressionList (children 2)              │
│     OrderByElement (children 1)             │
│      Identifier o_totalprice                │
│     OrderByElement (children 1)             │
│      Identifier o_orderdate                 │
│    Literal UInt64_10                        │

adofsauron commented 1 year ago

HashJoin: (0x7f2af22cb798) Keys: [(c_custkey) = (o_custkey)]

adofsauron commented 1 year ago

DiskLocal: Reserved 3.26 GiB on local disk _tmp_default, having unreserved 39.26 GiB.

adofsauron commented 1 year ago

Aggregator: Writing part of aggregation data into temporary file ./tmp/5107aaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107baaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107caaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107daaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107eaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107faaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107gaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107haaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107iaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107jaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107kaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107laaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107maaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107naaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107oaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107paaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107qaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107raaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107saaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107taaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107uaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107vaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107waaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107xaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107yaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107zaaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107abaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107bbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107cbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107dbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107ebaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107fbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107gbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107hbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107ibaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107jbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107kbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107lbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107mbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107nbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107obaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107pbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107qbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107rbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107sbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107tbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107ubaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107vbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107wbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107xbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107ybaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107zbaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107acaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107bcaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107ccaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107dcaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107ecaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107fcaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107gcaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107hcaaaa Aggregator: Writing part of aggregation data into temporary file ./tmp/5107icaaaa

adofsauron commented 1 year ago

2023.02.13 10:43:49.004272 [ 5309 ] {} AsynchronousMetrics: MemoryTracking: was 4.50 GiB, peak 4.58 GiB, free memory in arenas 133.41 MiB, will set to 4.58 GiB (RSS), difference: 86.09 MiB

adofsauron commented 1 year ago

Aggregator: Written part in 1.509 sec., 181235 rows, 9.20 MiB uncompressed, 4.32 MiB compressed, 53.232 uncompressed bytes per row, 24.969 compressed bytes per row, compression rate: 2.132 (120128.725 rows/sec., 6.10 MiB/sec. uncompressed, 2.86 MiB/sec. compressed)

adofsauron commented 1 year ago

(gdb) bt

0 DB::Aggregator::writeToTemporaryFile (this=0x7f34322d9210, data_variants=..., max_temp_file_size=3512367045) at ../src/Interpreters/Aggregator.cpp:1603

1 0x000000002aa049cc in DB::Aggregator::executeOnBlock (this=0x7f34322d9210, columns=..., row_begin=0, row_end=900335, result=..., key_columns=..., aggregate_columns=...,

no_more_keys=@0x7f343236c930: false) at ../src/Interpreters/Aggregator.cpp:1585

2 0x000000002d48c813 in DB::AggregatingTransform::consume (this=0x7f343236c818, chunk=...) at ../src/Processors/Transforms/AggregatingTransform.cpp:533

3 0x000000002d48a501 in DB::AggregatingTransform::work (this=0x7f343236c818) at ../src/Processors/Transforms/AggregatingTransform.cpp:492

4 0x000000002d077a03 in DB::executeJob (node=0x7f34323b0700, read_progress_callback=0x7f3432244060) at ../src/Processors/Executors/ExecutionThreadContext.cpp:47

5 0x000000002d077719 in DB::ExecutionThreadContext::executeTask (this=0x7f3494903b60) at ../src/Processors/Executors/ExecutionThreadContext.cpp:92

6 0x000000002d053861 in DB::PipelineExecutor::executeStepImpl (this=0x7f3432291018, thread_num=4, yield_flag=0x0) at ../src/Processors/Executors/PipelineExecutor.cpp:229

7 0x000000002d053b97 in DB::PipelineExecutor::executeSingleThread (this=0x7f3432291018, thread_num=4) at ../src/Processors/Executors/PipelineExecutor.cpp:195

8 0x000000002d055416 in DB::PipelineExecutor::spawnThreads()::$_0::operator()() const (this=0x7f343e15c088) at ../src/Processors/Executors/PipelineExecutor.cpp:320

9 0x000000002d055375 in std::1::invoke[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

10 0x000000002d055321 in std::1::apply_tuple_impl[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&, std::1::__tuple_indices<>) (f=..., __t=...) at ../contrib/llvm-project/libcxx/include/tuple:1789

11 0x000000002d055232 in std::1::apply[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::__1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&) (

__f=..., __t=...) at ../contrib/llvm-project/libcxx/include/tuple:1798

12 0x000000002d05511a in ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}::operator()() (this=0x7f34948e6730) at ../src/Common/ThreadPool.h:210

13 0x000000002d055055 in std::1::invoke[abi:v15000]<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

14 0x000000002d05501d in std::1::invoke_void_return_wrapper<void, true>::call<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&) (args=...) at ../contrib/llvm-project/libcxx/include/__functional/invoke.h:479

15 0x000000002d054ff5 in std::1::function::default_alloc_func<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}, void ()>::operator()[abi:v15000]() (this=0x7f34948e6730) at ../contrib/llvm-project/libcxx/include/functional/function.h:235

16 0x000000002d054fc0 in std::1::function::policy_invoker<void ()>::__call_impl<std::1::function::default_alloc_func<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}, void ()> >(std::1::function::__policy_storage const*) (__buf=0x7f343e15c348)

at ../contrib/llvm-project/libcxx/include/__functional/function.h:716

17 0x000000001a3bf0a6 in std::1::function::__policy_func<void ()>::operator()[abi:v15000]() const (this=0x7f343e15c348)

at ../contrib/llvm-project/libcxx/include/__functional/function.h:848

18 0x000000001a3ba9d5 in std::1::function<void ()>::operator()() const (this=0x7f343e15c348) at ../contrib/llvm-project/libcxx/include/functional/function.h:1187

19 0x000000001a4c7b6e in ThreadPoolImpl::worker (this=0x7f356534b280, thread_it=...) at ../src/Common/ThreadPool.cpp:315

20 0x000000001a4cf3e4 in ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}::operator()() const

(this=0x7f34a25c4008) at ../src/Common/ThreadPool.cpp:145

21 0x000000001a4cf375 in std::1::invoke[abi:v15000]<ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

22 0x000000001a4cf2a5 in std::1::thread_execute[abi:v15000]<std::1::unique_ptr<std::1::thread_struct, std::1::default_delete >, ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}>(std::1::tuple<std::1::unique_ptr<std::1::thread_struct, std::1::default_delete >, ThreadPoolImpl<std::1::thread>::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}>&, std::1::tuple_indices<>) (__t=...) at ../contrib/llvm-project/libcxx/include/thread:284

23 0x000000001a4cec02 in std::1::thread_proxy[abi:v15000]<std::1::tuple<std::1::unique_ptr<std::1::thread_struct, std::1::default_delete >, ThreadPoolImpl<std::1::thread>::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}> >(void*) (__vp=0x7f34a25c4000)

at ../contrib/llvm-project/libcxx/include/thread:295

24 0x00007f356620e802 in start_thread () from /lib64/libc.so.6

25 0x00007f35661ae450 in clone3 () from /lib64/libc.so.6

adofsauron commented 1 year ago

/** Flush data to disk if too much RAM is consumed.
  * Data can only be flushed to disk if a two-level aggregation structure is used.
  */
if (params.max_bytes_before_external_group_by
    && result.isTwoLevel()
    && current_memory_usage > static_cast<Int64>(params.max_bytes_before_external_group_by)
    && worth_convert_to_two_level)
{
    size_t size = current_memory_usage + params.min_free_disk_space;
    writeToTemporaryFile(result, size);
}

adofsauron commented 1 year ago

keys_size = 5, aggregates_size = 1, overflow_row = false, max_rows_to_group_by = 0, group_by_overflow_mode = DB::OverflowMode::THROW, group_by_two_level_threshold = 100000, group_by_two_level_threshold_bytes = 50000000, max_bytes_before_external_group_by = 1073741824, empty_result_for_aggregation_by_empty_set = false, tmp_data_scope = { _ptr = 0x7f3432219f58, _cntrl = 0x7f3432219f40 }, max_threads = 16, min_free_disk_space = 0, compile_aggregate_expressions = true, min_count_to_compile_aggregate_expression = 3, max_block_size = 65409, only_merge = false, enable_prefetch = true, --Type for more, q to quit, c to continue without paging-- stats_collecting_params = { key = 11662658185615503564, max_entries_for_hash_table_stats = 10000, max_size_to_preallocate_for_aggregation = 100000000 }

adofsauron commented 1 year ago

/** Data manipulating functions. */

/** Create empty data for aggregation with `placement new` at the specified location.
  * You will have to destroy them using the `destroy` method.
  */
virtual void create(AggregateDataPtr __restrict place) const = 0;

/// Delete data for aggregation.
virtual void destroy(AggregateDataPtr __restrict place) const noexcept = 0;

adofsauron commented 1 year ago

When aggregation operations are performed, the memory of the data results of all aggregation functions in a single row is allocated through the memory pool in the Aggregator. The constructor is called by createAggregateStates, which in turn calls the create method of each aggregate function

adofsauron commented 1 year ago

│ PROJECTION COLUMNS │ │ c_name String │ │ c_custkey Int32 │ │ o_orderkey Int32 │ │ o_orderdate Date │ │ o_totalprice Decimal(15, 2) │ │ sum(l_quantity) Decimal(38, 2) │ │ PROJECTION │ │ LIST id: 1, nodes: 6 │ │ COLUMN id: 2, column_name: c_name, result_type: String, source_id: 3 │ │ COLUMN id: 4, column_name: c_custkey, result_type: Int32, source_id: 3 │ │ COLUMN id: 5, column_name: o_orderkey, result_type: Int32, source_id: 6 │ │ COLUMN id: 7, column_name: o_orderdate, result_type: Date, source_id: 6 │ │ COLUMN id: 8, column_name: o_totalprice, result_type: Decimal(15, 2), source_id: 6 │ │ FUNCTION id: 9, function_name: sum, function_type: aggregate, result_type: Decimal(38, 2) │ │ ARGUMENTS │ │ LIST id: 10, nodes: 1 │ │ COLUMN id: 11, column_name: l_quantity, result_type: Decimal(15, 2), source_id: 12 │ │ JOIN TREE │ │ JOIN id: 13, kind: COMMA │ │ LEFT TABLE EXPRESSION │ │ JOIN id: 14, kind: COMMA │ │ LEFT TABLE EXPRESSION │ │ TABLE id: 3, table_name: tpch.customer │ │ RIGHT TABLE EXPRESSION │ │ TABLE id: 6, table_name: tpch.orders │ │ RIGHT TABLE EXPRESSION │ │ TABLE id: 12, table_name: tpch.lineitem │ │ WHERE │ │ FUNCTION id: 15, function_name: and, function_type: ordinary, result_type: UInt8 │ │ ARGUMENTS │ │ LIST id: 16, nodes: 2 │ │ FUNCTION id: 17, function_name: equals, function_type: ordinary, result_type: UInt8 │ │ ARGUMENTS │ │ LIST id: 18, nodes: 2 │ │ COLUMN id: 4, column_name: c_custkey, result_type: Int32, source_id: 3 │ │ COLUMN id: 19, column_name: o_custkey, result_type: Int32, source_id: 6 │ │ FUNCTION id: 20, function_name: equals, function_type: ordinary, result_type: UInt8 │ │ ARGUMENTS │ │ LIST id: 21, nodes: 2 │ │ COLUMN id: 5, column_name: o_orderkey, result_type: Int32, source_id: 6 │ │ COLUMN id: 22, column_name: l_orderkey, result_type: Int32, source_id: 12 │ │ GROUP BY │ │ LIST id: 23, nodes: 5 │ │ COLUMN id: 2, column_name: c_name, result_type: String, source_id: 3 │ │ COLUMN id: 4, column_name: c_custkey, result_type: Int32, source_id: 3 │ │ COLUMN id: 5, column_name: o_orderkey, result_type: Int32, source_id: 6 │ │ COLUMN id: 7, column_name: o_orderdate, result_type: Date, source_id: 6 │ │ COLUMN id: 8, column_name: o_totalprice, result_type: Decimal(15, 2), source_id: 6 │ │ ORDER BY │ │ LIST id: 24, nodes: 2 │ │ SORT id: 25, sort_direction: DESCENDING, with_fill: 0 │ │ EXPRESSION │ │ COLUMN id: 8, column_name: o_totalprice, result_type: Decimal(15, 2), source_id: 6 │ │ SORT id: 26, sort_direction: ASCENDING, with_fill: 0 │ │ EXPRESSION │ │ COLUMN id: 7, column_name: o_orderdate, result_type: Date, source_id: 6 │ │ LIMIT │ │ CONSTANT id: 27, constant_value: UInt64_10, constant_value_type: UInt8 │

adofsauron commented 1 year ago

│ Expression (Projection) │ │ Header: c_name String │ │ c_custkey Int32 │ │ o_orderkey Int32 │ │ o_orderdate Date │ │ o_totalprice Decimal(15, 2) │ │ sum(l_quantity) Decimal(38, 2) │ │ Actions: INPUT :: 0 -> c_name String : 0 │ │ INPUT :: 1 -> c_custkey Int32 : 1 │ │ INPUT :: 2 -> o_orderkey Int32 : 2 │ │ INPUT :: 3 -> o_orderdate Date : 3 │ │ INPUT :: 4 -> o_totalprice Decimal(15, 2) : 4 │ │ INPUT :: 5 -> sum(l_quantity) Decimal(38, 2) : 5 │ │ Positions: 0 1 2 3 4 5 │ │ Limit (preliminary LIMIT (without OFFSET)) │ │ Header: c_name String │ │ c_custkey Int32 │ │ o_orderkey Int32 │ │ o_orderdate Date │ │ o_totalprice Decimal(15, 2) │ │ sum(l_quantity) Decimal(38, 2) │ │ Limit 10 │ │ Offset 0 │ │ Sorting (Sorting for ORDER BY) │ │ Header: c_name String │ │ c_custkey Int32 │ │ o_orderkey Int32 │ │ o_orderdate Date │ │ o_totalprice Decimal(15, 2) │ │ sum(l_quantity) Decimal(38, 2) │ │ Sort description: o_totalprice DESC, o_orderdate ASC │ │ Limit 10 │ │ Expression (Before ORDER BY) │ │ Header: c_name String │ │ c_custkey Int32 │ │ o_orderkey Int32 │ │ o_orderdate Date │ │ o_totalprice Decimal(15, 2) │ │ sum(l_quantity) Decimal(38, 2) │ │ Actions: INPUT :: 0 -> c_name String : 0 │ │ INPUT :: 1 -> c_custkey Int32 : 1 │ │ INPUT :: 2 -> o_orderkey Int32 : 2 │ │ INPUT :: 3 -> o_orderdate Date : 3 │ │ INPUT :: 4 -> o_totalprice Decimal(15, 2) : 4 │ │ INPUT :: 5 -> sum(l_quantity) Decimal(38, 2) : 5 │ │ Positions: 0 1 2 3 4 5 │ │ Aggregating │ │ Header: c_name String │ │ c_custkey Int32 │ │ o_orderkey Int32 │ │ o_orderdate Date │ │ o_totalprice Decimal(15, 2) │ │ sum(l_quantity) Decimal(38, 2) │ │ Keys: c_name, c_custkey, o_orderkey, o_orderdate, o_totalprice │ │ Aggregates: │ │ sum(l_quantity) │ │ Function: sum(Decimal(15, 2)) → Decimal(38, 2) │ │ Arguments: l_quantity │ │ Expression (Before GROUP BY) │ │ Header: c_custkey Int32 │ │ c_name String │ │ o_totalprice Decimal(15, 2) │ │ o_orderdate Date │ │ o_orderkey Int32 │ │ l_quantity Decimal(15, 2) │ │ Actions: INPUT :: 0 -> c_custkey Int32 : 0 │ │ INPUT :: 1 -> c_name String : 1 │ │ INPUT :: 2 -> o_totalprice Decimal(15, 2) : 2 │ │ INPUT :: 3 -> o_orderdate Date : 3 │ │ INPUT :: 4 -> o_orderkey Int32 : 4 │ │ INPUT :: 5 -> l_quantity Decimal(15, 2) : 5 │ │ Positions: 0 1 2 3 4 5 │ │ Filter (WHERE) │ │ Header: c_custkey Int32 │ │ c_name String │ │ o_totalprice Decimal(15, 2) │ │ o_orderdate Date │ │ o_orderkey Int32 │ │ l_quantity Decimal(15, 2) │ │ Filter column: and(equals(c_custkey, o_custkey), equals(o_orderkey, l_orderkey)) (removed) │ │ Actions: INPUT :: 0 -> c_custkey Int32 : 0 │ │ INPUT :: 1 -> c_name String : 1 │ │ INPUT :: 2 -> o_custkey Int32 : 2 │ │ INPUT :: 3 -> o_totalprice Decimal(15, 2) : 3 │ │ INPUT :: 4 -> o_orderdate Date : 4 │ │ INPUT : 5 -> o_orderkey Int32 : 5 │ │ INPUT : 6 -> l_orderkey Int32 : 6 │ │ INPUT :: 7 -> l_quantity Decimal(15, 2) : 7 │ │ FUNCTION equals(o_orderkey : 5, l_orderkey :: 6) -> equals(o_orderkey, l_orderkey) UInt8 : 8 │ │ ALIAS equals(o_orderkey, l_orderkey) :: 8 -> and(equals(c_custkey, o_custkey), equals(o_orderkey, l_orderkey)) UInt8 : 6 │ │ Positions: 0 1 3 4 5 7 6 │ │ Join (JOIN FillRightFirst) │ │ Header: c_custkey Int32 │ │ c_name String │ │ o_custkey Int32 │ │ o_totalprice Decimal(15, 2) │ │ o_orderdate Date │ │ o_orderkey Int32 │ │ l_quantity Decimal(15, 2) │ │ l_orderkey Int32 │ │ Filter (( + (Before JOIN + (Projection + Before ORDER BY)))) │ │ Header: c_custkey Int32 │ │ c_name String │ │ o_custkey Int32 │ │ o_totalprice Decimal(15, 2) │ │ o_orderdate Date │ │ o_orderkey Int32 │ │ Filter column: equals(c_custkey, o_custkey) (removed) │ │ Actions: INPUT : 0 -> c_custkey Int32 : 0 │ │ INPUT :: 1 -> c_name String : 1 │ │ INPUT :: 2 -> o_orderkey Int32 : 2 │ │ INPUT : 3 -> o_custkey Int32 : 3 │ │ INPUT :: 4 -> o_totalprice Decimal(15, 2) : 4 │ │ INPUT :: 5 -> o_orderdate Date : 5 │ │ FUNCTION equals(c_custkey : 0, o_custkey : 3) -> equals(c_custkey, o_custkey) UInt8 : 6 │ │ Positions: 6 0 1 3 4 5 2 │ │ Filter (WHERE) │ │ Header: c_custkey Int32 │ │ c_name String │ │ o_orderkey Int32 │ │ o_custkey Int32 │ │ o_totalprice Decimal(15, 2) │ │ o_orderdate Date │ │ Filter column: equals(c_custkey, o_custkey) (removed) │ │ Actions: INPUT : 0 -> c_custkey Int32 : 0 │ │ INPUT :: 1 -> c_name String : 1 │ │ INPUT :: 2 -> o_orderkey Int32 : 2 │ │ INPUT : 3 -> o_custkey Int32 : 3 │ │ INPUT :: 4 -> o_totalprice Decimal(15, 2) : 4 │ │ INPUT :: 5 -> o_orderdate Date : 5 │ │ FUNCTION equals(c_custkey : 0, o_custkey : 3) -> equals(c_custkey, o_custkey) UInt8 : 6 │ │ Positions: 0 1 2 3 4 5 6 │ │ Join (JOIN FillRightFirst) │ │ Header: c_custkey Int32 │ │ c_name String │ │ o_orderkey Int32 │ │ o_orderdate Date │ │ o_totalprice Decimal(15, 2) │ │ o_custkey Int32 │ │ Expression (Before JOIN) │ │ Header: c_custkey Int32 │ │ c_name String │ │ Actions: INPUT :: 0 -> c_custkey Int32 : 0 │ │ INPUT :: 1 -> c_name String : 1 │ │ Positions: 0 1 │ │ ReadFromMergeTree (tpch.customer) │ │ Header: c_custkey Int32 │ │ c_name String │ │ ReadType: Default │ │ Parts: 2 │ │ Granules: 184 │ │ Indexes: │ │ PrimaryKey │ │ Condition: true │ │ Parts: 2/2 │ │ Granules: 184/184 │ │ Expression ((Joined actions + (Rename joined columns + (Projection + Before ORDER BY)))) │ │ Header: o_custkey Int32 │ │ o_orderkey Int32 │ │ o_orderdate Date │ │ o_totalprice Decimal(15, 2) │ │ Actions: INPUT :: 0 -> o_orderkey Int32 : 0 │ │ INPUT :: 1 -> o_custkey Int32 : 1 │ │ INPUT :: 2 -> o_totalprice Decimal(15, 2) : 2 │ │ INPUT :: 3 -> o_orderdate Date : 3 │ │ Positions: 1 0 3 2 │ │ ReadFromMergeTree (tpch.orders) │ │ Header: o_orderkey Int32 │ │ o_custkey Int32 │ │ o_totalprice Decimal(15, 2) │ │ o_orderdate Date │ │ ReadType: Default │ │ Parts: 5 │ │ Granules: 1835 │ │ Indexes: │ │ PrimaryKey │ │ Condition: true │ │ Parts: 5/5 │ │ Granules: 1835/1835 │ │ Expression ((Joined actions + (Rename joined columns + (Projection + Before ORDER BY)))) │ │ Header: l_orderkey Int32 │ │ l_quantity Decimal(15, 2) │ │ Actions: INPUT :: 0 -> l_orderkey Int32 : 0 │ │ INPUT :: 1 -> l_quantity Decimal(15, 2) : 1 │ │ Positions: 0 1 │ │ ReadFromMergeTree (tpch.lineitem) │ │ Header: l_orderkey Int32 │ │ l_quantity Decimal(15, 2) │ │ ReadType: Default │ │ Parts: 1 │ │ Granules: 4057 │ │ Indexes: │ │ PrimaryKey │ │ Condition: true │ │ Parts: 1/1 │ │ Granules: 4057/4057 │

adofsauron commented 1 year ago

If you restrict the nodes that produce two_level blocks to the order in which block_num increments, you can reduce memory usage because the data that needs to be merged must be in the same combination, and when you see a new block_num, it means that all previous merging operations have been completed

adofsauron commented 1 year ago

auto & out_stream = tmp_data->createStream(getHeader(false), max_temp_file_size);
ProfileEvents::increment(ProfileEvents::ExternalAggregationWritePart);

LOG_DEBUG(log, "Writing part of aggregation data into temporary file {}", out_stream.getPath());

/// Flush only two-level data and possibly overflow data.

define M(NAME) \

else if (data_variants.type == AggregatedDataVariants::Type::NAME) \
    writeToTemporaryFileImpl(data_variants, *data_variants.NAME, out_stream);

if (false) {} // NOLINT
APPLY_FOR_VARIANTS_TWO_LEVEL(M)

undef M

else
    throw Exception(ErrorCodes::UNKNOWN_AGGREGATED_DATA_VARIANT, "Unknown aggregated data variant");

/// NOTE Instead of freeing up memory and creating new hash tables and arenas, you can re-use the old ones.
data_variants.init(data_variants.type);
data_variants.aggregates_pools = Arenas(1, std::make_shared<Arena>());
data_variants.aggregates_pool = data_variants.aggregates_pools.back().get();
if (params.overflow_row || data_variants.type == AggregatedDataVariants::Type::without_key)
{
    AggregateDataPtr place = data_variants.aggregates_pool->alignedAlloc(total_size_of_aggregate_states, align_aggregate_states);
    createAggregateStates(place);
    data_variants.without_key = place;
}

auto stat = out_stream.finishWriting();

adofsauron commented 1 year ago

TemporaryFileStream & TemporaryDataOnDisk::createStreamToRegularFile(const Block & header, size_t max_file_size) { if (!volume) throw Exception(ErrorCodes::LOGICAL_ERROR, "TemporaryDataOnDiskScope has no volume");

DiskPtr disk;
if (max_file_size > 0)
{
    auto reservation = volume->reserve(max_file_size);
    if (!reservation)
        throw Exception(ErrorCodes::NOT_ENOUGH_SPACE, "Not enough space on temporary disk");
    disk = reservation->getDisk();
}
else
{
    disk = volume->getDisk();
}

auto tmp_file = std::make_unique<TemporaryFileOnDisk>(disk, current_metric_scope);

std::lock_guard lock(mutex);
TemporaryFileStreamPtr & tmp_stream = streams.emplace_back(std::make_unique<TemporaryFileStream>(std::move(tmp_file), header, this));
return *tmp_stream;

}

adofsauron commented 1 year ago

Blocks of the same block_num of multiple nodes performing pre-aggregation are combined so that different combinations can perform the merge operation in parallel

adofsauron commented 1 year ago

    const auto & tmp_data = params->aggregator.getTemporaryData();

    Pipe pipe;
    {
        Pipes pipes;

        for (auto * tmp_stream : tmp_data.getStreams())
            pipes.emplace_back(Pipe(std::make_unique<SourceFromNativeStream>(tmp_stream)));

        pipe = Pipe::unitePipes(std::move(pipes));
    }

adofsauron commented 1 year ago


#0  DB::NativeWriter::write (this=0x7f59040960a0, block=...) at ../src/Formats/NativeWriter.cpp:69
#1  0x000000002ba862da in DB::TemporaryFileStream::OutputWriter::write (this=0x7f5904096000, block=...) at ../src/Interpreters/TemporaryDataOnDisk.cpp:135
#2  0x000000002ba84711 in DB::TemporaryFileStream::write (this=0x7f5972f69100, block=...) at ../src/Interpreters/TemporaryDataOnDisk.cpp:239
#3  0x000000002aa93513 in DB::Aggregator::writeToTemporaryFileImpl<DB::AggregationMethodSerialized<TwoLevelHashMapTable<StringRef, HashMapCellWithSavedHash<StringRef, char*, DefaultHash<StringRef>, HashTableNoState>, DefaultHash<StringRef>, TwoLevelHashTableGrower<8ul>, Allocator<true, true>, HashMapTable> > > (this=0x7f5905a6ae10, data_variants=..., method=..., out=...)
    at ../src/Interpreters/Aggregator.cpp:1722
#4  0x000000002aa0562e in DB::Aggregator::writeToTemporaryFile (this=0x7f5905a6ae10, data_variants=..., max_temp_file_size=3515327323) at ../src/Interpreters/Aggregator.cpp:1612
#5  0x000000002aa049cc in DB::Aggregator::executeOnBlock (this=0x7f5905a6ae10, columns=..., row_begin=0, row_end=898603, result=..., key_columns=..., aggregate_columns=..., 
    no_more_keys=@0x7f5905aa5b30: false) at ../src/Interpreters/Aggregator.cpp:1585
#6  0x000000002d48c813 in DB::AggregatingTransform::consume (this=0x7f5905aa5a18, chunk=...) at ../src/Processors/Transforms/AggregatingTransform.cpp:533
#7  0x000000002d48a501 in DB::AggregatingTransform::work (this=0x7f5905aa5a18) at ../src/Processors/Transforms/AggregatingTransform.cpp:492
#8  0x000000002d077a03 in DB::executeJob (node=0x7f5905b14a00, read_progress_callback=0x7f5905af4900) at ../src/Processors/Executors/ExecutionThreadContext.cpp:47
#9  0x000000002d077719 in DB::ExecutionThreadContext::executeTask (this=0x7f596e8f5140) at ../src/Processors/Executors/ExecutionThreadContext.cpp:92
#10 0x000000002d053861 in DB::PipelineExecutor::executeStepImpl (this=0x7f5905ab5618, thread_num=12, yield_flag=0x0) at ../src/Processors/Executors/PipelineExecutor.cpp:229
#11 0x000000002d053b97 in DB::PipelineExecutor::executeSingleThread (this=0x7f5905ab5618, thread_num=12) at ../src/Processors/Executors/PipelineExecutor.cpp:195
#12 0x000000002d055416 in DB::PipelineExecutor::spawnThreads()::$_0::operator()() const (this=0x7f590e817088) at ../src/Processors/Executors/PipelineExecutor.cpp:320
#13 0x000000002d055375 in std::__1::__invoke[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&> (__f=...) at ../contrib/llvm-project/libcxx/include/__functional/invoke.h:394
#14 0x000000002d055321 in std::__1::__apply_tuple_impl[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::__1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::__1::tuple<>&, std::__1::__tuple_indices<>) (__f=..., __t=...) at ../contrib/llvm-project/libcxx/include/tuple:1789
#15 0x000000002d055232 in std::__1::apply[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::__1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::__1::tuple<>&) (
    __f=..., __t=...) at ../contrib/llvm-project/libcxx/include/tuple:1798
#16 0x000000002d05511a in ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}::operator()() (this=0x7f596e8c9130) at ../src/Common/ThreadPool.h:210
#17 0x000000002d055055 in std::__1::__invoke[abi:v15000]<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&> (__f=...) at ../contrib/llvm-project/libcxx/include/__functional/invoke.h:394
#18 0x000000002d05501d in std::__1::__invoke_void_return_wrapper<void, true>::__call<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&>(ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&) (__args=...) at ../contrib/llvm-project/libcxx/include/__functional/invoke.h:479
#19 0x000000002d054ff5 in std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}, void ()>::operator()[abi:v15000]() (this=0x7f596e8c9130) at ../contrib/llvm-project/libcxx/include/__functional/function.h:235
#20 0x000000002d054fc0 in std::__1::__function::__policy_invoker<void ()>::__call_impl<std::__1::__function::__default_alloc_func<ThreadFromGlobalPoolImpl<true>::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}, void ()> >(std::__1::__function::__policy_storage const*) (__buf=0x7f590e817348)
    at ../contrib/llvm-project/libcxx/include/__functional/function.h:716
#21 0x000000001a3bf0a6 in std::__1::__function::__policy_func<void ()>::operator()[abi:v15000]() const (this=0x7f590e817348)
    at ../contrib/llvm-project/libcxx/include/__functional/function.h:848
#22 0x000000001a3ba9d5 in std::__1::function<void ()>::operator()() const (this=0x7f590e817348) at ../contrib/llvm-project/libcxx/include/__functional/function.h:1187
#23 0x000000001a4c7b6e in ThreadPoolImpl<std::__1::thread>::worker (this=0x7f5a3e34b280, thread_it=...) at ../src/Common/ThreadPool.cpp:315
#24 0x000000001a4cf3e4 in ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::{lambda()#2}::operator()() const
    (this=0x7f597027f328) at ../src/Common/ThreadPool.cpp:145
#25 0x000000001a4cf375 in std::__1::__invoke[abi:v15000]<ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::{lambda()#2}> (__f=...) at ../contrib/llvm-project/libcxx/include/__functional/invoke.h:394
#26 0x000000001a4cf2a5 in std::__1::__thread_execute[abi:v15000]<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::{lambda()#2}>(std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::{lambda()--Type <RET> for more, q to quit, c to continue without paging--
#2}>&, std::__1::__tuple_indices<>) (__t=...) at ../contrib/llvm-project/libcxx/include/thread:284
#27 0x000000001a4cec02 in std::__1::__thread_proxy[abi:v15000]<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, ThreadPoolImpl<std::__1::thread>::scheduleImpl<void>(std::__1::function<void ()>, long, std::__1::optional<unsigned long>, bool)::{lambda()#2}> >(void*) (__vp=0x7f597027f320)
    at ../contrib/llvm-project/libcxx/include/thread:295
#28 0x00007f5a3f1ae802 in start_thread () from /lib64/libc.so.6
#29 0x00007f5a3f14e450 in clone3 () from /lib64/libc.so.6

adofsauron commented 1 year ago

0 ./24773aaaaaa 0 ./24773baaaaa 0 ./25102aaaaaa 4412 ./25102abaaaa 5532 ./25102baaaaa 4392 ./25102bbaaaa 5456 ./25102caaaaa 4412 ./25102cbaaaa 5516 ./25102daaaaa 4404 ./25102dbaaaa 5524 ./25102eaaaaa 4392 ./25102ebaaaa 5552 ./25102faaaaa 5520 ./25102gaaaaa 4400 ./25102haaaaa 4372 ./25102iaaaaa 4388 ./25102jaaaaa 4412 ./25102kaaaaa 4420 ./25102laaaaa 4404 ./25102maaaaa 4420 ./25102naaaaa 4376 ./25102oaaaaa 4456 ./25102paaaaa 4396 ./25102qaaaaa 4388 ./25102raaaaa 4408 ./25102saaaaa 4432 ./25102taaaaa 4396 ./25102uaaaaa 4428 ./25102vaaaaa 4396 ./25102waaaaa 4408 ./25102xaaaaa 4416 ./25102yaaaaa 4436 ./25102zaaaaa 5524 ./25369aaaaaa 4416 ./25369abaaaa 560 ./25369acaaaa 5528 ./25369baaaaa 4404 ./25369bbaaaa 5516 ./25369caaaaa 4408 ./25369cbaaaa 5552 ./25369daaaaa 4384 ./25369dbaaaa 5532 ./25369eaaaaa 4392 ./25369ebaaaa 5456 ./25369faaaaa 4392 ./25369fbaaaa 5520 ./25369gaaaaa 4412 ./25369gbaaaa 4388 ./25369haaaaa 4392 ./25369hbaaaa 4412 ./25369iaaaaa 4408 ./25369ibaaaa 4420 ./25369jaaaaa 3316 ./25369jbaaaa 4404 ./25369kaaaaa 3304 ./25369kbaaaa 4400 ./25369laaaaa 3312 ./25369lbaaaa 4372 ./25369maaaaa 3304 ./25369mbaaaa 4420 ./25369naaaaa 3300 ./25369nbaaaa 4376 ./25369oaaaaa 4396 ./25369obaaaa 4408 ./25369paaaaa 4404 ./25369pbaaaa 4432 ./25369qaaaaa 656 ./25369qbaaaa 4388 ./25369raaaaa 4456 ./25369saaaaa 4396 ./25369taaaaa 4396 ./25369uaaaaa 4408 ./25369vaaaaa 4412 ./25369waaaaa 4428 ./25369xaaaaa 4460 ./25369xbaaaa 4436 ./25369yaaaaa 4396 ./25369ybaaaa 4396 ./25369zaaaaa 4392 ./25369zbaaaa

adofsauron commented 1 year ago

0 DB::MergingAggregatedBucketTransform::transform (this=0x7ff2bb134718, chunk=...) at ../src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:318

1 0x0000000026b6e4a2 in DB::ISimpleTransform::transform (this=0x7ff2bb134718, input_chunk=..., output_chunk=...) at ../src/Processors/ISimpleTransform.h:32

2 0x000000002d037c01 in DB::ISimpleTransform::work (this=0x7ff2bb134718) at ../src/Processors/ISimpleTransform.cpp:89

3 0x000000002d07ca03 in DB::executeJob (node=0x7ff250b11800, read_progress_callback=0x7ff24d0388a0) at ../src/Processors/Executors/ExecutionThreadContext.cpp:47

4 0x000000002d07c719 in DB::ExecutionThreadContext::executeTask (this=0x7ff2bb0dece0) at ../src/Processors/Executors/ExecutionThreadContext.cpp:92

5 0x000000002d058861 in DB::PipelineExecutor::executeStepImpl (this=0x7ff24cf4fc18, thread_num=10, yield_flag=0x0) at ../src/Processors/Executors/PipelineExecutor.cpp:229

6 0x000000002d058b97 in DB::PipelineExecutor::executeSingleThread (this=0x7ff24cf4fc18, thread_num=10) at ../src/Processors/Executors/PipelineExecutor.cpp:195

7 0x000000002d05a416 in DB::PipelineExecutor::spawnThreads()::$_0::operator()() const (this=0x7ff260757088) at ../src/Processors/Executors/PipelineExecutor.cpp:320

8 0x000000002d05a375 in std::1::invoke[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

9 0x000000002d05a321 in std::1::apply_tuple_impl[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&, std::1::__tuple_indices<>) (f=..., __t=...) at ../contrib/llvm-project/libcxx/include/tuple:1789

10 0x000000002d05a232 in std::1::apply[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::__1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&) (

__f=..., __t=...) at ../contrib/llvm-project/libcxx/include/tuple:1798

11 0x000000002d05a11a in ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}::operator()() (this=0x7ff2bb0c3640) at ../src/Common/ThreadPool.h:210

12 0x000000002d05a055 in std::1::invoke[abi:v15000]<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

13 0x000000002d05a01d in std::1::invoke_void_return_wrapper<void, true>::call<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&>(ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&) (args=...) at ../contrib/llvm-project/libcxx/include/__functional/invoke.h:479

14 0x000000002d059ff5 in std::1::function::default_alloc_func<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}, void ()>::operator()[abi:v15000]() (this=0x7ff2bb0c3640) at ../contrib/llvm-project/libcxx/include/functional/function.h:235

15 0x000000002d059fc0 in std::1::function::policy_invoker<void ()>::__call_impl<std::1::function::default_alloc_func<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}, void ()> >(std::1::function::__policy_storage const*) (__buf=0x7ff260757348)

at ../contrib/llvm-project/libcxx/include/__functional/function.h:716

16 0x000000001a3c40a6 in std::1::function::__policy_func<void ()>::operator()[abi:v15000]() const (this=0x7ff260757348)

at ../contrib/llvm-project/libcxx/include/__functional/function.h:848

17 0x000000001a3bf9d5 in std::1::function<void ()>::operator()() const (this=0x7ff260757348) at ../contrib/llvm-project/libcxx/include/functional/function.h:1187

18 0x000000001a4ccb6e in ThreadPoolImpl::worker (this=0x7ff38534b280, thread_it=...) at ../src/Common/ThreadPool.cpp:315

19 0x000000001a4d43e4 in ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}::operator()() const (this=0x7ff2b7f96ea8) at ../src/Common/ThreadPool.cpp:145

20 0x000000001a4d4375 in std::1::invoke[abi:v15000]<ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

21 0x000000001a4d42a5 in std::1::thread_execute[abi:v15000]<std::1::unique_ptr<std::1::thread_struct, std::1::default_delete >, ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}>(std::1::tuple<std::1::unique_ptr<std::1::thread_struct, std::1::default_delete >, ThreadPoolImpl<std::1::thread>::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}>&, std::1::tuple_indices<>) (__t=...) at ../contrib/llvm-project/libcxx/include/thread:284

22 0x000000001a4d3c02 in std::1::thread_proxy[abi:v15000]<std::1::tuple<std::1::unique_ptr<std::1::thread_struct, std::1::default_delete >, ThreadPoolImpl<std::1::thread>::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}> >(void*) (__vp=0x7ff2b7f96ea0)

at ../contrib/llvm-project/libcxx/include/thread:295

23 0x00007ff3862b1802 in start_thread () from /lib64/libc.so.6

24 0x00007ff386251450 in clone3 () from /lib64/libc.so.6

adofsauron commented 1 year ago

static void executeJob(ExecutingGraph::Node node, ReadProgressCallback read_progress_callback) { try { node->processor->work();

    /// Update read progress only for source nodes.
    bool is_source = node->back_edges.empty();

    if (is_source && read_progress_callback)
    {
        if (auto read_progress = node->processor->getReadProgress())
        {
            if (read_progress->counters.total_rows_approx)
                read_progress_callback->addTotalRowsApprox(read_progress->counters.total_rows_approx);

            if (!read_progress_callback->onProgress(read_progress->counters.read_rows, read_progress->counters.read_bytes, read_progress->limits))
                node->processor->cancel();
        }
    }
}
catch (Exception & exception)
{
    if (checkCanAddAdditionalInfoToException(exception))
        exception.addMessage("While executing " + node->processor->getName());
    throw;
}

}

adofsauron commented 1 year ago

Flush data in the RAM to disk also. It's easier than merging on-disk and RAM data.

adofsauron commented 1 year ago

Pre-aggregates data from ports, holding in RAM only one or more (up to merging_threads) blocks from each source. This saves RAM in case of using two-level aggregation, where in each source there will be up to 256 blocks with parts of the result.

Aggregate functions in blocks should not be finalized so that their states can be combined.

Used to solve two tasks:

External aggregation with data flush to disk. Partially aggregated data (previously divided into 256 buckets) is flushed to some number of files on the disk. We need to read them and merge them by buckets - keeping only a few buckets from each file in RAM simultaneously.
Merge aggregation results for distributed query processing. Partially aggregated data arrives from different servers, which can be split down or not, into 256 buckets, and these buckets are passed to us by the network from each server in sequence, one by one. You should also read and merge by the buckets.

The essence of the work:

There are a number of sources. They give out blocks with partially aggregated data. Each source can return one of the following block sequences:

"unsplitted" block with bucket_num = -1;
"split" (two_level) blocks with bucket_num from 0 to 255; In both cases, there may also be a block of "overflows" with bucket_num = -1 and is_overflows = true;

We start from the convention that split blocks are always passed in the order of bucket_num. That is, if a < b, then the bucket_num = a block goes before bucket_num = b. This is needed for a memory-efficient merge

so that you do not need to read the blocks up front, but go all the way up by bucket_num.

In this case, not all bucket_num from the range of 0..255 can be present. The overflow block can be presented in any order relative to other blocks (but it can be only one).

It is necessary to combine these sequences of blocks and return the result as a sequence with the same properties. That is, at the output, if there are "split" blocks in the sequence, then they should go in the order of bucket_num.

The merge can be performed using several (merging_threads) threads. For this, receiving of a set of blocks for the next bucket_num should be done sequentially, and then, when we have several received sets, they can be merged in parallel.

When you receive next blocks from different sources, data from sources can also be read in several threads (reading_threads) for optimal performance in the presence of a fast network or disks (from where these blocks are read).

adofsauron commented 1 year ago

auto & output = outputs.front();

auto info = std::make_shared<ChunksToMerge>();
info->bucket_num = bucket;
info->is_overflows = is_overflows;
info->chunks = std::make_unique<Chunks>(std::move(chunks));

Chunk chunk;
chunk.setChunkInfo(std::move(info));
output.push(std::move(chunk));

adofsauron commented 1 year ago

        auto block = tmp_stream->read();
        if (!block)
        {
            tmp_stream = nullptr;
            return {};
        }
        return convertToChunk(block);

adofsauron commented 1 year ago

class AggregatedChunkInfo : public ChunkInfo { public: bool is_overflows = false; Int32 bucket_num = -1; UInt64 chunk_num = 0; // chunk number in order of generation, used during memory bound merging to restore chunks order };

adofsauron commented 1 year ago

Working with states of aggregate functions in the pool is arranged in the following (inconvenient) way:

when aggregating, states are created in the pool using IAggregateFunction::create (inside - placement new of arbitrary structure);
they must then be destroyed using IAggregateFunction::destroy (inside - calling the destructor of arbitrary structure);
if aggregation is complete, then, in the Aggregator::convertToBlocks function, pointers to the states of aggregate functions are written to ColumnAggregateFunction; ColumnAggregateFunction "acquires ownership" of them, that is - calls destroy in its destructor.
if during the aggregation, before call to Aggregator::convertToBlocks, an exception was thrown, then the states of aggregate functions must still be destroyed, otherwise, for complex states (eg, AggregateFunctionUniq), there will be memory leaks;
in this case, to destroy states, the destructor calls Aggregator::destroyAggregateStates method, but only if the variable aggregator (see below) is not nullptr;
that is, until you transfer ownership of the aggregate function states in the ColumnAggregateFunction, set the variable aggregator, so that when an exception occurs, the states are correctly destroyed.

PS. This can be corrected by making a pool that knows about which states of aggregate functions and in which order are put in it, and knows how to destroy them. But this can hardly be done simply because it is planned to put variable-length strings into the same pool. In this case, the pool will not be able to know with what offsets objects are stored.

adofsauron commented 1 year ago

// Disable consecutive key optimization for Uint8/16, because they use a FixedHashMap
// and the lookup there is almost free, so we don't need to cache the last lookup result
std::unique_ptr<AggregationMethodOneNumber<UInt8, AggregatedDataWithUInt8Key, false>>           key8;
std::unique_ptr<AggregationMethodOneNumber<UInt16, AggregatedDataWithUInt16Key, false>>         key16;

std::unique_ptr<AggregationMethodOneNumber<UInt32, AggregatedDataWithUInt64Key>>         key32;
std::unique_ptr<AggregationMethodOneNumber<UInt64, AggregatedDataWithUInt64Key>>         key64;
std::unique_ptr<AggregationMethodStringNoCache<AggregatedDataWithShortStringKey>>               key_string;
std::unique_ptr<AggregationMethodFixedStringNoCache<AggregatedDataWithShortStringKey>>          key_fixed_string;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithUInt16Key, false, false, false>>  keys16;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithUInt32Key>>                   keys32;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithUInt64Key>>                   keys64;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys128>>                   keys128;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys256>>                   keys256;
std::unique_ptr<AggregationMethodSerialized<AggregatedDataWithStringKey>>                serialized;

std::unique_ptr<AggregationMethodOneNumber<UInt32, AggregatedDataWithUInt64KeyTwoLevel>> key32_two_level;
std::unique_ptr<AggregationMethodOneNumber<UInt64, AggregatedDataWithUInt64KeyTwoLevel>> key64_two_level;
std::unique_ptr<AggregationMethodStringNoCache<AggregatedDataWithShortStringKeyTwoLevel>>       key_string_two_level;
std::unique_ptr<AggregationMethodFixedStringNoCache<AggregatedDataWithShortStringKeyTwoLevel>>  key_fixed_string_two_level;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithUInt32KeyTwoLevel>>           keys32_two_level;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithUInt64KeyTwoLevel>>           keys64_two_level;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys128TwoLevel>>           keys128_two_level;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys256TwoLevel>>           keys256_two_level;
std::unique_ptr<AggregationMethodSerialized<AggregatedDataWithStringKeyTwoLevel>>        serialized_two_level;

std::unique_ptr<AggregationMethodOneNumber<UInt64, AggregatedDataWithUInt64KeyHash64>>   key64_hash64;
std::unique_ptr<AggregationMethodString<AggregatedDataWithStringKeyHash64>>              key_string_hash64;
std::unique_ptr<AggregationMethodFixedString<AggregatedDataWithStringKeyHash64>>         key_fixed_string_hash64;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys128Hash64>>             keys128_hash64;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys256Hash64>>             keys256_hash64;
std::unique_ptr<AggregationMethodSerialized<AggregatedDataWithStringKeyHash64>>          serialized_hash64;

/// Support for nullable keys.
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys128, true>>             nullable_keys128;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys256, true>>             nullable_keys256;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys128TwoLevel, true>>     nullable_keys128_two_level;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys256TwoLevel, true>>     nullable_keys256_two_level;

/// Support for low cardinality.
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodOneNumber<UInt8, AggregatedDataWithNullableUInt8Key, false>>> low_cardinality_key8;
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodOneNumber<UInt16, AggregatedDataWithNullableUInt16Key, false>>> low_cardinality_key16;
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodOneNumber<UInt32, AggregatedDataWithNullableUInt64Key>>> low_cardinality_key32;
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodOneNumber<UInt64, AggregatedDataWithNullableUInt64Key>>> low_cardinality_key64;
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodString<AggregatedDataWithNullableStringKey>>> low_cardinality_key_string;
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodFixedString<AggregatedDataWithNullableStringKey>>> low_cardinality_key_fixed_string;

std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodOneNumber<UInt32, AggregatedDataWithNullableUInt64KeyTwoLevel>>> low_cardinality_key32_two_level;
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodOneNumber<UInt64, AggregatedDataWithNullableUInt64KeyTwoLevel>>> low_cardinality_key64_two_level;
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodString<AggregatedDataWithNullableStringKeyTwoLevel>>> low_cardinality_key_string_two_level;
std::unique_ptr<AggregationMethodSingleLowCardinalityColumn<AggregationMethodFixedString<AggregatedDataWithNullableStringKeyTwoLevel>>> low_cardinality_key_fixed_string_two_level;

std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys128, false, true>>      low_cardinality_keys128;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys256, false, true>>      low_cardinality_keys256;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys128TwoLevel, false, true>> low_cardinality_keys128_two_level;
std::unique_ptr<AggregationMethodKeysFixed<AggregatedDataWithKeys256TwoLevel, false, true>> low_cardinality_keys256_two_level;

stoneatom / stonedb

feature: Cache disk files for intermediate results of aggregate operations to avoid OOM #949

Abstract:

Note of context:

Functional Requirements:

1. The HASH result of the aggregation operation is cached in the disk file and properly participates in the operation

Performance requirements:

1. The compression ratio of the cache file size to the original data content, and the ratio of disk space to memory

2. Disk cache file write speed and read speed

3. The impact on the performance of aggregation operations

Development cycle:

Abstract:

Cache RAM analysis of current aggregated intermediate results:

Summary design for adding disk cache:

Design Idea:

Design Strategy:

Architecture Design:

Static structure:

Refer to the AIO used by mysql for asynchronous IO

The data flow that interacts with the file system

buffer Settings for mysql5.7.36

Buffer Pool

Multiple Buffer Pool Instances

Data write file

Data segmentation for parallel aggregation

ExpressionTransform

execute

0 DB::Aggregator::writeToTemporaryFile (this=0x7f34322d9210, data_variants=..., max_temp_file_size=3512367045) at ../src/Interpreters/Aggregator.cpp:1603

1 0x000000002aa049cc in DB::Aggregator::executeOnBlock (this=0x7f34322d9210, columns=..., row_begin=0, row_end=900335, result=..., key_columns=..., aggregate_columns=...,

2 0x000000002d48c813 in DB::AggregatingTransform::consume (this=0x7f343236c818, chunk=...) at ../src/Processors/Transforms/AggregatingTransform.cpp:533

3 0x000000002d48a501 in DB::AggregatingTransform::work (this=0x7f343236c818) at ../src/Processors/Transforms/AggregatingTransform.cpp:492

4 0x000000002d077a03 in DB::executeJob (node=0x7f34323b0700, read_progress_callback=0x7f3432244060) at ../src/Processors/Executors/ExecutionThreadContext.cpp:47

5 0x000000002d077719 in DB::ExecutionThreadContext::executeTask (this=0x7f3494903b60) at ../src/Processors/Executors/ExecutionThreadContext.cpp:92

6 0x000000002d053861 in DB::PipelineExecutor::executeStepImpl (this=0x7f3432291018, thread_num=4, yield_flag=0x0) at ../src/Processors/Executors/PipelineExecutor.cpp:229

7 0x000000002d053b97 in DB::PipelineExecutor::executeSingleThread (this=0x7f3432291018, thread_num=4) at ../src/Processors/Executors/PipelineExecutor.cpp:195

8 0x000000002d055416 in DB::PipelineExecutor::spawnThreads()::$_0::operator()() const (this=0x7f343e15c088) at ../src/Processors/Executors/PipelineExecutor.cpp:320

9 0x000000002d055375 in std::1::invoke[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

10 0x000000002d055321 in std::1::apply_tuple_impl[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&, std::1::__tuple_indices<>) (f=..., __t=...) at ../contrib/llvm-project/libcxx/include/tuple:1789

11 0x000000002d055232 in std::1::apply[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::__1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&) (

12 0x000000002d05511a in ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}::operator()() (this=0x7f34948e6730) at ../src/Common/ThreadPool.h:210

13 0x000000002d055055 in std::1::invoke[abi:v15000]<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

17 0x000000001a3bf0a6 in std::1::function::__policy_func<void ()>::operator()[abi:v15000]() const (this=0x7f343e15c348)

18 0x000000001a3ba9d5 in std::1::function<void ()>::operator()() const (this=0x7f343e15c348) at ../contrib/llvm-project/libcxx/include/functional/function.h:1187

19 0x000000001a4c7b6e in ThreadPoolImpl::worker (this=0x7f356534b280, thread_it=...) at ../src/Common/ThreadPool.cpp:315

20 0x000000001a4cf3e4 in ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}::operator()() const

21 0x000000001a4cf375 in std::1::invoke[abi:v15000]<ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

23 0x000000001a4cec02 in std::1::thread_proxy[abi:v15000]<std::1::tuple<std::1::unique_ptr<std::1::thread_struct, std::1::default_delete >, ThreadPoolImpl<std::1::thread>::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}> >(void*) (__vp=0x7f34a25c4000)

24 0x00007f356620e802 in start_thread () from /lib64/libc.so.6

25 0x00007f35661ae450 in clone3 () from /lib64/libc.so.6

define M(NAME) \

undef M

0 DB::MergingAggregatedBucketTransform::transform (this=0x7ff2bb134718, chunk=...) at ../src/Processors/Transforms/MergingAggregatedMemoryEfficientTransform.cpp:318

1 0x0000000026b6e4a2 in DB::ISimpleTransform::transform (this=0x7ff2bb134718, input_chunk=..., output_chunk=...) at ../src/Processors/ISimpleTransform.h:32

2 0x000000002d037c01 in DB::ISimpleTransform::work (this=0x7ff2bb134718) at ../src/Processors/ISimpleTransform.cpp:89

3 0x000000002d07ca03 in DB::executeJob (node=0x7ff250b11800, read_progress_callback=0x7ff24d0388a0) at ../src/Processors/Executors/ExecutionThreadContext.cpp:47

4 0x000000002d07c719 in DB::ExecutionThreadContext::executeTask (this=0x7ff2bb0dece0) at ../src/Processors/Executors/ExecutionThreadContext.cpp:92

5 0x000000002d058861 in DB::PipelineExecutor::executeStepImpl (this=0x7ff24cf4fc18, thread_num=10, yield_flag=0x0) at ../src/Processors/Executors/PipelineExecutor.cpp:229

6 0x000000002d058b97 in DB::PipelineExecutor::executeSingleThread (this=0x7ff24cf4fc18, thread_num=10) at ../src/Processors/Executors/PipelineExecutor.cpp:195

7 0x000000002d05a416 in DB::PipelineExecutor::spawnThreads()::$_0::operator()() const (this=0x7ff260757088) at ../src/Processors/Executors/PipelineExecutor.cpp:320

8 0x000000002d05a375 in std::1::invoke[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

9 0x000000002d05a321 in std::1::apply_tuple_impl[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&, std::1::__tuple_indices<>) (f=..., __t=...) at ../contrib/llvm-project/libcxx/include/tuple:1789

10 0x000000002d05a232 in std::1::apply[abi:v15000]<DB::PipelineExecutor::spawnThreads()::$_0&, std::__1::tuple<>&>(DB::PipelineExecutor::spawnThreads()::$_0&, std::1::tuple<>&) (

11 0x000000002d05a11a in ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}::operator()() (this=0x7ff2bb0c3640) at ../src/Common/ThreadPool.h:210

12 0x000000002d05a055 in std::1::invoke[abi:v15000]<ThreadFromGlobalPoolImpl::ThreadFromGlobalPoolImpl<DB::PipelineExecutor::spawnThreads()::$_0>(DB::PipelineExecutor::spawnThreads()::$_0&&)::{lambda()#1}&> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

16 0x000000001a3c40a6 in std::1::function::__policy_func<void ()>::operator()[abi:v15000]() const (this=0x7ff260757348)

17 0x000000001a3bf9d5 in std::1::function<void ()>::operator()() const (this=0x7ff260757348) at ../contrib/llvm-project/libcxx/include/functional/function.h:1187

18 0x000000001a4ccb6e in ThreadPoolImpl::worker (this=0x7ff38534b280, thread_it=...) at ../src/Common/ThreadPool.cpp:315

19 0x000000001a4d43e4 in ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}::operator()() const (this=0x7ff2b7f96ea8) at ../src/Common/ThreadPool.cpp:145

20 0x000000001a4d4375 in std::1::invoke[abi:v15000]<ThreadPoolImpl::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}> (f=...) at ../contrib/llvm-project/libcxx/include/functional/invoke.h:394

22 0x000000001a4d3c02 in std::1::thread_proxy[abi:v15000]<std::1::tuple<std::1::unique_ptr<std::1::thread_struct, std::1::default_delete >, ThreadPoolImpl<std::1::thread>::scheduleImpl(std::1::function<void ()>, long, std::1::optional, bool)::{lambda()#2}> >(void*) (__vp=0x7ff2b7f96ea0)

23 0x00007ff3862b1802 in start_thread () from /lib64/libc.so.6

24 0x00007ff386251450 in clone3 () from /lib64/libc.so.6