search

AutoMQ / automq

AutoMQ is a cloud-first alternative to Kafka by decoupling durability to S3 and EBS. 10x Cost-Effective. No Cross-AZ Traffic Cost. Autoscale in seconds. Single-digit ms latency.
https://www.automq.com/docs
Other
3.81k stars 219 forks source link

feat(s3stream): change append callback thread number #1950

Closed lifepuzzlefun closed 2 months ago

lifepuzzlefun commented 2 months ago

when single node in high produce qps ( 8w qps for 8c16g single node ) the thread may not enough cause high produce latency. image

superhx commented 2 months ago

when single node in high produce qps ( 8w qps for 8c16g single node ) the thread may not enough cause high produce latency. image

Could we automatically make them scaling by CPU_CORES likes

    public StreamReaders(long size, ObjectManager objectManager, ObjectStorage objectStorage,
        ObjectReaderFactory objectReaderFactory) {
        this(size, objectManager, objectStorage, objectReaderFactory, Systems.CPU_CORES);
    }

Currently, the thread pool threads number is based on 2 Core.

lifepuzzlefun commented 2 months ago

when single node in high produce qps ( 8w qps for 8c16g single node ) the thread may not enough cause high produce latency. image

Could we automatically make them scaling by CPU_CORES likes

    public StreamReaders(long size, ObjectManager objectManager, ObjectStorage objectStorage,
        ObjectReaderFactory objectReaderFactory) {
        this(size, objectManager, objectStorage, objectReaderFactory, Systems.CPU_CORES);
    }

Currently, the thread pool threads number is based on 2 Core.

agree. and do you think we should introduce the thread number or just automatically set to 2 * Systems.CPU_CROES.

for 2cpu deploy the thread number will be drop from 8 to 4.

or keep the config and change the default to 2 * CPU

superhx commented 2 months ago

when single node in high produce qps ( 8w qps for 8c16g single node ) the thread may not enough cause high produce latency. image

Could we automatically make them scaling by CPU_CORES likes

    public StreamReaders(long size, ObjectManager objectManager, ObjectStorage objectStorage,
        ObjectReaderFactory objectReaderFactory) {
        this(size, objectManager, objectStorage, objectReaderFactory, Systems.CPU_CORES);
    }

Currently, the thread pool threads number is based on 2 Core.

agree. and do you think we should introduce the thread number or just automatically set to 2 * Systems.CPU_CROES.

for 2cpu deploy the thread number will be drop from 8 to 4.

or keep the config and change the default to 2 * CPU

I think auto-set is enough