hibiken / asynqmon

Web UI for Asynq task queue
MIT License
605 stars 132 forks source link

Only 1 active worker from 800 #321

Closed ludovit-ubrezi closed 4 months ago

ludovit-ubrezi commented 4 months ago

Hello, running worker with this configuration

func main() {
    concurrency := runtime.NumCPU() * 200

    srv := asynq.NewServer(
        asynq.RedisClientOpt{Addr: redisHost + ":" + redisPort, Password: redisPassword, DB: 1},
        asynq.Config{
            Concurrency:    concurrency,
            Queues:         map[string]int{tasks.QueueName: 9},
            StrictPriority: true,
        },
    )

    // close publisher connection functionality
    defer tasks.Publisher.Channel.Close()

    // Use asynq.HandlerFunc adapter for a handler function
    if err := srv.Run(asynq.HandlerFunc(tasks.HandleMonitorDeliveryTask)); err != nil {
        log.Fatal(err)
    }
}

Problem is that even Concurency is set to 800 workers there is always only 1 worker processing queue. This is why the Pending queue is not processing correctly and whole system has delays. (screenshots below)

Code of function tasks.HandleMonitorDeliveryTask is working correctly and fast because when i load all the data, it processes Task with group of 5000 items and it load into RabbitMQ -> 1 million items in e.g. 1 minute.

But when Task has 1 item it slows down due to only 1 worker processing this queue even when there is 800 workers setup for server it only run 1.

Is there something what i missing in configuration? How to setup it?

Screenshot 2024-02-27 at 12 30 38 Screenshot 2024-02-27 at 12 30 06

ludovit-ubrezi commented 4 months ago

bad forum