indimail / indimail-mta

qmail fork with IPV6, TLS, DANE, DKIM, SRS2, SPF, daemontools, qmailanalog, mess822, & ucspi-tcp
https://github.com/indimail/indimail-mta/wiki/0-IndiMail-Wiki
GNU General Public License v3.0
12 stars 3 forks source link

dynamic queue for indimail-mta #29

Closed mbhangui closed 2 years ago

mbhangui commented 2 years ago
  1. add qscheduler to .gitignore, TARGETS - done
  2. add qtop program - done
  3. Add feature to dynamically increase queue by passing message to qscheduler daemon using /qscheduler message queue - done
  4. Write setqload to increase queue at runtime - done
  5. Write a cron script that monitors the queue load to dynamically increase or decrease queue count - done with qmonitor program
  6. Measure Performance - Done. See qmail-perf.
  7. QA Checks - ongoing

Description

indimail can be configured to use multiple queues for mail deliveries, using the qscheduler(8) daemon. These can be configured by setting few environment variables and by specifying command-line switches to the qscheduler(8) daemon. The command line options can be used either to configure a set of fixed static queues or have the queue increased dynamically with load. The load is the ratio of concurrency used and total concurrency configured for a queue. These are actually two ratios. One ratio for local deliveries and another ratio for remote deliveries.

A queue in indimail is denoted by a number and a base path. The compile time default is /var/indimail/queue, but can be changed by the control file /etc/indimail/control/queue_base or the environment variable QUEUE_BASE. The environment variable takes precedence. The queue number and the base path constitutes the actual path for the queue. e.g.

   queue numbered 1, means /var/indimail/queue/queue1
   queue numbered 2, means /var/indimail/queue/queue2
   queue numbered n, means /var/indimail/queue/queuen

The compile time default for the first queue of the multi-queue environment is queue1. This can be changed by setting QUEUE_START environment variable. The compile time default for the number of queues that indimail uses is 5, which can be changed by setting QUEUE_COUNT environment variable.

ENVIRONMENT VARIABLES

The following environment variables are used to configure the queues.

QUEUE_START - defines the first queue that should be used. QUEUE_START=1 implies that /var/indimail/queue/queue1 be the first queue

QUEUE_COUNT - defines the total number of queues that should be used for mail deliveries. This is the fixed upper limit when qscheduler(8) is passed the -s argument

As explained earlier, qscheduler(8) can be used for configuring mail delivery with fixed number of queues or load dependent dynamic queues. The following additional environment variables are needed when dynamic queues are used.

QUEUE_MAX - defines the maximum queues that qscheduler can use for deliveries when dynamic queues is configured (-d argument to qscheduler(8).

QUEUE_LOAD - defines the max queue load after which qscheduler(8) should increment the queue count by starting an additional qmail-send(8), qmail-todo(8) instance.

Inter-Process Communication

qscheduler(8) uses POSIX inter-process communication namely shared memory segments and message queues. The various forms of IPC are described as below

shared memory /qscheduler

This is created by qscheduler(8) using shm_open(3), owned by root:root with perm 0644. This segment is used to store two integers

  1. queue count - number of queues in use by qscheduler(8). This value will be the value of QUEUE_COUNT environment variable on qscheduler(8) startup. This is a dynamic number that is incremented when the total concurrency load goes beyond a threshold (defined by QUEUE_LOAD environment variable). qscheduler(8) will increase the queue count upto a maximum defined by the QUEUE_MAX environment variable.
  2. queue configured - number of existing queues in /var/indimail directory. This number is incremented by qscheduler when it creates a new queue. This number cannot go beyond the value of QUEUE_MAX environment variable.

shared memory /queueN

This is created by qscheduler(8) but used by qmail-send(8) to write its current concurrency value and the maximum concurrency configured (for both local and remote deliveries). The concurrency is incremented whenever a job is opened and decremented whenever a job is closed. This is owned by qmails:qmail with perm 0644. The queue load is the ratio of current concurrency and the maximum concurrency.

message queue /qscheduler

This is created by qscheduler(8) and used by qmail-send(8) to communicate back, when its queue reaches, the maximum threshold percentage for concurrency to total concurrency ratio. When the average concurrency load reaches QUEUE_LOAD, qscheduler(8) starts a new qmail-send(8), qmail-send(8) instance to install a new queue. This is owned by root:root with perm 0600.

The value stored in the first segment is the queue countqcount. qmail-multi(8) cycles through qcount queues to find the queue with the least concurrency. It then sets the environment variable QUEUEDIR to this queue before calling qmail-queue(8).

message queue /queueN

For each queue, qscheduler(8) creates a message queue using mq_open(3). Here N refers to the queue number). This is owned by qmailq:qmail with perm 0640.

This message queue is used by qmail-queue(8) to communicate to qmail-todo(8) the split dir and the inode number of the message in the outgoing queue. This is more efficient than the trigger method that qmail-queue uses, where the trigger wakes up qmail-send(8) to scan the todo directory. Scanning the todousing opendir(3), readdir(2), also introduces what is known as the Silly Qmail Syndrome, where a single instance of qmail-send(8) cannot keep up with the injection rate from multiple qmail-queue clients. The solution to this problem is to have multiple instances of qmail-send. These instances of qmail-send can be a fixed number (qscheduler -s) or increased dynamically (qscheduler -d).