dreness
commented
6 years ago Hello,
The relevant config parameters are:
MultiProcess -> ProcessCount - defines the number of worker processes used
MaxDBConnectionsPerPool - the number of DB connections allowed for each process
The maximum number of DB connections used by a CalendarServer host is ProcessCount * MaxDBConnectionsPerPool.
Maybe what you are really saying is "I know how to configure things. CalendarServer uses an unexpectedly high number of DB connections to perform my workload." This is a reasonable characterization in my experience, at least in some cases. Large-scale CalendarServer deployments likely need a strong DB server configured to allow a lot of concurrency, but it still might vary widely based on workload. For example, all CardDAV workloads are far simpler and cheaper to process than common CalDAV workloads involving scheduling with multiple users.
We do have an experimental SharedConnectionPool option that is described in the comment for MaxDBConnectionsPerPool:
<!-- The maximum number of outstanding database connections per database
connection pool. When SharedConnectionPool (see above) is set to True,
this is the total number of outgoing database connections allowed to the
entire server; when SharedConnectionPool is False - this is the default -
this is the number of database connections used per worker process. -->Using this might let you get by with a lower number of DB connections, but whether this is advisable in your case depends a lot on the nature of your workload and the size / scale of your server(s). Be advised, though: SharedConnectionPool is considered experimental and hasn't been used at scale, as far as I know. In general, the Python concurrency model means we have to go somewhat wider with process count than might otherwise be necessary. There is a whole lot of space between a per-process DB connection pool (where connections are shared between 'threads' in that process) and a server-wide DB connection pool, but those are the only two options you have. In the first case, you have potentially a lot of idle or mostly idle DB connections, and in the second case you risk bottlenecking in the process hosting the shared pool. For serious deployments, the DB needs to be perfectly content to handle the maximum concurrency as configured by CalendarServer, because resource usage is often incredibly bursty even with common workloads, assuming you are optimizing for minimum request times. There is also the option of tuning the WorkQueue priority thresholds to penalize expensive work by deferring it during burst periods, but this really only works to smooth out the peaks and valleys. If you really want to get fine-grained, you can also customize the relative cost ('weight') and priority of each kind of WorkItem (also read all the comments in this file!). Each different kind of work has a corresponding WORK table in the DB. Sample config for making expensive CalDAV scheduling work less concurrent by raising the weight (which defaults to 5) might look like:
<key>WorkQueue</key>
<dict>
<key>workParameters</key>
<dict>
<key>SCHEDULE_ORGANIZER_WORK</key>
<dict>
<key>priority</key>
<integer>1</integer>
<key>weight</key>
<integer>7</integer>
</dict>
<key>SCHEDULE_ORGANIZER_SEND_WORK</key>
<dict>
<key>priority</key>
<integer>1</integer>
<key>weight</key>
<integer>7</integer>
</dict>
<key>SCHEDULE_AUTO_REPLY_WORK</key>
<dict>
<key>priority</key>
<integer>1</integer>
<key>weight</key>
<integer>7</integer>
</dict>
<key>SCHEDULE_REPLY_WORK</key>
<dict>
<key>priority</key>
<integer>1</integer>
<key>weight</key>
<integer>7</integer>
</dict>
<key>SCHEDULE_REFRESH_WORK</key>
<dict>
<key>priority</key>
<integer>1</integer>
<key>weight</key>
<integer>7</integer>
</dict>
</dict>
</dict>A good overview of your workload can be produced by using contrib/tools/protocolanalysis.py to make reports from your access logs. If you'd like to share a report and some details about your server hardware, perhaps we can make recommendations for tuning.
Does this help?
sandeepksingh13
Application is making too many database connections