prefill metadata cache option

[jrse]

Fetching mail metadata may be slow. This is due to the caching strategy.

The current strategy is cache size optimized, which means that mail metadata will be added to the cache after first access. First access to the metadata means in this case reading the omap values of the mail object from the rados storage.

In some cases, the index mail cache size is not as important as the response time and the behavior should be configurable:

PREFILL_METADATA_CACHE=false|true

The default value is false. If set to true, the following E-Mail metadata should be added to the index cache.

Mail guid, Save Date, Receive Date, Pop3.uild

Log:

using doveadm fetch to get mail attributes takes a lot of time: 2019/01/09 18:43:10 mdclient[28123]: Debug: Execute command "doveadm -c /conf/dovecot.conf fetch -u u1 guid date.received date.sent flags pop3.uidl seq size.virtual uid user mailbox-guid mailbox ALL > /tmp/u1/maillist_source 2> /dev/null" 2019/01/09 18:45:04 mdclient[28123]: Debug: Get list of mailboxes for user u1 using config

[jrse]

mail guid is already saved in the extension header.

[jrse]

received_date and save_date are already in the cache: (index_mail_cache_dates in index-mail.c).

[jrse]

rbox: doveadm dump RECORD: seq=10, uid=10, flags=0x00

• ext 3 cache : 1244 (dc040000)
• cache offset=1244 size=88, prev_offset = 0
  • hdr.From:
  • hdr.Message-ID:
  • hdr.Subject:
  • flags: (30000000)
  • mime.parts: (08000000d900000000000000e0000000000000002000000000000000220000000000000002000000) (pos=0 hdr.p=217 hdr.v=224 body.p=32 body.v=34 flags=8)

sdbox: doveadm dump:

RECORD: seq=10, uid=10, flags=0x00

• ext 3 cache : 1244 (dc040000)
• cache offset=1244 size=88, prev_offset = 0
  • hdr.From:
  • hdr.Message-ID:
  • hdr.Subject:
  • flags: (30000000)
  • mime.parts: (08000000d900000000000000e0000000000000002000000000000000220000000000000002000000) (pos=0 hdr.p=217 hdr.v=224 body.p=32 body.v=34 flags=8)

[jrse]

to force pre caching of save_date, receive_date, virtual_size, physical_size. the cache decision field can be updated.

e.g. struct mail_save_context _ctx; struct mail_cache cache = _ctx->transaction->box->cache; cache->fields[MAIL_CACHE_SAVE_DATE].field.decision = MAIL_CACHE_DECISION_YES;

if this is done before rbox_save_continue, the given fields will be saved in the cache.

for chache fields other then date and size, the following function: index_mail_cache_add((struct index_mail *)_ctx->dest_mail, MAIL_CACHE_POP3_UIDL, &_ctx->data.pop3_uidl, sizeof(_ctx->data.pop3_uidl)); can be used.

[jrse]

Dovecot configuration allows the definition of cache fields:

-> mail_always_cache_fields => list of cache fields with cache DECISION always MAIL_CACHE_DECISION_YES | MAIL_CACHE_DECISION_FORCED -> mail_never_cache_fields => list of cache fields with cache disabled MAIL_CACHE_DECISION_NO | | MAIL_CACHE_DECISION_FORCED -> mail_cache_fields => list of cache fields with cache DECISION MAIL_CACHE_DECISION_TEMP .

The caching algorithm described in the dovecot wiki will only apply, if the mailbox was created with mail_cache_fields = ....

The field cache information is written to the cache file. In case of MAIL_CACHE_DECISION_FORCED or MAIL_CACHE_DECISION_NO field is permanently marked as YES or NO.

[jrse]

dovecot allows the manual configuration of index cache fields via struct mail_storage_settings, mail_cache_fields mail_always_cache_fields mail_never_cache_fields those can be set in e.g. 20-mail.conf