search

djcb / mu

maildir indexer/searcher + emacs mail client + guile bindings
http://www.djcbsoftware.nl/code/mu
GNU General Public License v3.0
1.61k stars 386 forks source link

[mu4e bug] mu 1.6.1 emacs interface very slow #2089

Closed telmich closed 1 year ago

telmich commented 3 years ago

Describe the bug

As reported before, mu4e can be extremely slow. Right now with mu 1.6.1 there is however a huge difference between searching inside emacs and searching using mu find:

To Reproduce

Index ca. 1.5M messages, execute a simple search like:

find flag:flagged OR maildir:/ungleich/inbox OR maildir:/gmail/inbox OR maildir:/otherfish/inbox

Time with mu find: ca. 1s Time with mu4e: minutes (ca 2-3m)

If I repeat the same query, it always takes extremely long inside emacs and is always fast outside.

While in my previous report it was related to updating and did not always happen, the situation changed now:

(independent of whether update is running or not)

Environment

nb3:~# apk list -I mu mu4e emacs-x11
mu-1.6.1-r0 x86_64 {mu} (GPL-3.0) [installed]
mu4e-1.6.1-r0 x86_64 {mu} (GPL-3.0) [installed]
emacs-x11-27.2-r1 x86_64 {emacs} (GPL-3.0-or-later) [installed]

[17:18] nb3:~% du -sh Maildir/
35G Maildir/
[17:17] nb3:~% find Maildir/ -type f | wc -l
1510809

Checklist

djcb commented 3 years ago

What options did you use with the mu find vs mu4e (w.r.t. related messages / threads)? Does this happen with a singular query, i.e. just flag:flagged?

telmich commented 3 years ago

The problem also happens with only flag:flagged (still running in mu4e as of writing this). In terms of configuration, I am not sure which ones are affecting the related messages/threads, so I'll add my complete mu4e related settings without the contexts:

(require 'mu4e)
(setq mu4e-view-use-gnus nil)
(add-to-list 'mu4e-view-actions
             '("ViewInBrowser" . mu4e-action-view-in-browser) t)

;; needed for some emails in mu4e
(setq max-specpdl-size 5000)  ; default is 1000, reduce the backtrace level

(add-to-list 'mu4e-view-actions
             '("ViewInBrowser" . mu4e-action-view-in-browser) t)

(add-hook 'message-mode-hook 'turn-on-orgtbl)

(setq mu4e-compose-complete-only-after (format-time-string
                                        "%Y-%m-%d"
                                        (time-subtract (current-time) (days-to-time 600))))

(setq mu4e-headers-sort-direction 'descending)

(setq mu4e-headers-include-related nil)

(setq mu4e-attachment-dir  "~/Downloads")
(setq mu4e-drafts-folder "/ungleich/drafts")
(setq mu4e-sent-folder   "/ungleich/sent")
(setq mu4e-trash-folder  "/ungleich/trash")
(setq mu4e-refile-folder
      (lambda (msg) (concat "/ungleich/" (format-time-string "%Y"))))

;; mbsync workaround
;; https://stackoverflow.com/questions/39513469/mbsync-error-uid-is-beyond-highest-assigned-uid
(setq mu4e-change-filenames-when-moving t)
(setq mu4e-bookmarks '())

; allow all results
(setq mu4e-headers-full-search t)
(add-to-list 'mu4e-bookmarks
             '("maildir:/ungleich/archive" "archive" ?a))

(add-to-list 'mu4e-bookmarks
             '("maildir:/ungleich/drafts" "drafts" ?d))

(add-to-list 'mu4e-bookmarks
             '("maildir:/sent OR maildir:/outgoing OR maildir:/ungleich/sent" "sent" ?s))

(add-to-list 'mu4e-bookmarks
             '("(flag:flagged OR maildir:/ungleich/inbox OR maildir:/gmail/inbox) AND redmine" "redmine" ?r))

(add-to-list 'mu4e-bookmarks
             '("flag:flagged OR maildir:/ungleich/inbox OR maildir:/gmail/inbox OR maildir:/otherfish/inbox"
               "inbox" ?i))

(add-to-list 'mu4e-bookmarks
             '("maildir:/otherfish/inbox"       "otherfish" ?f))

(add-to-list 'mu4e-bookmarks
             '("maildir:/gmail/inbox"       "gmail" ?g))

(add-to-list 'mu4e-bookmarks
             '("maildir:/ungleich/inbox"       "ungleich" ?u))

(setq mu4e-hide-index-messages t           ;; Don't annoy the user
      mu4e-get-mail-command "hole_mails"   ;; custom script for security
      mu4e-attachment-dir "~")             ;; [Don't] fill up the home dir

(add-hook 'mu4e-view-mode-hook 'visual-line-mode)

;; Always sign
(add-hook 'message-setup-hook 'mml-secure-sign-pgpmime)

(setq message-kill-buffer-on-exit t)

(require 'org-mu4e)
(define-key mu4e-headers-mode-map (kbd "C-c c") 'mu4e-org-store-and-capture)
(define-key mu4e-view-mode-map    (kbd "C-c c") 'mu4e-org-store-and-capture)
(setq mu4e-headers-skip-duplicates nil)

(setq org-mu4e-link-query-in-headers-mode nil)
telmich commented 3 years ago

I just checked on another machine, using 1.4.15 queries with the same configuration are fast:

[21:14] line:~% find Maildir/ -type f  | wc -l
1509616
[21:14] line:~% dpkg -l | grep mu4e           
ii  mu4e                                     1.4.15-1                                     all          e-mail client for Emacs based on mu (maildir-utils)
haji-ali commented 3 years ago

I had a similar issue, not sure if yours is the same. In my case, the first search in mu4e after an mbsync update was always slow because it seems that mu4e indexes messages before searching; which is slow on a large database. Since I updated frequently, this meant that most searches were slow.

djcb commented 3 years ago

The command-line query needs to have the equivalent flags to allow for comparison, so

$ mu find  --threads flag:flagged

and perhaps, to isolate the server part from mu4e:

$ mu server --eval '(find :query "flag:flagged" :threads t)'

How many results do you have for that query? If there are many, (setq mu4e-headers-full-search t) will make things slow.

telmich commented 3 years ago 
mu find  --threads flag:flagged

runs in < 1s

[21:51] nb3:~% mu server --eval '(find :query "flag:flagged" :threads t)' 
[b](:erase t)
[b](:found 0)

runs in ca. 1s

I actually have zero results for that particular query. Using something slightly bigger takes 13s:

[22:17] nb3:~% date; mu server --eval '(find :query "maildir:/ungleich/2021" :threads t)'  >/dev/null; date
Sun Aug 15 22:17:44 CEST 2021
Sun Aug 15 22:17:57 CEST 2021

with mu find still being much faster:

[22:19] nb3:~% date; mu find --threads  "maildir:/ungleich/2021" > /dev/null; date 
Sun Aug 15 22:19:33 CEST 2021
Sun Aug 15 22:19:34 CEST 2021
djcb commented 3 years ago

You mentioned the problem happened in mu4e even with flag:flagged -- how long does it take there? (that's an interesting case since apparently it only takes one sec with mu server eval).

Some other things that would be interesting to check: what happens when you (setq mu4e-headers-full-search nil) (i.e. the default value)? And for various values of mu4e-headers-results-limit (like, 500, 1000, 5000)?

telmich commented 3 years ago

I just (setq mu4e-headers-full-search nil) and searching for the inbox (bookmark bi) takes about 2 minutes:

[12:18] nb3:~% date
Mon Aug 16 12:18:08 CEST 2021
[12:18] nb3:~% date
Mon Aug 16 12:20:11 CEST 2021

Searching for flag:flagged takes 4m

[12:20] nb3:~% date
Mon Aug 16 12:20:22 CEST 2021
[12:23] nb3:~% date
Mon Aug 16 12:24:38 CEST 2021

I noticed that during the mu4e search the SSD is 100% busy.

I cannot observe the same behaviour with mu find, as that one exists faster than I can see the disk stats.

Setting (setq mu4e-headers-result-limit 500) additional to (setq mu4e-headers-full-search nil) returned the result instantly.

I'll keep that setting on for the moment to see if it behaves consistent or not.

telmich commented 3 years ago

While the search result was fast, loading a message is still very slow. It took about 5m:

[12:27] nb3:~% date
Mon Aug 16 12:27:48 CEST 2021
[12:27] nb3:~% date
Mon Aug 16 12:28:56 CEST 2021
[12:29] nb3:~% date             
Mon Aug 16 12:33:39 CEST 2021

Besides the SSD usage spiking 100%, CPU usage also spikes 100% from the mu process - not in waiting state, but actually running. Over 10s (atop interval) I see the mu process reading around 500MB data for trying to display a message.

telmich commented 3 years ago

Another observation: for every query in mu4e, I see the mu process reading Gigabytes of data. The mu process itself barely exceeds 300MB RAM size.

Is it possible that there is a query / maximum memory size in the mu process and because the maildir is 35GB (xapian in the cache directory: 11GB) that for every query from mu4e the whole database is read over and over again?

There is another thought on how to improve the speed: my main queries are "give me mails in the following three maildirs", which would be a sub-second query given that the filesystem was queried directly.

I was wondering if there could be a flag telling mu/mu4e to directly query the fs, if the query only contains maildir:... expressions? I would not mind waiting for 3m once in a while, but for it is a bit excessive for every inbox query.

theophilusx commented 3 years ago

Note that you might find using the 'time' command more informative than date and more accurate.

telmich commented 3 years ago

Thanks for the emacs -q pointer in the other issue, @theophilusx. I have been using mu4e fast for about 20 minutes, until I started retrieving emails.

The index process after retrieving emails takes about 1-2 minutes and I had to set

(setq mu4e-update-interval nil)

to be able to use mu4e. By default I have

(setq mu4e-update-interval 240)

But given that the update process takes 120s+, half of the time mu is spending time in indexing. I am now continuing to test with it being set to nil, maybe 1.6.1 has just become much slower in indexing.

However it still does not explain why the CLI still replies with answers while mu4e hangs.

telmich commented 3 years ago

Coming back to this discussion / following up on the "xapian is not slow" comment from the other ticket: at the moment in a fresh, emacs -q minimal environment an update post mbsync retrieval is taking about 4 minutes inside mu4e or outside mu4e (just mu index) of time during which no queries can be answered.

Am I doing something specifically wrong or is there anything I can do to improve this situation?

I have recorded the index process on https://www.nico.schottelius.org/temp/mu-index-2021-08-23.mkv to show how it looks like for me.

djcb commented 3 years ago

This looks typical for the first-time indexing after init -- seems you're indexing all messages starting from zero; however, normally it should be much faster -- i.e., If you do another mu index after that first one.

telmich commented 3 years ago

Quick note, the following setup loads usually loads mails in less than 40s:

mu4e                                     1.4.15-1
libxapian30:amd64                        1.4.18-3  
emacs                                    1:27.1+1-3.1
telmich commented 3 years ago

I just ran another test, with mbsync disabled and only running mu index in a loop:

[10:57] nb3:~% while true; do echo -n "start: "; date; time mu index; echo -n "end: " ; date; sleep 10; done
start: Tue Aug 24 10:57:13 CEST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
- indexing messages; processed: 122084; updated/new: 122217; cleaned-up: 7
mu index  117.03s user 70.19s system 96% cpu 3:14.67 total
end: Tue Aug 24 11:00:28 CEST 2021
start: Tue Aug 24 11:00:38 CEST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
| indexing messages; processed: 122011; updated/new: 122141; cleaned-up: 0
mu index  120.86s user 70.42s system 95% cpu 3:19.95 total
end: Tue Aug 24 11:03:58 CEST 2021
start: Tue Aug 24 11:04:08 CEST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
| indexing messages; processed: 122049; updated/new: 122141; cleaned-up: 0
mu index  121.76s user 69.15s system 98% cpu 3:12.94 total
end: Tue Aug 24 11:07:21 CEST 2021
start: Tue Aug 24 11:07:31 CEST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
| indexing messages; processed: 122080; updated/new: 122141; cleaned-up: 0
mu index  121.95s user 66.02s system 101% cpu 3:04.95 total
end: Tue Aug 24 11:10:36 CEST 2021
start: Tue Aug 24 11:10:46 CEST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
\ indexing messages; processed: 122045; updated/new: 122141; cleaned-up: 0
mu index  125.05s user 68.13s system 101% cpu 3:09.70 total
end: Tue Aug 24 11:13:56 CEST 2021
start: Tue Aug 24 11:14:06 CEST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
\ indexing messages; processed: 122066; updated/new: 122141; cleaned-up: 0
mu index  122.90s user 66.12s system 101% cpu 3:05.70 total
end: Tue Aug 24 11:17:11 CEST 2021

So it seems that mu believes there are always around 120k new messages, which is certainly not the case, as there is no MUA running and no mbsync either. Can I ask mu index

I also see that mu index seems to use +/- exactly one core and wonder if that could be changed to support multi process/threading.

I am now running mu index --debug, but so far the output looks the same as if it was running without that flag

theophilusx commented 3 years ago

I think it look like the updated index information is not being written to the xapian database correctly. While an initial index will take a few minutes when you have 100K+ messages, subsequent runs should be almost instantaneous if there have been no changes.

Might be worthwhile posting output from mu info to check the schema version is correct.

I'm assuming when you updated to 1.6.3 you deleted the previous .cache/mu directory, re-ran mu init and then ran mu index?

telmich commented 3 years ago

I am actually on 1.6.1 (latest in Alpine Linux) and I did run mu init again after upgrading.

[11:25] nb3:~% mu info
maildir           : /home/nico/Maildir
database-path     : /home/nico/.cache/mu/xapian
schema-version    : 452
max-message-size  : 100000000
batch-size        : 250000
messages in store : 1509533
created           : Sat Aug 14 14:46:58 2021
personal-address  : <none>
[11:43] nb3:~% mu --version
mu (mail indexer/searcher) version 1.6.1
Copyright (C) 2008-2020 Dirk-Jan C. Binnema
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[11:43] nb3:~%
theophilusx commented 3 years ago

The fact every time you run mu index it re-indexes everything indicates something is wrong with your install. It is not 'normal' behaviour.

The schema version and other settings look correct, so either it is something wrong with the build provided by Alpine or something didn't work correctly with mu init.

I would clone the mu repository, build and install (after removing the alpine version). I'd then delete .cache/mu, re-run mu init and try mu index and see if that helps.

Out of interest, what file system are you running?

telmich commented 3 years ago

@theophilusx mu index does not re-index everythink - it only redoes 120k messages out of 1.5M.

Re filesystem: I am on ext4.

I have just compiled 1.6.4 and the issue still persists / mu re-indexes 120k messages every time:

[12:17] nb3:~% mu index        
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
/ indexing messages; processed: 122027; updated/new: 122141; cleaned-up: 0
[12:20] nb3:~% mu index
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
- indexing messages; processed: 122069; updated/new: 122141; cleaned-up: 0
[12:24] nb3:~% mu --version
mu (mail indexer/searcher) version 1.6.4
Copyright (C) 2008-2021 Dirk-Jan C. Binnema
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[12:31] nb3:~% mu info
maildir           : /home/nico/Maildir
database-path     : /home/nico/.cache/mu/xapian
schema-version    : 452
max-message-size  : 100000000
batch-size        : 250000
messages in store : 1509533
created           : Sat Aug 14 14:46:58 2021
personal-address  : <none>
[12:31] nb3:~%

I am now resetting ~/.cache/mu and re-indexing.

djcb commented 3 years ago

For the debug logs, check journal (if you use systemd) or the log file. Anyway, please check what those 120K messages are (logs should help) and what makes them different from the rest.

telmich commented 3 years ago

Just reran the loop with 1.6.4 and the 120k message re-index is still there:

[12:17] nb3:~% mu --version
mu (mail indexer/searcher) version 1.6.4
Copyright (C) 2008-2021 Dirk-Jan C. Binnema
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[12:17] nb3:~% rm -rf .cache/mu/
[12:32] nb3:~% mu init 
maildir           : /home/nico/Maildir
database-path     : /home/nico/.cache/mu/xapian
schema-version    : 452
max-message-size  : 100000000
batch-size        : 250000
messages in store : 0
created           : Tue Aug 24 12:32:50 2021
personal-address  : <none>

store created; use the 'index' command to fill/update it.
[12:32] nb3:~% date; mu index; date; echo "1st done"; date; mu index; date
Tue Aug 24 12:33:12 CEST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
/ indexing messages; processed: 1509533; updated/new: 1509533; cleaned-up: 0
Tue Aug 24 13:25:32 CEST 2021
1st done
Tue Aug 24 13:25:32 CEST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
| indexing messages; processed: 122070; updated/new: 122141; cleaned-up: 0
Tue Aug 24 13:30:24 CEST 2021

Now running mu index --debug to see what is happening

telmich commented 3 years ago

I am taking a random message that is added again:

2021-08-24 13:36:33 ** (mu:13905): DEBUG: 13:36:33.241: created message from /home/nico/Maildir/ungleich/2005/cur/1617069045.29086_200746.nb3,U=43607:2,S in 294 μs
2021-08-24 13:36:33 ** (mu:13905): DEBUG: 13:36:33.242: added message @ /home/nico/Maildir/ungleich/2005/cur/1617069045.29086_200746.nb3,U=43607:2,S; docid = 672503

As it is a public message, I attached the mail to this thread. mail.txt

Running it a third time, this time grep'ing for 43607:

2021-08-24 13:42:28 ** (mu:14658): DEBUG: 13:42:28.336: created message from /home/nico/Maildir/ungleich/2005/cur/1617069045.29086_200746.nb3,U=43607:2,S in 162 μs
2021-08-24 13:42:28 ** (mu:14658): DEBUG: 13:42:28.337: added message @ /home/nico/Maildir/ungleich/2005/cur/1617069045.29086_200746.nb3,U=43607:2,S; docid = 672503
telmich commented 3 years ago

For reference: the aport in Alpine Linux changed from 1.4.15 (does not have this problem) to 1.6.1 and the compile parameters did not change on version change:

commit f55dcc04958b57e3c1e26b7269b3ae296106366f
Author: Justin Berthault <justin.berthault@zaclys.net>
Date:   Mon Aug 2 12:54:46 2021 +0200

    community/mu: upgrade to 1.6.1

diff --git a/community/mu/APKBUILD b/community/mu/APKBUILD
index 5c1689b53e..4135f4a667 100644
--- a/community/mu/APKBUILD
+++ b/community/mu/APKBUILD
@@ -4,7 +4,7 @@
 # Contributor: Nico Schottelius <nico.schottelius@ungleich.ch>
 # Maintainer: Justin Berthault <justin.berthault@zaclys.net>
 pkgname=mu
-pkgver=1.4.15
+pkgver=1.6.1
 pkgrel=0
 pkgdesc="A collection of utilities for indexing and searching Maildirs"
 # armv7 and x86 blocked by gmime
@@ -66,4 +66,6 @@ mu4e() {
        mv "$pkgdir"/usr/share "$subpkgdir"/usr
 }

-sha512sums="dd0426c1fb7fe2430b47fc1232e067d1c017b3c428bbd8012c542f6e1ffa7741bb145c6851f40cbc945403534dabb4aa99faf369166167d187f083dde1d30546  mu-1.4.15.tar.gz"
+sha512sums="
+6d18f417dd1de73d6c16d1a6ba11a6b7489026b1f551e24f41cfb6cd57eb789f48801da59cbdd603c846d2a6248fee7ee6fa6f6e5de648b54ad8a8e14946e0bf  mu-1.6.1.tar.gz
+"
djcb commented 3 years ago

Thanks for checking. It seems that mu considers those message(s) not to be up to date for some reason.

Nothing special about that message, suspect it's the metadata. Are these message that need re-indexing always the same messages? Same maildir? Is anything touching those messages (check the time stamps)?

And/or are the message actually indexed and written etc., ie. can you mu find them after indexing?

telmich commented 3 years ago

Regarding the metadata:

[13:44] nb3:~% ls -l /home/nico/Maildir/ungleich/2005/cur/1617069045.29086_200746.nb3,U=43607:2,S 
-rw------- 1 nico nico 5081 Mar 30 03:50 '/home/nico/Maildir/ungleich/2005/cur/1617069045.29086_200746.nb3,U=43607:2,S'
[17:45] nb3:~% stat /home/nico/Maildir/ungleich/2005/cur/1617069045.29086_200746.nb3,U=43607:2,S 
  File: /home/nico/Maildir/ungleich/2005/cur/1617069045.29086_200746.nb3,U=43607:2,S
  Size: 5081        Blocks: 16         IO Block: 4096   regular file
Device: fd00h/64768d    Inode: 29406777    Links: 1
Access: (0600/-rw-------)  Uid: ( 1000/    nico)   Gid: ( 1000/    nico)
Access: 2021-08-24 11:00:08.465251405 +0200
Modify: 2021-03-30 03:50:45.596691524 +0200
Change: 2021-03-30 03:50:45.596691524 +0200
 Birth: -

I checked and the mail in question seems to be 4 times in the index, however re-indexing does not add it a 5th time:

[17:48] nb3:~% mu find from:john.e.maher@gmail.com "subject:automating gnupg decryption" 
Fri Nov 11 23:20:04 2005 John Maher <john.e.maher@gmail.com> Re: automating gnupg decryption
Fri Nov 11 23:20:04 2005 John Maher <john.e.maher@gmail.com> Re: automating gnupg decryption
Fri Nov 11 23:20:04 2005 John Maher <john.e.maher@gmail.com> Re: automating gnupg decryption
Fri Nov 11 23:20:04 2005 John Maher <john.e.maher@gmail.com> Re: automating gnupg decryption
[17:48] nb3:~% mu index
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
\ indexing messages; processed: 122007; updated/new: 122141; cleaned-up: 0
[17:51] nb3:~% mu find from:john.e.maher@gmail.com "subject:automating gnupg decryption"
Fri Nov 11 23:20:04 2005 John Maher <john.e.maher@gmail.com> Re: automating gnupg decryption
Fri Nov 11 23:20:04 2005 John Maher <john.e.maher@gmail.com> Re: automating gnupg decryption
Fri Nov 11 23:20:04 2005 John Maher <john.e.maher@gmail.com> Re: automating gnupg decryption
Fri Nov 11 23:20:04 2005 John Maher <john.e.maher@gmail.com> Re: automating gnupg decryption
[17:58] nb3:~%

The re-index seems to happen only in the ungleich/2005 maildir, essentially by checking using

tail -F mu.log | grep -v 2005
[18:02] nb3:~% ls -ltra Maildir/ungleich/2005/ 
total 19400
-rw-------  1 nico nico       18 Mar 29 19:14 .uidvalidity
drwx------  2 nico nico     4096 Mar 29 19:14 tmp
drwx------  2 nico nico 16302080 Mar 29 19:14 cur
-rw-------  1 nico nico  3541554 Mar 29 19:14 .mbsyncstate
drwxr-xr-x 36 nico nico     4096 Mar 29 22:12 ..
drwx------  2 nico nico     4096 Mar 30 03:49 new
drwx------  5 nico nico     4096 Aug 24 10:56 .

Is there a per maildir maximum limit somewhere? Because this maildir seems to be the biggest one:

[18:05] nb3:~% ls Maildir/ungleich/2005/cur/ | wc -l
234648

[18:05] nb3:ungleich% for i in *; do echo -n $i: ; find $i -type f | wc -l ; done 
1999:2
2000:34830
2001:25330
2002:226
2003:128
2004:96612
2005:234650
2006:158466
2007:195966
2008:181821
2009:131027
2010:155082
2011:25206
2012:23779
2013:34307
2014:58277
2015:27976
2016:15849
2017:15738
2018:8368
2019:15846
2020:21728
2021:20460
djcb commented 3 years ago

So you need to index the messages in the 2005 maildir, say, 4 times until the indexing won't try them again? And are the messages present (searchable) after, say, 1 times of indexing? In other words, is the problem that the messages don't make it to the store? And this is with 1.6.4?

But after that, is 2005 okay, or does the unwanted re-indexing return? (You might be able to avoid that with a .noupdate)

telmich commented 3 years ago

Sorry, above output might have been confusing. The situation is:

The weird thing is that mu4e 1.4.15 did work normal, since 1.6.1 something changed - it's still the same maildir.

telmich commented 3 years ago

I just downgraded to 1.4.15 and running mu index a second time takes about 30s:

[10:22] nb3:~% mu --version
mu (mail indexer/searcher) version 1.4.15
Copyright (C) 2008-2020 Dirk-Jan C. Binnema
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
[10:22] nb3:~% mu info     
database-path      : /home/nico/.cache/mu/xapian
messages in store  : 0
schema-version     : 451
created            : Wed Aug 25 10:22:42 2021
maildir            : /home/nico/Maildir
personal-addresses : <none>
[10:23] nb3:~% date; mu index; date
Wed Aug 25 10:23:09 CEST 2021
database-path      : /home/nico/.cache/mu/xapian
messages in store  : 0
schema-version     : 451
created            : Wed Aug 25 10:22:42 2021
maildir            : /home/nico/Maildir
personal-addresses : <none>
\ processing mail; processed: 1509000; updated/new: 1509000, cleaned-up: 0
cleaning up messages [/home/nico/.cache/mu/xapian]
| processing mail; processed: 1509031; updated/new: 0, cleaned-up: 0
elapsed: 14 second(s), ~ 107787 msg/s
/ processing mail; processed: 1509031; updated/new: 0, cleaned-up: 0
elapsed: 3351 second(s), ~ 450 msg/s
Wed Aug 25 11:19:01 CEST 2021
[11:19] nb3:~% date; mu index; date
Wed Aug 25 11:20:30 CEST 2021
database-path      : /home/nico/.cache/mu/xapian
messages in store  : 1509031
schema-version     : 451
created            : Wed Aug 25 10:22:42 2021
maildir            : /home/nico/Maildir
personal-addresses : <none>
\ processing mail; processed: 1509000; updated/new: 0, cleaned-up: 0
cleaning up messages [/home/nico/.cache/mu/xapian]
| processing mail; processed: 1509031; updated/new: 0, cleaned-up: 0
elapsed: 5 second(s), ~ 301806 msg/s
/ processing mail; processed: 1509031; updated/new: 0, cleaned-up: 0
elapsed: 29 second(s), ~ 52035 msg/s
Wed Aug 25 11:20:59 CEST 2021
[11:20] nb3:~%
telmich commented 3 years ago

Independently of the 1.4.x vs 1.6.x speed issue: In the postgres ticket it was mentioned that searching during index should be possible. Is that correct? Because in both 1.4.15 and 1.6.1/1.6.4 the mu4e search is completely blocking during index update.

Or is it that mu can search & index, but mu4e can't due to emacs single process design?

djcb commented 3 years ago

Okay, so mu does index the message, but then for some reason thinks they're not up to date. But not always. Hmmm....

mu4e cannot do things with the mu server when it's busy (e.g. with indexing); but that's unrelated to emacs' design and command-line mu.

djcb commented 3 years ago

Anyway, don't really understand what's going on your system; as a temp measure, you can drop a .update in your 2005 dir (see mu index manpage), that should avoid the prob.

telmich commented 3 years ago

I quickly did a comparison with notmuch, which is basically using the same indexer. The initial index time is similar (1.5h), however there are quite some differences afterwards:

[9:45] nb3:~% notmuch search --output=messages '*' | wc -l        
728206
[9:47] nb3:~% date; notmuch new; date           
Sat Aug 28 09:47:20 CEST 2021
Processed 26 total files in almost no time.
No new mail.
Sat Aug 28 09:47:20 CEST 2021

@djcb I am not too deep into the mu internals, but seeing above timing I wonder if notmuch is taking the atime/mtime of a folder to decide on whether or not to skip indexing. Would that possible make sense for mu as well?

Update: I did above comparison to see whether or not musl / Alpine Linux can be the triggering issue here - however with notmuch being fast, the problem is probably not in the xapian core.

telmich commented 3 years ago

I might have discovered another very slow case / dead lock: I hit bi for the inbox query about 20 minutes ago at which time an index was running. At this point right now the inbox query did not yet finish and it is now running for 3m+ with the mu server being idle. Quitting mu4e inside emacs and then starting again fails with the mu server still running.

I few minutes later, mu server is still running

nico      5462  1.2  0.2  54200 45608 ?        Ss   Aug30  49:29 /usr/bin/mu server

This is now with mu 1.4.15. After 5 minutes of waiting, I killed mu server.

djcb commented 3 years ago

A backtrace or it didn't happen! And I don't think the 1.4 series will see much change...

Oh, note, there's a change in 1.7.x (87ac89b0), which should be cherry-pickable for 1.6, and helps for at least some cases of re-indexing the same messages again and again.

telmich commented 2 years ago

A short update from 1.6.1 use: I did fully disable indexing and a query for a maildir:/... search still takes minutes, while mu find returns within a second. I am using a separate emacs instance to exclude any emacs daemon/sharing related issues.

I'll try to compile mu 1.7.x these days for comparison and come back to this ticket.

telmich commented 2 years ago

I just compiled 1.6.5-46-g191f0f47 and the query

flag:flagged OR maildir:/a/inbox OR maildir:/b/inbox OR maildir:/c/inbox OR maildir:/c/inbox

has been running for 2 minutes. The last mail retrieval is 2 hours ago, but mu server is running with 90% cpu and SSDs are 100% busy. I am not sure whether the index from 2 hours ago is still running or the query caused the high load.

After 7 minutes, I just got the result of the query. The same query executed with mu find returns within 1~2 seconds (even in parallel). Reloading the query now works fast.

The re-index of existing messages seems to be solved in this version:

[14:52] nb3:~% date; time mu index; date 
Sat Sep 18 14:53:41 KST 2021
indexing maildir /home/nico/Maildir -> store /home/nico/.cache/mu/xapian
- indexing messages; processed: 0; updated/new: 0; cleaned-up: 0
mu index  7.78s user 18.47s system 85% cpu 30.659 total
Sat Sep 18 14:54:12 KST 2021
[14:54] nb3:~%

Running mbsync -a outside of mu4e and then calling M-x mu4e-update-index blocks message loading for about 40s, which seems to be +/- in line with the indexing. However as noted above, this is not always the case with load times up to 7m.

I am running all tests now in an isolated emacs instance to avoid any possible interaction of other modes/modules.

I compared the run time with notmuch new that I did not run for some weeks and it ran in about 15s. I understand that this is the time xapian based solutions will need, but it would be great if there was any way to decouple indexing and loading messages / querying the index.

ArneBab commented 2 years ago

I see this in 1.6.6:

$ time mu index; time mu index
indexing maildir /home/arne/.local/share/mail -> store /home/arne/.cache/mu/xapian
/ indexing messages; processed: 65740; updated/new: 65743; cleaned-up: 0

real    3m59,968s
user    2m45,607s
sys 0m51,293s
indexing maildir /home/arne/.local/share/mail -> store /home/arne/.cache/mu/xapian
| indexing messages; processed: 65739; updated/new: 65743; cleaned-up: 0

real    3m19,634s
user    2m29,826s
sys 0m43,685s
djcb commented 2 years ago

Aha.... so it is indeed the same problem. For some reason, a subset of the messages is not recognized as being indexed already, and hence re-indexed each time.

ArneBab commented 2 years ago

How can I help track down which messages are not recognized?

djcb commented 2 years ago

You could try to run in --debug mode (from the command-line) or with (setq mu4e-mu-debug t) (in mu4e), and check what's in the log files.

I think it's about messages in specific directories, that somehow aren't recognized as "already seen" (when in lazy mode). It seems it's (re)indexing 65739 messages each time - are those all you have?

If you have time, you might want to give 1.7.3 a try, and see if/how that differs.

ArneBab commented 2 years ago

Sorry for the late answer; I was bogged down with other issues.

Those 65739 messages are not all. I have 420 094 messages in my maildir (verified with fd | wc -l).

Here’s an example message from a public mailing list that turned up in the log multiple times (parts of email addresses replaced by XXXXXXXXXXXXXXXX) and is located at ~/.local/share/mail/to.arne_bab/new/1593168968.5935_2.fluss:

Return-Path: <htcondor-users-bounces@cs.wisc.edu>
Received: from pop3.web.de [212.227.17.177]
    by localhost with POP3 (fetchmail-6.4.6)
    for <arne@localhost> (single-drop); Fri, 26 Jun 2020 12:56:08 +0200 (CEST)
Received: from jeeves.cs.wisc.edu ([128.105.6.16]) by mx-ha.web.de (mxweb112
 [212.227.17.8]) with ESMTPS (Nemesis) id 1MdMYE-1jFXaM2gnA-00ZKvt for
 <XXXXXXXXXXXXXXXXXXXXXXX@web.de>; Fri, 26 Jun 2020 01:28:11 +0200
Received: from jeeves.cs.wisc.edu (localhost [127.0.0.1])
    by jeeves.cs.wisc.edu (8.14.4/8.14.4) with ESMTP id 05PNLgek013419;
    Thu, 25 Jun 2020 18:22:23 -0500
Received: from shale.cs.wisc.edu (shale.cs.wisc.edu [128.105.6.25])
    by jeeves.cs.wisc.edu (8.14.4/8.14.4) with ESMTP id 05PNLaf0013414
    (version=TLSv1/SSLv3 cipher=AES256-GCM-SHA384 bits=256 verify=OK)
    for <htcondor-users@jeeves.cs.wisc.edu>; Thu, 25 Jun 2020 18:21:36 -0500
Received: from smtp7.wiscmail.wisc.edu (wmmta4.doit.wisc.edu [144.92.197.245])
    by shale.cs.wisc.edu (8.14.4/8.14.4) with ESMTP id 05PNLaMK013694
    (version=TLSv1/SSLv3 cipher=DHE-RSA-AES128-GCM-SHA256 bits=128
    verify=NO)
    for <htcondor-users@cs.wisc.edu>; Thu, 25 Jun 2020 18:21:36 -0500
Received: from USG02-CY1-obe.outbound.protection.office365.us
    ([23.103.209.108]) by smtp7.wiscmail.wisc.edu
    (Oracle Communications Messaging Server 8.0.2.4.20190812 64bit (built
    Aug 12
    2019)) with ESMTPS id <0QCI042LC8VUXFC0@smtp7.wiscmail.wisc.edu> for
    htcondor-users@cs.wisc.edu (ORCPT htcondor-users@cs.wisc.edu); Thu,
    25 Jun 2020 18:21:31 -0500 (CDT)
X-Spam-Report: IsSpam=no, Probability=11%, Hits= RETURN_RECEIPT 0.5,
    FROM_US_TLD 0.1, HTML_00_01 0.05, HTML_00_10 0.05, SUPERLONG_LINE 0.05, 
    BODYTEXTP_SIZE_3000_LESS 0, BODY_SIZE_10000_PLUS 0, DKIM_SIGNATURE 0,
    KNOWN_MTA_TFX 0, NO_URI_HTTPS 0, SPF_PASS 0, SXL_IP_TFX_WM 0,
    WEBMAIL_SOURCE 0, WEBMAIL_XOIP 0, WEBMAIL_X_IP_HDR 0, __ANY_URI 0,
    __ARCAUTH_DKIM_PASSED 0, __ARCAUTH_DMARC_PASSED 0, __ARCAUTH_PASSED 0, 
    __ATTACHMENT_SIZE_0_10K 0, __ATTACHMENT_SIZE_10_25K 0,
    __BODY_NO_MAILTO 0, 
    __CT 0, __CTYPE_HAS_BOUNDARY 0, __CTYPE_MULTIPART 0, __HAS_ATTACHMENT 0,
    __HAS_ATTACHMENT1 0, __HAS_ATTACHMENT2 0, __HAS_FROM 0, __HAS_MSGID 0, 
    __HAS_XOIP 0, __HIGHBITS 0, __MIME_TEXT_P 0, __MIME_TEXT_P1 0,
    __MIME_TEXT_P2 0, __MIME_VERSION 0, __MULTIPLE_RCPTS_TO_X2 0,
    __NO_HTML_TAG_RAW 0, __RETURN_RECEIPT_TO 0, __SANE_MSGID 0,
    __TO_MALFORMED_2 0, __TO_NAME 0, __TO_NAME_DIFF_FROM_ACC 0,
    __TO_NO_NAME 0, 
    __TO_REAL_NAMES 0, __URI_IN_BODY 0, __URI_MAILTO 0, __URI_NOT_IMG 0,
    __URI_NO_PATH 0, __URI_NS , __URI_WITHOUT_PATH 0
X-Wisc-Doma: XXXXXXXXXXXXXXXXXXXXXXXX@numerica.us,numerica.us
X-Wisc-Env-From-B64: d2VzbGV5LnRheWxvckBudW1lcmljYS51cw==
X-Spam-PmxInfo: Server=avs-13, Version=6.4.7.2805085,
    Antispam-Engine: 2.7.2.2107409, Antispam-Data: 2020.6.25.231519,
    AntiVirus-Engine: 5.74.0, AntiVirus-Data: 2020.6.25.5740002,
    SenderIP=[23.103.209.108]
X-Wisc-DKIM-Verify: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXX@numerica.us,numericaus.onmicrosoft.com!pass
X-Spam-Score: *
ARC-Seal: i=1; a=rsa-sha256; s=arcselector5401; d=microsoft.com; cv=none;
    b=KyXoddJsnsHsBwhdlO5rcljgMRaylJAUAxWTjG4jQL1C8XJAMgeERtH2sRffdjibYUFfSuDUNJmrTrvrbjKGUt2I8J2M2MgUB/upMoroVPNBrP1Fy9wMeZJQuSS4r4KjZZktsl2i8eq667pzOZO6+wX2IA5M7YtxDqglcWOE6btWzbABVjx+9eCXMt0eMd1+UI6ABK8Frd33EFQLKT0h/cxidWR9l+0gCMAcRxsLrQ82+ckU606AIV/DA1E4Tq7ADe/+CRv4QszDN93pWL/1N2/OOh9vFTs9g9ZG6uXjN+Km/IAdylPbfHgKW60ev3/Bvv6N3pA7DjpuiKj6BnW7mQ==
ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; 
    s=arcselector5401;
    h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
    bh=OZrj1we1ZUH0xBMhJ5/F6EQnB0cmitFs2xZW1fLMRNs=;
    b=Pq07a3u26s2UdpucJuVQ0h68272wx46Wp61x/30TelPPFLCRxVjmlH1U3IBmIsZ1jOEtGXFJRv65L3HmwGxRUdLlMOdPRB64BBfHQ9NGWUBykKQmOrJNGJs635nEdpugpzngzIdcg1PS5vHxPJAnOeqoo71OVPI3JqPrPEn2TJJgb9J6PApexkqIbVl35prGPsyS/t2IlYw3/ihWzORG6wvqJeqedgpJTBXeGaDoMa+MQ1BeUsdvybh8+hau4ASpM5lwyeXlGmJ5mUTZi39jp+dFdDrmCj/VM4ezeuXeH9+HFtDjKLZJaTDWUID0IBcr91BaoQE/4r6y+lpkah6LLQ==
ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass
    smtp.mailfrom=numerica.us;
    dmarc=pass action=none header.from=numerica.us; 
    dkim=pass header.d=numerica.us; arc=none
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
    d=numericaus.onmicrosoft.com; s=selector1-numericaus-onmicrosoft-com;
    h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck;
    bh=OZrj1we1ZUH0xBMhJ5/F6EQnB0cmitFs2xZW1fLMRNs=;
    b=cFn0eL5k2IKry9U8qa8mbVaxRiyicUAWzRc3NUtj+VEbgShfrz8SO6FPX20WTQQJg/Fu/3isqsSEUt+9NSEEbgd5eQ1EVz5E/JVeNjPe9GXR0JEF/g3f6yM7CO+kKTvXSRvQjce683U0j7Aj1pSDEktoVNP4xvOS2Gx9VjdWTmc=
Received: from DM3P110MB0474.NAMP110.PROD.OUTLOOK.COM (2001:489a:200:413::10)
    by DM3P110MB0490.NAMP110.PROD.OUTLOOK.COM (2001:489a:200:413::14)
    with Microsoft SMTP Server
    (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384)
    id 15.20.3109.25; Thu, 25 Jun 2020 23:21:07 +0000
Received: from DM3P110MB0474.NAMP110.PROD.OUTLOOK.COM
    ([fe80::f548:f084:9867:9375]) by DM3P110MB0474.NAMP110.PROD.OUTLOOK.COM
    ([fe80::f548:f084:9867:9375%11]) with mapi id 15.20.3131.024; Thu,
    25 Jun 2020 23:21:07 +0000
From: Wesley Taylor <XXXXXXXXXXXXXXXXXXXX@numerica.us>
To: "'htcondor-users@cs.wisc.edu'" <htcondor-users@cs.wisc.edu>
Thread-topic: OPINIONS WANTED: Are there any blatent downsides I am missing to
    the following Condor configuration
Thread-index: AdZLRbEvYoEDBZChS62aOHgPzKD8kw==
Date: Thu, 25 Jun 2020 23:21:06 +0000
Message-id: <DM3P110MB04746CDBA55B3E597EFD1877FA920@DM3P110MB0474.NAMP110.PROD.OUTLOOK.COM>
Accept-Language: en-US
Content-language: en-US
X-MS-Has-Attach: yes
X-MS-TNEF-Correlator: 
X-Originating-IP: [50.233.29.54]
x-ms-publictraffictype: Email
x-ms-office365-filtering-correlation-id: f4edecdf-5582-4b2d-226a-08d8195e7007
x-ms-traffictypediagnostic: DM3P110MB0490:
x-microsoft-antispam-prvs: <DM3P110MB04907C313C4243B4FE42A20CFA920@DM3P110MB0490.NAMP110.PROD.OUTLOOK.COM>
x-ms-oob-tlc-oobclassifiers: OLM:10000;
x-forefront-prvs: 0445A82F82
x-ms-exchange-senderadcheck: 1
x-microsoft-antispam: BCL:0;
X-Forefront-Antispam-Report: CIP:255.255.255.255; CTRY:; LANG:en; SCL:1; SRV:;
    IPV:NLI; SFV:NSPM; H:DM3P110MB0474.NAMP110.PROD.OUTLOOK.COM;
    PTR:; CAT:NONE; SFTY:;
    SFS:(346002)(366004)(6916009)(83380400001)(71200400001)(8936002)(55016002)(5660300002)(8676002)(9686003)(66616009)(64756008)(33656002)(52536014)(66446008)(66476007)(66556008)(66946007)(99936003)(76116006)(186003)(86362001)(26005)(7696005)(6506007)(44832011)(508600001)(2906002)(80162005)(80862006)(491001)(554374003);
    DIR:OUT; SFP:1102; 
x-ms-exchange-transport-forked: True
MIME-version: 1.0
X-OriginatorOrg: numerica.us
X-MS-Exchange-CrossTenant-Network-Message-Id: f4edecdf-5582-4b2d-226a-08d8195e7007
X-MS-Exchange-CrossTenant-OriginalArrivalTime: 25 Jun 2020 23:21:06.8341 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Hosted
X-MS-Exchange-CrossTenant-id: fae7a2ae-df1d-444e-91be-babb0900b9c2
X-MS-Exchange-CrossTenant-mailboxtype: HOSTED
X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM3P110MB0490
Subject: [HTCondor-users] OPINIONS WANTED: Are there any blatent downsides I
 am missing to the following Condor configuration
X-BeenThere: htcondor-users@cs.wisc.edu
X-Mailman-Version: 2.1.19
Precedence: list
List-Id: HTCondor-Users Mail List <htcondor-users.cs.wisc.edu>
List-Unsubscribe: <https://lists.cs.wisc.edu/mailman/options/htcondor-users>, 
    <mailto:htcondor-users-request@cs.wisc.edu?subject=unsubscribe>
List-Archive: <https://www-auth.cs.wisc.edu/lists/htcondor-users/>
List-Post: <mailto:htcondor-users@cs.wisc.edu>
List-Help: <mailto:htcondor-users-request@cs.wisc.edu?subject=help>
List-Subscribe: <https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users>,
    <mailto:htcondor-users-request@cs.wisc.edu?subject=subscribe>
Reply-To: HTCondor-Users Mail List <htcondor-users@cs.wisc.edu>
Content-Type: multipart/mixed; boundary="===============0678627779074767862=="
Errors-To: htcondor-users-bounces@cs.wisc.edu
Sender: "HTCondor-users" <htcondor-users-bounces@cs.wisc.edu>
Envelope-To: <XXXXXXXXXXXXXXXXXXXXXXXXXXXX@web.de>
X-UI-Filterresults: unknown:2;V03:K0:cdojl5YHfkg=:jhTbQXp38SL2za/LB4M7aUwpyw
 5rDHoN1+/ScH/O9/G1fKWbGryQ203thF+1ZrHUOOwq8MVOc5SsoqzSTsaNbEAdthFcDDz3Oui
 SHxX1hdpV3UOjZEHzWlpjEjRe7t74g2RI/ESELmkPuLg/LZC7SjAsg70cTJBIfDPYxJkJAcUl
 9W6OEBsmtTDO0va/EQRYjfkpoF9tjfmfMNw9KSKHuDdqZu2Xfak8mQKnWsoxWeUkD31r60iPC
 yikbj7KP5AlHaWMzyTTdlvtjYRLfSuUSe1uqjI5NWCnZDDjz7zODoaWPp7p2U/MQenXEjN6+M
 WnZL6ZC8AGtze/hYgOCXcLf4ydQ7m9YueJiY5nDn7g+cwnhxypVNFTL5NjSpKKXbkzbyu9Tdl
 ez+92g/9pGW17iOo5NrFtfctLlmCEH0RxjouKI7FBmv3bIvFC4FvfghiNf7OZmRg2/nT5i+1o
 AICYNAx2y5CezKsKM2f1tm60dkydQIR8pK45dDKZPz3i7NeJm9dknZ2OYFTnucUvdPaT8nR43
 cK3kk2QUE48Ngo/0NwepSGrV9TkOt+hY3PUYkXWp/mwP2QPSjy4cALyvLyKwG24qZ9CiiRLMV
 KqPFlCRnoDG5MHJ4d0krFlqmg8rNsWzV3oWfMNKFZmD24lVUmWGb+ExxbCFc0xzIt12o/EqBw
 nVkXLu+E0apM+cmG6ubYfOymRoUpiKsZI9ivc+mAaEE+v2RBzcAURzlhzQHIn81onvzbQwCge
 tMtBkSEfqyoa1HjalX4B9WQ90M42K+7xW039ydakQ7JOeYVpkPXYoBF7mbrXRckhMXjQatLQ8
 MWA8+U031Xfa1ueOIfCCkzJ49wyx1LoLPyqdjCvnzaRd72yNEMJ5zM/itMIPE9reIHBtpom0i
 RhIYdJFDrL+SKqE68lcJakCcF3R+VLApwLKOr0HChGQjdEk7c/rm5E0dF5f3oYlHf591QoXIJ
 h16yfcJYe6fMo1YYunkvbEDFPpzttIq7aIk0FzxrOdRvj3yQajDbwOpYI/5T/DaabPn3M8lK7
 8pn7LrbmyCaLHhkYMS4h3SDkYWsifza6vkldizrK7IPf6KhS7AhTkbnEonWS6454GLUg1nYGX
 W5Qp/G9LzvjtEGQMcwnCN5jb5zq7o3f+9FrROKjpwFxL+mL85CEXY/KMOVpf6hDJJfSyu6/X6
 FpbwlJVLFdGeA0/+xcKcmutpkJACgK2kHqvZ8MZxt+5jBJWVlIDLZKa8/IoGWC+ikLX2/hPNB
 4TU89QYG5ygPmwwDXruFG7N7jVURZceHqWNKtqegS6YQ5nirsPJWJR7jzgr+HbntUaQETXNpn
 QrxpsVHXfqRu2GlP5h28RaIpvBVUcwqrs+eLJELStvBzyAmaVPVoKFjEWFfwrmE89W6Bmz2W3
 kHExOq3hI3gDsGXKjTjT/kjHkaHmtnVUXr4vqovf8Ht4Vwmtf0S4xsgpYjnYjUIzG9eiwIFAZ
 hL2gvjwW51qtMvybf01C50xTiS9GSfO0SR7meBPA67skcA+wFo11wmwXsUk1irpKnC+Y9hVZX
 2vPkfZ1T2VXNo997cQC59lBpi/TU5gnuM7H/Vcl7tF3Lqtmqut7s6HkPWCegDZ3O2W7shH7aZ
 1bOXbO+W/SNC+WcMnj+fhuP+dHcrt0Vw4RD9knJOOzdZTH3OCli/vpjqgTbCKEaWMhCIeM2g0
 RiLFxTeTEEBCa49bwa8n2r4T/vA3duZd8F/DNKvWTfhRr1Mxtz3n15EOar13fFijtnieEiv4/
 vO/5uRF+H86Fcoua7B8AswThbiG1vou6M48g0Zo6iGEcrueKEaHMI4XM7wQF77KazMdn5f1BP
 +KyQX83aHJN/qGniXgF8yu+h0M7Nf0YrTteYQd2C/HZrIA8IaLqqvLoGRl7dRBnbZiP7jRdQm
 1YEYtjX4XBoShrXPfIxPnJBUBnnOaePYxOJkS2FaBv19jPkMnyc9xuJYD7JOTFnXKzAnoaBqT
 OR+dGrLLGZ1MM/0gqclKTv7Hcce+6CJyTWkx5mq42w49HFI/kdHBRxU8xIRv4B9l0ePf9EbWr
 cDcrssee//6KXiRmF4fm7jq828/uhj8MIJet9sIU5ncKwHEse3I4YmVT5+dB+ZGZh0gbJPFj6
 xcICpshhYct+euMCdNfy3lkxiRr76RwfBzLAOP5+1U3GAx/hcsL2AgyBHMwWo+Kkeq8pPy4YI
 pQMxJyylI6JMa/DbBggnDk+xNZpRKo/XA4lAJY57DCOPL2ZcL8kU2aCd5LjtYHK0ZWSFtOjxs
 oIEr/f2vvg+zibxzaANBzylZn3yPe9pI/IBefu9fL4MVaYY3aboxuncX4fyi0VH0WbFkSYXRi
 a7LIu3LI2LTU13C/LE7j9hmxP6TApyiXi14f0GSa2sbF6HWp2v2rhYM7h67AAn3SQgvcJLpgb
 Hz5ABb/OAk6ABVEl+a483zexJ6iT2P0gYc08zmewy8Jf8AD9r846k9pGZuhBaOHREx3bA16Bj
 uWYh3QzSI6MQoJM3XbBGLVkX36Lfj54T9kk97lLaxfbGPuNoyOV9iTBKxts3m2KD+52iH3EEi
 glbH6HNIUHyCHdEXsXyGVFwfM9V7OQcVO/g266KIQ74wU16x/Zdsq4p/1PcRXHRnoMxP/pUrj
 EOLWzFU71qzC/OSkYWRil9HXUyucTFGQ0N08jZNXctI9lElWtgq3iI+Cz2F20rz+LJGhSHSkZ
 0G5JgXrtspeJN5yoH6TOE0hblr5sZcAM0wiSP7x/hPBeYHswzTA5/laWMn++9aTPVgpPaJ9/x
 wyLm55OZr4Jl+StWd3MqLCgiRB3cNGrDX7f8Eqnj4wfCHiGIUHewD4qrfXraZQhIk17W+9JyD
 osmUiVD9ZRdNCY2eNnu8ZkJ4uzKl44lwLL43sInKBjdAHlnoxrR2FOrYXbnU31ujwxdeUr6Hs
 xPFy0Git0CpWCWYmaz37KA8GW7PE4ffWzcfCmz6AKBrbHcCreeUnyqnSEDy9ubnz7mcLRnu3W
 RAWi6diI8gcS9g0+r4z5PtZX9rveXRekHJ4k08VuYVmdiz3gjXmHPlm9IKPEAbygP2EYgjwGE
 RbReLc8xHJlfLbwdXyGw0HU=

--===============0678627779074767862==
Content-language: en-US
Content-type: multipart/signed; protocol="application/x-pkcs7-signature";
 micalg=2.16.840.1.101.3.4.2.3;
 boundary="----=_NextPart_000_0018_01D64B14.F58791A0"

------=_NextPart_000_0018_01D64B14.F58791A0
Content-Type: text/plain;
    charset="utf-8"
Content-Transfer-Encoding: quoted-printable

Hey!

I am architecting our final HTCondor configuration over here and I have =
an idea I am unsure about and I would like to ask some experienced users =
for their opinion.

Background, we have a small, relatively homogenous cluster (with no =
special universes) and less than 10 users. Since each user has their own =
workstation separate from our cluster I thought the following =
configuration would suit our needs, but I want to make sure there isn't =
a huge disadvantage I am missing:

1. Set the Central Manager to be highly available to the point of =
tolerating N cluster machine failures
2. Put a Submit on each of the users' workstations (I am a little =
worried about the resource usage of the condor_shadow and condor_schedd, =
my users are already running into RAM consumption issues over time as it =
is)
3. Place an Execute on each of the cluster machines, which would lead to =
the central manager being on a machine that is also executing jobs

Fortunately both my users' and cluster machines all have access to the =
same network storage, and we have centralized authentication so we can =
just use our users' credentials to authenticate everywhere.=20

Before I set this in dry mud, does anyone have any retrospective =
recommendations I could benefit hearing from, since I am still pretty =
new to the project?

Thank you!
-Wes

Wesley Taylor =E2=80=93 Cluster Manager
Numerica Corporation (www.numerica.us)
5042 Technology Parkway #100
Fort Collins, Colorado 80528
=E2=98=8E=EF=B8=8F (970) 207 2233
=F0=9F=93=A7 XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX@numerica.us

------=_NextPart_000_0018_01D64B14.F58791A0
Content-Type: application/pkcs7-signature;
    name="smime.p7s"
Content-Transfer-Encoding: base64
Content-Disposition: attachment;
    filename="smime.p7s"

MIAGCSqGSIb3DQEHAqCAMIACAQExDzANBglghkgBZQMEAgMFADCABgkqhkiG9w0BBwEAAKCCEv4w
ggWpMIIDkaADAgECAhAV2Tfkh0+gtEu0gskeSMTdMA0GCSqGSIb3DQEBCwUAMFsxEjAQBgoJkiaJ
k/IsZAEZFgJ1czEYMBYGCgmSJomT8ixkARkWCG51bWVyaWNhMRIwEAYKCZImiZPyLGQBGRYCYWQx
FzAVBgNVBAMTDmFkLUdJTEdBTEFELUNBMB4XDTE2MDcyNDE5NTcxM1oXDTM2MDcyNDIwMDcxMlow
WzESMBAGCgmSJomT8ixkARkWAnVzMRgwFgYKCZImiZPyLGQBGRYIbnVtZXJpY2ExEjAQBgoJkiaJ
k/IsZAEZFgJhZDEXMBUGA1UEAxMOYWQtR0lMR0FMQUQtQ0EwggIiMA0GCSqGSIb3DQEBAQUAA4IC
DwAwggIKAoICAQCq+/935KPrc8clxrq76k7GrrUHRbsM4FCfyrWicGPZsOKbJfcoloF2EAfj6AYR
QyU/l9um/8NqW+cu6/TY6YcY622L+UtT1QWC/Kt0kVL7cTtZN+VK/BkjcDVbUOqdeFY1q0tMzdco
WFxqjayGRYnX6oEZ7krDsGtJBBET/504Z3vDq/0ZD3lNG2dCWp1y+3VzUcb+OKkOPwMGHpw3gZM5
lZN/znB7d7qwxFSRoLzZZB3nZKKJHcp2ZuyJR+pCT5VdHGGV4gpVQKuL49/UoJBA0o8Kv0DGPByD
+LVwhlyFMi2jlnCd5lqiWRw9JAE3fqS/Di/cGbMjXMI2CplBj+GmZH8fgy4BQRwmsOUELTaYkJyJ
otcHGENO1+xYrR/lFEQLhh+8V2IJvBM2G1dgJ3EuEslL4q0xGeYLZJd7Z9xvXkAJaX/eWjHWICFI
zbsH/6fBqXYow/V8hfZhb20dGGnPESXPqMv/1mLgUIqr++Fjl6zKM5mYZuHlmrtd+eLgg7VsjDvh
cMxdQnju+jzJflxlmY2KSwt5lsu7viqmQyqVUnHFaEsV116B0uCROc5o1pBdRMdeeLrRoj6xPVlc
IzmIZz3wZERxCAWeJqBx5d1kXe+cDL4pMNQ/hmah4mshjtyOGv+oEgcdxzUQ72W7JNLhSv8C6gpU
eQwPq8usFAvUOwIDAQABo2kwZzATBgkrBgEEAYI3FAIEBh4EAEMAQTAOBgNVHQ8BAf8EBAMCAYYw
DwYDVR0TAQH/BAUwAwEB/zAdBgNVHQ4EFgQUF+CLMX/eZk96ElRSeiEHqnsujqEwEAYJKwYBBAGC
NxUBBAMCAQAwDQYJKoZIhvcNAQELBQADggIBACcwALtn+SFUx+YTrLCFY+Ghh4yubQt3YdEI6hOQ
JnmNPKsUEzCvoRE5L2ZLkG2VhJNX3KAJmXgkZMCGBPbiA/65r/cbYqZATQEG/g9aVicz/IBHXvg4
7+YDDN9VpRy8c93AZNNTRf83Pw+CDsdIGG7mg8rc0tiCgt0V3gN0wF8oRSsb/trqd+ujk41bvaPw
Rl+8JUeRN0Pq9lH4VGGk9GEIQv8JXhr2VKFmJcGKLB+qvMRvWQZ5oPTGDE3pUYI5q8f7/fMiJKU6
hb9l+tXP7uDLWIawg/MoUc2BwAThyXFk9LZhkYWYpzbaf2Ez2JYieD4ey8RjEKvis9mF6Z/p6+69
GbYvuf2bRikYenrmboXCUO820totjP2UyHczexZsMP/XznmyDJuN+BDLzLjm7ks8lXDwpF/Kqnjm
1EyiQI0OB4cn889yM039U7raJeHpuiwju2/YO6krE+plLQhkM7pl6v6Ly/ZKICwDfbcU8k8LE4+K
3VaXmVYRYbSXx8l2Ke0CWKNfehBGQ024gKjNt8t7gCgInG5s+roumqeKyfCWlhYll1FAxEQmwP/6
966y7uJrGLra0VUjdppbZpAENSF0pdX08VfsasSZ20hnCaLWO1b3i0ZOBLBAoNzeCm+BdS6DAOhy
JnHHZ+OBoiaYwCSjSvTDmHyQkNK3wmu+/wyNMIIGnDCCBISgAwIBAgITbwAAAEFhCq43is5OqAAA
AAAAQTANBgkqhkiG9w0BAQsFADBbMRIwEAYKCZImiZPyLGQBGRYCdXMxGDAWBgoJkiaJk/IsZAEZ
FghudW1lcmljYTESMBAGCgmSJomT8ixkARkWAmFkMRcwFQYDVQQDEw5hZC1HSUxHQUxBRC1DQTAe
Fw0xOTA3MjIxNDE4MDFaFw0yMTA3MjIxNDI4MDFaMFwxEjAQBgoJkiaJk/IsZAEZFgJ1czEYMBYG
CgmSJomT8ixkARkWCG51bWVyaWNhMRIwEAYKCZImiZPyLGQBGRYCYWQxGDAWBgNVBAMTD2FkLUNF
TEVCUklBTi1DQTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAKRLgjg0yC0P2jLwTCIA
V/zEGk/PEc3pZxNAo7m0I/SXdNulUEkjxai5Wq53i0EhWVLpUU8XY3joXax46yCMqh0PUn90QmMD
BybLyFDX6av8tVS5cQs0HbTZdIuj7A/dsKzKKIrSHd3SQ9MLNPRkSRdhagmf5LCF1Y4xEEiuAA/H
XdYAxGIcl8n6b2CcLlZzq4W13Ipv8FIZoDsG1u0b9NGfeSOOHidi5kdD6r8lM5PaSPmZsl5PdKK6
+E1Y6rBCvITu0MBo5Tjuwt5cok3Ve0BK5Fg89aIL2/rMicm20qG6nbqxLhHeR0mhPO98KIIzDoeL
rLpAlWS7GoPvJqbRzxsCAwEAAaOCAlYwggJSMBAGCSsGAQQBgjcVAQQDAgEBMCMGCSsGAQQBgjcV
AgQWBBSv5TU1Bjnw5n3u1iO2y+BHQXk7MTAdBgNVHQ4EFgQUoeMyqBhiyBcgwJN8zbr7pRbgs+sw
GQYJKwYBBAGCNxQCBAweCgBTAHUAYgBDAEEwDgYDVR0PAQH/BAQDAgGGMA8GA1UdEwEB/wQFMAMB
Af8wHwYDVR0jBBgwFoAUF+CLMX/eZk96ElRSeiEHqnsujqEwgdMGA1UdHwSByzCByDCBxaCBwqCB
v4aBvGxkYXA6Ly8vQ049YWQtR0lMR0FMQUQtQ0EsQ049R2lsZ2FsYWQsQ049Q0RQLENOPVB1Ymxp
YyUyMEtleSUyMFNlcnZpY2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9YWQsREM9
bnVtZXJpY2EsREM9dXM/Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENsYXNz
PWNSTERpc3RyaWJ1dGlvblBvaW50MIHGBggrBgEFBQcBAQSBuTCBtjCBswYIKwYBBQUHMAKGgaZs
ZGFwOi8vL0NOPWFkLUdJTEdBTEFELUNBLENOPUFJQSxDTj1QdWJsaWMlMjBLZXklMjBTZXJ2aWNl
cyxDTj1TZXJ2aWNlcyxDTj1Db25maWd1cmF0aW9uLERDPWFkLERDPW51bWVyaWNhLERDPXVzP2NB
Q2VydGlmaWNhdGU/YmFzZT9vYmplY3RDbGFzcz1jZXJ0aWZpY2F0aW9uQXV0aG9yaXR5MA0GCSqG
SIb3DQEBCwUAA4ICAQBmRoSlPe++k7tsAJOvq0+0dNI6yk6gOBmY4g5jL9NTEjSxPWkeYegIwLr2
UqpiIIZmAh9e9v3z0T2egVyRqNezLPXLkg/2gUfV6D0kRyKtG5mL0yAn/0hkkVyf6jWJpCKmH77x
0w3UpnfKs79jv5YpQDhC2eRFivN50HhIkigLWScPq4zd81ghmN8VFTHVQmsGua/mm1Oj5/pBFuQF
B4ljon1N//wX5ZJZaUlJR9eR9tM9m+Gyds2flr5+mZT6Zgm26fKiC5zs91aGnzqGx6s30jfXELP2
FjFrrR46ooV7ehhnyBlCACxIWqXe5sSZsSh9oEYZ7Ux5Vq0thkfArBWsF7HA+LovKCUyHLcXbVBB
6/VAwZ3GLYi/bqbVIEFlVRu4nv/JyKWwoGbAhGyzZNWoeHszFrEIQbQMoMsEumVkMZreE6AxP+zb
6JPPOjlhpymtMo54z1MDYJPyo4HmcpL4xUjHZgqgOxMrbHC4oIVLvKZ/scbVBhPnd0tHHSZqj3ZS
gfTvG/ut/tLNTXXe48PkLBw4KguhbLm61Elu3wJALT0UL+ENgUWwb7csUGQBqOyPAHXGYnf/ACOc
UBqQckcrK8Jq3u8rnCloW3uDw86hw7MFM+YjmhVRdYRxpJmhKVPT6Amufp2WsSVId8q3CSqTH33L
fcxbV1n7hLWHA67MhTCCBq0wggWVoAMCAQICEycAAAsJMaw2RjtHZFUAAQAACwkwDQYJKoZIhvcN
AQELBQAwXDESMBAGCgmSJomT8ixkARkWAnVzMRgwFgYKCZImiZPyLGQBGRYIbnVtZXJpY2ExEjAQ
BgoJkiaJk/IsZAEZFgJhZDEYMBYGA1UEAxMPYWQtQ0VMRUJSSUFOLUNBMB4XDTIwMDUxMjE1MDk0
MloXDTIxMDcyMjE0MjgwMVowgcExEjAQBgoJkiaJk/IsZAEZFgJ1czEYMBYGCgmSJomT8ixkARkW
CG51bWVyaWNhMRIwEAYKCZImiZPyLGQBGRYCYWQxETAPBgNVBAsTCE51bWVyaWNhMQ4wDAYDVQQL
EwVVc2VyczEYMBYGA1UECxMPUHJlc2VudCBJbnRlcm5zMRYwFAYDVQQDEw1XZXNsZXkgVGF5bG9y
MSgwJgYJKoZIhvcNAQkBFhl3ZXNsZXkudGF5bG9yQG51bWVyaWNhLnVzMIIBIjANBgkqhkiG9w0B
AQEFAAOCAQ8AMIIBCgKCAQEA5clDLapXkiLVXhAFP9GJv+JJkt+cacyvWaX9xEvqMQXOXb7MqO5E
DJE8XPMfxaX84WhuMMePOc9SNUKpDtTa2SHz+AOom+JH38ce2gfrdOPwez/e6RrUb3o8ZvMr3hJl
Yy+6vEFEADIICfHSlIjkLJbGNFTRDccvkOPjD2W+fmzFAtWyNb/eqM+mwdTuXjOxTvP6V34zJsvc
YKJUzhhD8jI7GdqOoNoirTlaMVTH5udK0P2KvzD6F0LfwcOlc3bTvY9uI585xhdniK4yAIka8OMq
5zmyEQLYOadcVSscjAlkC1sQ0gbwL3AdwS+bntryq+2Ds380OJ+Z1Uy7TRkeBQIDAQABo4IDADCC
AvwwPAYJKwYBBAGCNxUHBC8wLQYlKwYBBAGCNxUI9/Bss4wDhbmBGISeqheH4YBfgSWC6qJEgcjE
IgIBZQIBKDATBgNVHSUEDDAKBggrBgEFBQcDBDAOBgNVHQ8BAf8EBAMCBaAwGwYJKwYBBAGCNxUK
BA4wDDAKBggrBgEFBQcDBDBEBgkqhkiG9w0BCQ8ENzA1MA4GCCqGSIb3DQMCAgIAgDAOBggqhkiG
9w0DBAICAIAwBwYFKw4DAgcwCgYIKoZIhvcNAwcwHQYDVR0OBBYEFDZHoDwoOKD5uzpF/2CcZSeg
XWLmMB8GA1UdIwQYMBaAFKHjMqgYYsgXIMCTfM26+6UW4LPrMIHVBgNVHR8Egc0wgcowgceggcSg
gcGGgb5sZGFwOi8vL0NOPWFkLUNFTEVCUklBTi1DQSxDTj1DZWxlYnJpYW4sQ049Q0RQLENOPVB1
YmxpYyUyMEtleSUyMFNlcnZpY2VzLENOPVNlcnZpY2VzLENOPUNvbmZpZ3VyYXRpb24sREM9YWQs
REM9bnVtZXJpY2EsREM9dXM/Y2VydGlmaWNhdGVSZXZvY2F0aW9uTGlzdD9iYXNlP29iamVjdENs
YXNzPWNSTERpc3RyaWJ1dGlvblBvaW50MIHHBggrBgEFBQcBAQSBujCBtzCBtAYIKwYBBQUHMAKG
gadsZGFwOi8vL0NOPWFkLUNFTEVCUklBTi1DQSxDTj1BSUEsQ049UHVibGljJTIwS2V5JTIwU2Vy
dmljZXMsQ049U2VydmljZXMsQ049Q29uZmlndXJhdGlvbixEQz1hZCxEQz1udW1lcmljYSxEQz11
cz9jQUNlcnRpZmljYXRlP2Jhc2U/b2JqZWN0Q2xhc3M9Y2VydGlmaWNhdGlvbkF1dGhvcml0eTBS
BgNVHREESzBJoCwGCisGAQQBgjcUAgOgHgwcd2VzbGV5LnRheWxvckBhZC5udW1lcmljYS51c4EZ
d2VzbGV5LnRheWxvckBudW1lcmljYS51czANBgkqhkiG9w0BAQsFAAOCAQEAX3zFhiDYU+vQap2J
hiysyC9L7nkL7VI2OQWg4Z/JnNJTFiA6BwtoDYAT4qq1Jix4hZc+g78Gj99OnkhlBQDe9Hq12yI9
muboQSDAYO6iDK76wQv3Rt8Fl4SUD4Ygwy52QrkTDrj/HZxTNask5p/2ilGBJnG9KT2VbEgGJkP9
kXn1vAgOl3BCxgjdWekWCvxpmffr+Z3UtmQIiZAB3OsKcgdsSy9pveTMjxtKJemaH3kpXQiTgCev
CMuWZb3YnqXI8Fd+uUw6HwA4c+ZH62G9Q8KGkwXyhOPizmm3UeSlMo27yUCE+cF5EIHBxpGJ6z83
7MbxMVKnS1Wz1n8MtW2ezDGCBCEwggQdAgEBMHMwXDESMBAGCgmSJomT8ixkARkWAnVzMRgwFgYK
CZImiZPyLGQBGRYIbnVtZXJpY2ExEjAQBgoJkiaJk/IsZAEZFgJhZDEYMBYGA1UEAxMPYWQtQ0VM
RUJSSUFOLUNBAhMnAAALCTGsNkY7R2RVAAEAAAsJMA0GCWCGSAFlAwQCAwUAoIICfzAYBgkqhkiG
9w0BCQMxCwYJKoZIhvcNAQcBMBwGCSqGSIb3DQEJBTEPFw0yMDA2MjUyMzIwNDRaME8GCSqGSIb3
DQEJBDFCBEBaj66vdgjAhEO0p7lO6X44h+LpUlAcROa5Hi4Jp5aWS4hU8CuqOrH12y2GRNmNhKLa
0YieL4fCL3YqDRfop79NMFIGCyqGSIb3DQEJEAIBMUMwQQQdAAAAABAAAACgLzslsB99TKIYKeHy
Wh5cAQAAAACAAQAwHTAbgRl3ZXNsZXkudGF5bG9yQG51bWVyaWNhLnVzMIGCBgkrBgEEAYI3EAQx
dTBzMFwxEjAQBgoJkiaJk/IsZAEZFgJ1czEYMBYGCgmSJomT8ixkARkWCG51bWVyaWNhMRIwEAYK
CZImiZPyLGQBGRYCYWQxGDAWBgNVBAMTD2FkLUNFTEVCUklBTi1DQQITJwAACwkxrDZGO0dkVQAB
AAALCTCBhAYLKoZIhvcNAQkQAgsxdaBzMFwxEjAQBgoJkiaJk/IsZAEZFgJ1czEYMBYGCgmSJomT
8ixkARkWCG51bWVyaWNhMRIwEAYKCZImiZPyLGQBGRYCYWQxGDAWBgNVBAMTD2FkLUNFTEVCUklB
Ti1DQQITJwAACwkxrDZGO0dkVQABAAALCTCBkwYJKoZIhvcNAQkPMYGFMIGCMAsGCWCGSAFlAwQB
KjALBglghkgBZQMEARYwCgYIKoZIhvcNAwcwCwYJYIZIAWUDBAECMA4GCCqGSIb3DQMCAgIAgDAN
BggqhkiG9w0DAgIBQDALBglghkgBZQMEAgMwCwYJYIZIAWUDBAICMAsGCWCGSAFlAwQCATAHBgUr
DgMCGjANBgkqhkiG9w0BAQEFAASCAQBNFxhcbK6Rmw0Xyu+79cH5kUsXENcdUaJPKlegcY/gl2BZ
0CPpGcRnwz6z8OPYjvw3jrkiAE8nBbuCKu1CPtuk1h4Cybk7exyMybYvK5xge+N+dz2mFipRfGSY
rl/ztX1jyvcDruxaSJwb8WMhAGs505yfaCJfwgFOI3QGi+wUunbOIKy3QQZTXDv89yslZqi0wmeI
8sVRqSAYZRIPEylwS9CU2ReK9BJlfVLZnNP1At4gHE6S2hk8T0eVeLT8uhQiUXXJe4644UoPhoA4
Fxgm7Q62KT6yP9O7c4eZzmQ4A9hdlWM6CtZ5pgMAzLOrVFdypzSc+S1j8DqcFkALCw83AAAAAAAA

------=_NextPart_000_0018_01D64B14.F58791A0--

--===============0678627779074767862==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
HTCondor-users mailing list
To unsubscribe, send a message to htcondor-users-request@cs.wisc.edu with a
subject: Unsubscribe
You can also unsubscribe by visiting
https://lists.cs.wisc.edu/mailman/listinfo/htcondor-users

The archives can be found at:
https://lists.cs.wisc.edu/archive/htcondor-users/
--===============0678627779074767862==--

This is with mu (mail indexer/searcher) version 1.6.10

djcb commented 2 years ago

If you run mu with --debug and/or mu4e with mu4e-mu-debug set to non-nil, you should see a bit more output from the indexer (in the journal if you use systemd, or in the log file) and why it decides to visit / index certain directories or not.

ArneBab commented 2 years ago

I don’t see that — it just shows that it enters the directory: mu index --debug and tail -F ~/.cache/mu/mu.log:

2022-03-13 22:42:03 ** (mu:10906): DEBUG: 22:42:03.434: process ~/.local/share/mail/archiv/cur/tmp
2022-03-13 22:42:03 ** (mu:10906): DEBUG: 22:42:03.436: created message from ~/.local/share/mail/archiv/cur/1375740521.17978_27494.kaverne:2, in 1863 μs
2022-03-13 22:42:03 ** (mu:10906): DEBUG: 22:42:03.481: added message @ ~/.local/share/mail/archiv/cur/1375740521.17978_27494.kaverne:2,; docid = 16782
2022-03-13 22:42:03 ** (mu:10906): DEBUG: 22:42:03.482: started worker
2022-03-13 22:42:03 ** (mu:10906): DEBUG: 22:42:03.486: created message from ~/.local/share/mail/archiv/cur/1375740303.17978_23610.kaverne:2, in 4986 μs
2022-03-13 22:42:03 ** (mu:10906): DEBUG: 22:42:03.496: added message @ ~/.local/share/mail/archiv/cur/1375740303.17978_23610.kaverne:2,; docid = 16786

When I run this again, I see the same message with the same docid:

$ tail -F ~/.cache/mu/mu.log  | grep 1375740303.17978_23610
2022-03-13 22:49:44 ** (mu:15315): DEBUG: 22:49:44.493: created message from ~/.local/share/mail/archiv/cur/1375740303.17978_23610.kaverne:2, in 4105 μs
2022-03-13 22:49:44 ** (mu:15315): DEBUG: 22:49:44.501: added message @ ~/.local/share/mail/archiv/cur/1375740303.17978_23610.kaverne:2,; docid = 16786
ArneBab commented 2 years ago

Is there something else I can do to give you additional debugging information?

The only error I see is:

2022-03-13 22:56:06  ** (mu:9264): WARNING **: 22:56:06.437: error adding message @ ~/.local/share/mail/drafts/cur/1569906673.82520a3fc48e6b21.fluss:2,DS: failed to create message: message seems invalid, ignoring (~/.local/share/mail/drafts/cur/1569906673.82520a3fc48e6b21.fluss:2,DS)
ArneBab commented 2 years ago

Could you replicate the problem with this info?

djcb commented 2 years ago

Hmmm, it's hard to say. From these logs, there's nothing immediately suspicious....

Did you ever try the 1.7.x (git master) series? That one has a bunch of changes to indexing.

ArneBab commented 2 years ago

mu 1.7.x is not yet in guix — once it is, I’ll try right away.

If you copy the mail I quoted to your maildir, does parsing it fail, too?

djcb commented 2 years ago

No, it seems to work just fine. Does it not for you?

ArneBab commented 2 years ago

No, it gets indexed again and again on every run of mu index — as one of now 32393 files that get indexed and named as "updated/new" on every run.