astroidmail / astroid

A graphical threads-with-tags style, lightweight and fast, e-mail client for Notmuch
http://astroidmail.github.io
Other
613 stars 65 forks source link

Unread emails not marked as read in Maildir #31

Closed hugoroy closed 8 years ago

hugoroy commented 8 years ago

I've noticed that reading an email in astroid removes the notmuch 'unread' tag but does not seem to affect the status of the email as 'new' in maildir.

Is there something wrong with my notmuch setup or should this be handled by astroid?

Edit: apparently, astroid does not synchronize tag changes back to maildir flags with notmuch_message_tags_to_maildir_flags according to j4ni on #notmuch on IRC

gauteh commented 8 years ago

Do you mean that it still has the tag 'new' or that the unread flag is not removed on the server? I usually remove the 'new' tag after having done any tagging on new mail in the notmuch post-new hook or my poll script.

Neither really is the responsibility of astroid, notmuch can synchronize some of the tags to maildir flags [0]. These will normally show up on the server side as well. There are some configuration flags for notmuch to set that up.

[0] https://notmuchmail.org/special-tags/

Hugo Roy writes on January 3, 2016 16:33:

I've noticed that reading an email in astroid removes the notmuch 'unread' tag but does not seem to affect the status of the email as 'new' in maildir.

Is there something wrong with my notmuch setup or should this be handled by astroid?


Reply to this email directly or view it on GitHub: https://github.com/gauteh/astroid/issues/31

hugoroy commented 8 years ago

No, I remove the notmuch 'new' tag with a post-new hook. What I am concerned with are emails which are tagged as "unread" and which are, well, unread.

In Maildir, such emails will be in the "new" directory.

When I open and read such emails, astroid will correctly remove the "unread" tag but this isn't synced with the Maildir "new" directory, so when I sync this with the server (with offlineimap) the read/unread status isn't changed.

As the link suggests, notmuch can take care of "Synchronization between tags and maildir flags". The page says:

The above tags are recognized by the lib, and synchronized with the corresponding maildir flags through the following lib function calls:

   notmuch_message_maildir_flags_to_tags()
   notmuch_message_tags_to_maildir_flags()
gauteh commented 8 years ago

Hugo Roy writes on January 3, 2016 21:31:

No, I remove the notmuch 'new' tag is removed with a post-new hook. What I am concerned with are emails which are tagged as "unread" and which are, well, unread.

In Maildir, such emails will be in the "new" directory.

When I open and read such emails, astroid will correctly remove the "unread" tag but this isn't synced with the Maildir "new" directory, so when I sync this with the server (with offlineimap) the read/unread status isn't changed.

As the link suggests, notmuch can take care of "Synchronization between tags and maildir flags". The page says:

The above tags are recognized by the lib, and synchronized with the corresponding maildir flags through the following lib function calls:

   notmuch_message_maildir_flags_to_tags()
   notmuch_message_tags_to_maildir_flags()

Hm, I don't think this is something astroid should do. You can have notmuch dirs that don't have the new/ cur/ tmp/ structure. I am sure this has been discussed at notmuch, have you found anything there?

hugoroy commented 8 years ago

From IRC:

21:48 if you read email using astroid, and astroid changes tags directly using libnotmuch (and not the notmuch cli commands), astroid needs to call notmuch_message_tags_to_maildir_flags to sync the tag changes to flag chances

I already have this in notmuch-config:

[maildir]
synchronize_flags=true
gauteh commented 8 years ago

Hugo Roy writes on January 3, 2016 22:35:

From IRC:

21:48 if you read email using astroid, and astroid changes tags directly using libnotmuch (and not the notmuch cli commands), astroid needs to call notmuch_message_tags_to_maildir_flags to sync the tag changes to flag chances

Hm, ok! That seems cumbersome to me, I would expect notmuch new to do that. Needs to be fixed though!

hugoroy commented 8 years ago

Yes, for me too which is why i wasn't sure where the issue was exactly.

I suppose this is a valid discussion to continue in #notmuch :-)

gauteh commented 8 years ago

Hugo Roy writes on January 4, 2016 10:33:

Yes, for me too which is why i wasn't sure where the issue was exactly.

I suppose this is a valid discussion to continue in #notmuch :-)

After some discussion there it seems that some way of doing this automatically through libnotmuch would be preferable, but there are a bunch of considerations and tedious coding on the notmuch side to be done. Short answer, won't be done for a while anyway, so any code modifying tags should call the sync_back function as well. Some considerations:

hugoroy commented 8 years ago

After some discussion there it seems that some way of doing this automatically through libnotmuch would be preferable,

Yes that's what I expected too which is why I was surprised it didn't work with astroid.

but there are a bunch of considerations and tedious coding on the notmuch side to be done. Short answer, won't be done for a while anyway, so any code modifying tags should call the sync_back function as well.

OK. Well, I think this makes sense because it looks like quite a blocker for people like me using several devices and interfaces for email (currently at work, using webmail) and where at least the unread/read status needs to be in sync.

config.cc should read ~/.notmuch-config (or $NOTMUCH_CONFIG) and use it for a bunch of default values

Yes, this would be most useful. Maybe also for name, primary email, etc.

db.sync_back_tag_to_maildir_flags=[yes|no|notmuch]

with "notmuch" being default, meaning it is read from notmuch config.

  • config.cc should not read notmuch-config when testing or in debug mode.

it makes sense to me but I'm not a developer :)

  • files may be renamed after this, meaning that a sent message might not have the name after the call.

This may be related to some issues that I've run into with notmuch while using mutt sometimes (getting errors complaining about no file or directory, etc.). But I can't help much here.

Anyway, thanks a lot for looking into this. It looks like I may switch to astroid completely on my personal laptop instead of just experimenting :)

gauteh commented 8 years ago

Hugo Roy writes on January 4, 2016 17:09:

This may be related to some issues that I've run into with notmuch while using mutt sometimes (getting errors complaining about no file or directory, etc.). But I can't help much here.

Anyway, thanks a lot for looking into this. It looks like I may switch to astroid completely on my personal laptop instead of just experimenting :)

Cool, the fastest thing to do is to just have a yes|no option without integration to the notmuch-config yet. Might have to go for that first as an intermediate step.

hugoroy commented 8 years ago

↪ janvier 4, 2016 10:25 , Gaute Hope:

Cool, the fastest thing to do is to just have a yes|no option without integration to the notmuch-config yet. Might have to go for that first as an intermediate step.

If that means astroid's unread email will also be synced back through maildir as unread in other devices, I'm all for easiest and fastest :)

ff2000 commented 8 years ago

Having an option to parse notmuch-config for unset config options would be really great. See e.g. #63 As I said there I don't think it's good to do that automatically by default, but one could ask the user. But if you look at https://notmuchmail.org/pipermail/notmuch/2016/021617.html the notmuch-devs don't think relying on ~/.notmuch-config is great, because it's only for the CLI and has no specified format.

gauteh commented 8 years ago

Yeah, perhaps that is not such a bad idea. One option to read values from ~/.notmuch-config (or one specfied!) into the config and overwrite the current/default values.

gauteh commented 8 years ago

Gaute Hope writes on January 17, 2016 15:12:

Yeah, perhaps that is not such a bad idea. One option to read values from ~/.notmuch-config (or one specfied!) into the config and overwrite the current/default values.

I think there is a boost ini parser that could read the .notmuch-config.

ff2000 commented 8 years ago

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values from ~/.notmuch-config (or one specfied!) into the config and overwrite the current/default values.

I think it would be better the other way around, so the user can overwrite settings from notmuch-config in astroid. Reading notmuch-config should be labeled as fallback solution.

gauteh commented 8 years ago

ff2000 writes on January 22, 2016 9:57:

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values from ~/.notmuch-config (or one specfied!) into the config and overwrite the current/default values.

I think it would be better the other way around, so the user can overwrite settings from notmuch-config in astroid. Reading notmuch-config should be labeled as fallback solution.

Yes. I think the easiest would be to provide the possibility to user to run e.g.:

$ astroid --read-notmuch-config /optional/path/to/config

then the relevant values are read into the existing (or fresh) astroid config. I think it could easily lead to some weird behaviour if we read it every time. In that case we would need to specify one astroid config and one notmuch config, since a user might have several notmuch setups specified using environment variables or command line arguments to notmuch.

Relevant values are:

So if you make changes to your notmuch config, you could re-run astroid --read-notmuch-config and get them propagated into the astroid config.

Untill notmuch provides a way to access the config through the library, I think the best is to still keep some detachment from the config file.

ff2000 commented 8 years ago

Excerpts from Gaute Hope's message of Januar 22, 2016 12:28 :

ff2000 writes on January 22, 2016 9:57:

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values from ~/.notmuch-config (or one specfied!) into the config and overwrite the current/default values.

I think it would be better the other way around, so the user can overwrite settings from notmuch-config in astroid. Reading notmuch-config should be labeled as fallback solution.

Yes. I think the easiest would be to provide the possibility to user to run e.g.:

$ astroid --read-notmuch-config /optional/path/to/config

then the relevant values are read into the existing (or fresh) astroid config. I think it could easily lead to some weird behaviour if we read it every time. In that case we would need to specify one astroid config and one notmuch config, since a user might have several notmuch setups specified using environment variables or command line arguments to notmuch.

That's not what I've been thinking of:

Don't know if that makes sense to you.

Relevant values are:

  • account information
  • mail db
  • synchronize tags
  • muted tags
  • ??

So if you make changes to your notmuch config, you could re-run astroid --read-notmuch-config and get them propagated into the astroid config.

Untill notmuch provides a way to access the config through the library, I think the best is to still keep some detachment from the config file.


Reply to this email directly or view it on GitHub: https://github.com/gauteh/astroid/issues/31#issuecomment-173889181

gauteh commented 8 years ago

Ah, yeah. That works. But lets specify the path of the fallback config.

Den lørdag 23. januar 2016 skrev ff2000 notifications@github.com følgende:

Excerpts from Gaute Hope's message of Januar 22, 2016 12:28 :

ff2000 writes on January 22, 2016 9:57:

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values from ~/.notmuch-config (or one specfied!) into the config and overwrite the current/default values.

I think it would be better the other way around, so the user can overwrite settings from notmuch-config in astroid. Reading notmuch-config should be labeled as fallback solution.

Yes. I think the easiest would be to provide the possibility to user to run e.g.:

$ astroid --read-notmuch-config /optional/path/to/config

then the relevant values are read into the existing (or fresh) astroid config. I think it could easily lead to some weird behaviour if we read it every time. In that case we would need to specify one astroid config and one notmuch config, since a user might have several notmuch setups specified using environment variables or command line arguments to notmuch.

That's not what I've been thinking of:

  • add a config like "astroid.notmuch.fallback_config", default is ""
  • read that config file (if specified) and populate db, excluded_tags etc. but do not write it to astroid-config
  • if the user set e.g. astroid.notmuch.excluded_tags use that instead of the value read from notmuch_config
  • probably one could add a program_option to change to a different notmuch.fallback_config.

Don't know if that makes sense to you.

Relevant values are:

  • account information
  • mail db
  • synchronize tags
  • muted tags
  • ??

So if you make changes to your notmuch config, you could re-run astroid --read-notmuch-config and get them propagated into the astroid config.

Untill notmuch provides a way to access the config through the library, I think the best is to still keep some detachment from the config file.


Reply to this email directly or view it on GitHub: https://github.com/gauteh/astroid/issues/31#issuecomment-173889181

— Reply to this email directly or view it on GitHub https://github.com/gauteh/astroid/issues/31#issuecomment-174170788.

ff2000 commented 8 years ago

Excerpts from Gaute Hope's message of Januar 23, 2016 11:59 :

Ah, yeah. That works. But lets specify the path of the fallback config.

Specifying the path by default would also resolve #63 automagically ;) So probably not that bad.

Though this could potentially kill some special setups, where the db set in ~/.notmuch-config is not what the user wants to use in astroid. As that is set in astroid config, it will be overwritten by the logic described in my last post, so everything is fine, again. But if he changes now to a custom notmuch.[fallback_]config but forgets to unset notmuch.db he will probably get unexpected results... Also we probably want to completely ignore notmuch.*-config options if the user specifies --notmuch-config cmdline-arg (if we implement it).

Another option would be deprecation of notmuch.* config options. Then we can read them + warn that "this will be removed in Astroid 0.7, please use astroid.notmuch.config" or similar. If the user really wants some very special settings just for astroid, he can copy ~/.notmuch-config, apply changes and specify that file in "astroid.notmuch.config".

(And sorry for thinking out loud - there are many ways to deal with this, and to break users setup ;))

Den lørdag 23. januar 2016 skrev ff2000 notifications@github.com følgende:

Excerpts from Gaute Hope's message of Januar 22, 2016 12:28 :

ff2000 writes on January 22, 2016 9:57:

Excerpts from Gaute Hope's message of Januar 17, 2016 3:12 :

Yeah, perhaps that is not such a bad idea. One option to read values from ~/.notmuch-config (or one specfied!) into the config and overwrite the current/default values.

I think it would be better the other way around, so the user can overwrite settings from notmuch-config in astroid. Reading notmuch-config should be labeled as fallback solution.

Yes. I think the easiest would be to provide the possibility to user to run e.g.:

$ astroid --read-notmuch-config /optional/path/to/config

then the relevant values are read into the existing (or fresh) astroid config. I think it could easily lead to some weird behaviour if we read it every time. In that case we would need to specify one astroid config and one notmuch config, since a user might have several notmuch setups specified using environment variables or command line arguments to notmuch.

That's not what I've been thinking of:

  • add a config like "astroid.notmuch.fallback_config", default is ""
  • read that config file (if specified) and populate db, excluded_tags etc. but do not write it to astroid-config
  • if the user set e.g. astroid.notmuch.excluded_tags use that instead of the value read from notmuch_config
  • probably one could add a program_option to change to a different notmuch.fallback_config.

Don't know if that makes sense to you.

Relevant values are:

  • account information
  • mail db
  • synchronize tags
  • muted tags
  • ??

So if you make changes to your notmuch config, you could re-run astroid --read-notmuch-config and get them propagated into the astroid config.

Untill notmuch provides a way to access the config through the library, I think the best is to still keep some detachment from the config file.


Reply to this email directly or view it on GitHub: https://github.com/gauteh/astroid/issues/31#issuecomment-173889181

— Reply to this email directly or view it on GitHub https://github.com/gauteh/astroid/issues/31#issuecomment-174170788.


Reply to this email directly or view it on GitHub: https://github.com/gauteh/astroid/issues/31#issuecomment-174171545

ff2000 commented 8 years ago

I just started to refactor config access:

That way we should be able to access the config as usual (except direct "astroid.notmuch.db" which we can avoid or catch in Astroid::config()), avoid writing notmuch-config values into astroid-config (seperate ptree-object) - and restrict access to Astroid::config-object.

If you agree with the approach I will continue later. (Need some break, think I'm falling ill...)

gauteh commented 8 years ago

Looks good, afk this weekend so, no rush on my part.

Den lørdag 23. januar 2016 skrev ff2000 notifications@github.com følgende:

I just started to refactor config access:

  • make Astroid::config a private member
  • add ptree Config::notmuch_config which is populated from ~/.notmuch_config
  • create function "Astroid::config(string path)" to return specific ptree object:
  • config.notmuch_config if path == astroid.notmuch
  • else config.get_child (path)

That way we should be able to access the config as usual (except direct "astroid.notmuch.db" which we can avoid or catch in Astroid::config()), avoid writing notmuch-config values into astroid-config (seperate ptree-object) - and restrict access to Astroid::config-object.

If you agree with the approach I will continue later. (Need some break, think I'm falling ill...)

— Reply to this email directly or view it on GitHub https://github.com/gauteh/astroid/issues/31#issuecomment-174189143.

ff2000 commented 8 years ago

@hugoroy you might want to test #74 Everything works here, tests pass. If anything fails please comment in the pull request.

Note: That branch reads the notmuch config file. It defaults to ~/.notmuch-config. So make sure maildir.synchronize_flags is set to true there.

make-github-pseudonymous-again commented 8 years ago

Tested for unread/read mail, it works.

hugoroy commented 8 years ago

↪ janvier 24, 2016 2:32 , Aurélien Ooms:

Tested for unread/read mail, it works.

This works for me, including for flaggeed status as well :)

ff2000 commented 8 years ago

Aurélien, Hugo: Thx for testing. So everything works, and you didn't have any weird issues? Great :) Notmuch will automatically sync tags to flags for these cases:

(copied from /usr/include/notmuch.h)

@gauteh There are two more changes I plan:

So I think a seperate PR makes sense.

gauteh commented 8 years ago

ff2000 writes on January 25, 2016 8:32:

Aurélien, Hugo: Thx for testing. So everything works, and you didn't have any weird issues? Great :) Notmuch will automatically sync tags to flags for these cases:

  • 'D' iff the message has the "draft" tag
  • 'F' iff the message has the "flagged" tag
  • 'P' iff the message has the "passed" tag
  • 'R' iff the message has the "replied" tag
  • 'S' iff the message does not have the "unread" tag

(copied from /usr/include/notmuch.h)

@gauteh There are two more changes I plan:

  • Performance: first iterate over messages, then over tags. That way I do not have to sync one message multiple times, just because several tags changed.
  • Safety: Freeze message while tagging (notmuch_message_freeze)

So I think a seperate PR makes sense.

Excellent!

I think this should go in a separate PR, but you can base it off the refactored-config PR.

Some issues:

So, some questions or approaches to that: is the notmuch message and thread object still valid after sync has been run? in that case we can make sure that filenames are never cached, but always queried from the db.

if not, we might have to reload the thread/message on every sync call that resulted in a change. perhaps this can be achieved through a message_filename_changed (string messageid, string newfname) signal.

regards, gaute

gauteh commented 8 years ago

By the way; ner had a fairly nice (but perhaps incomplete) c++ wrapper for the notmuch library: https://github.com/gauteh/astroid/pull/44#issuecomment-170337316

hugoroy commented 8 years ago

This was working but it seems that it was not merged in the last release

gauteh commented 8 years ago

Yes, the plan was to do this in a separete PR. There were some more places where tags could change that were not taken into account, as well as a few implications related to the potential change in filename.

I don't know if @ff2000 was planning to do something on this..?

-g

gauteh commented 8 years ago

With the merge of #95 there are Actions for all write operations, this should make it easier to do the flag-sync.

hugoroy commented 8 years ago

That would be awesome. This is 1 of the 3 reasons why I still need to open mutt nowadays :-)

(the other 2 being: sending encrypted email, and getting rids of some drafts that for some reason aren't tagged properly)

gauteh commented 8 years ago

Hugo Roy writes on March 14, 2016 23:25:

That would be awesome. This is 1 of the 3 reasons why I still need to open mutt nowadays :-)

1)

Some comments on the patch from @ff2000:

(the other 2 being: sending encrypted email, and getting rids of some drafts that for some reason aren't tagged properly)

2) Yes, that would be nice.

3) You can manually tag the message as a draft from the thread-view.

mkazlauskas commented 8 years ago

I've made myself a little workaround script that reset unread tag using notmuch cli tool before syncing with offlineimap, maybe it could be useful to anyone as I guess it can be used as astroids polling script: gist

gauteh commented 8 years ago

Please test #112.

hugoroy commented 8 years ago

↪ avril 4, 2016 4:18 , Gaute Hope:

Please test #112.

Yes, the basics work (marking read/unread). I'll test for some time and see if any issue comes up.

Thank you!

gauteh commented 8 years ago

Fixed.

hugoroy commented 8 years ago

Thank you!

make-github-pseudonymous-again commented 8 years ago

Thank you