How can I move emails to another folder when archiving?

[guillaumecherel]

My IMAP server has quotas and I wont be able to store all my mails there as their quantity will grow. I use offlineimap to sync it to a local maildir and use sup on this maildir. I would like to create another separate "archive" maildir, which won't be synchronized with a server. How can I move emails that I'm archiving in sup to this separate maildir? I've thought of a couple of ways but I couldn't figure out how to realize them:

• Interactive way: use a hook to automatically move emails when they are archived (I couldn't find an appropriate hook).
• Batch way: get the list of messages that don't containt the label "inbox" and that are in the first maildir (the one which is synced with the server), get their corresponding filenames, and move them.

I think the first approach is the best. Is there any way I could make it work, with hooks or another approach (maybe writing a function that does archive & move and rebind the 'a' key to it)?

Thanks

[rakoo]

I would always favor the second solution, if only because it's compatible with your existing mails. I think the simplest way would be to have a after-archive hook that is run once archiving one (or multiple) messages is successful. You wouldn't necessarily need to rebind a.

Note that a poll after deleting mails from the first maildir may be necessary to update sup's index

[guillaumecherel]

Ok. Moving the messages with a shell script is easy once I know the filenames. How can I get them?

[ppwwyyxx]

I'm also trying to have similar functionality in sup. #211 might be a helpful: It looks like we can override the default "archive" keybinding to a custom operation.

As for the second solution, it looks like you'll have to iterate over the whole xapian index. I suspect it won't be very efficient.

Please share if you're having progress. Thanks.

[guillaumecherel]

It seems that the core problem is how to get the relevant filenames, perhaps by adding a batch mode in sup that performs a search and then output the files (or any other info like whole mails, subjects or any field...) to stdout. Then I could write a script "sup-archive" that does all I need.

I haven't had time to work on it lately but I will post my solution here if I get to one.

[ppwwyyxx]

I did find that you can get the filename from xapian index. For example you can iterate over the index like this (code modified from sup-dump):

xapian = Xapian::Database.new File.join(BASE_DIR, 'xapian')
xapian.postlist('Kmail').each do |x|
  begin
    entry = Marshal.load(xapian.document(x.docid).data)
    puts entry[:subject], entry[:labels], entry[:locations]
  rescue
    $stderr.puts "failed to dump document #{x.docid}"
  end
end

The locations key would be the filename.

[gauteh]

Check out devel/console.sh and devel/start-console.rb. Also the ~ keybinding.

[ppwwyyxx]

Are you suggesting that we use the console for a custom archive method? Is it possible to simply move the thread file to a different maildir folder, while sup is still running? Are there any synchronization issues I need to take care, or how do I notice sup of these changes?

[guillaumecherel]

I've got some progress. I can modify the script sup-dump so that it outputs the relative path to the email file (relative to the source path) rather than the message id by doing:

      puts "#{entry[:locations].map{|l| l[1] }}"

at line 39. My first question is, I've noticed some emails can have more than one location. Why is that? If I want to move the email files to another location, should I move them all?

I would like now to print the absolute path. I saw that there exists a SourceManager and that I can get the source path with SourceManager(source_id).file_path. And I can get the email's corresponding source_id as the first element in each element of entry[:location]. I expected the following to print the full path:

      puts "#{entry[:locations].map{|l| SourceManager[l[0]].file_path + l[1] }}"

But there seems to be a problem like SourceManager not being defined in sup-dump. Any idea how I can make this work?

I've no experience with Ruby, so sorry if my questions are naive.

[gauteh]

guillaumecherel writes on mai 10, 2016 11:13:

I've got some progress. I can modify the script sup-dump so that it outputs the relative path to the email file (relative to the source path) rather than the message id by doing:

      puts "#{entry[:locations].map{|l| l[1] }}"

at line 39. My first question is, I've noticed some emails can have more than one location. Why is that? If I want to move the email files to another location, should I move them all?

That depends on your setup, if you move the same messsage to the maildir-location you might create some conflicts.

If you for instance send an email message to yourself there will be one copy in the sent-dir and one in the inbox, then you will have two copies of the same.

Also, someone might send you an email with the same message-id as already exists (e.g. the sender might know you are subscribed to a mailinglist), then one of the messages will shadow the other one - and you cannot be sure that they are identical. Especially mailinglist tend to add a footer to the original email.

You will have to decide how you want to archive the email in this case. I am not sure how your IMAP server will react with several messages with the same MID the same folder.

[rakoo]

I am not sure how your IMAP server will react with several messages with the same MID the same folder.

IMAP doesn't have a special handling of message-ids (other than the fact that it's an interesting field that is returned in some commands), it works with IDs and UIDs.

[ralfebert]

Following two very experimental code examples for achieving something like this.

First, the following script iterates all mails via the xapian index and if a mail is in SOURCE_INBOX but doesn't have the label inbox assigned, it moves it to the archive maildir folder and changes the index metadata accordingly. offlineimap would then delete the mail from the inbox folder and upload it again to the archive folder (which works fine but seems unnecessary):

require 'xapian'
require 'fileutils'

# TODO: source ids hard-coded
SOURCE_INBOX = 1
SOURCE_ARCHIVE = 3

xapian = Xapian::WritableDatabase.new('xapian', Xapian::DB_OPEN)
xapian.postlist('Kmail').each do |x|
  doc = xapian.document(x.docid)
  entry = Marshal.load(doc.data)
  entry[:locations].each do |loc|
    if loc.first == SOURCE_INBOX and !entry[:labels].include?(:inbox)
      puts "Archiving: #{entry[:date]} #{entry[:subject]}"
      inbox_path = File.join(File.expand_path("~"), "Mail/INBOX", loc[1])
      archive_path = File.join(File.expand_path("~"), "Mail/Archive", loc[1])
      if File.exist?(inbox_path)
        doc.add_term "I#{SOURCE_ARCHIVE}"
        begin
          doc.remove_term "I#{SOURCE_INBOX}"
        rescue Exception => e
          p e
        end
        FileUtils.mv(inbox_path, archive_path)
        loc[0] = SOURCE_ARCHIVE
        doc.data = Marshal.dump(entry)
      end
    end
    xapian.replace_document x.docid, doc
  end
end
xapian.flush()
xapian.close()

While this worked very well, I wasn't too happy with the solution, esp. as I want to use offlineimap more and more only for syncing from remote to local. So I built an 'archive' hook that moves the message directly on the IMAP server from sup. When syncing the next time, the move will be detected by offlineimap and sup. Very rough first sketch:

.sup/hooks/startup.rb

require 'net/imap'
require 'keychain'

class Redwood::Thread
  def imap_archive
    self.each do |msg|
      if msg && msg.has_label?(:inbox)
        server = 'imap.example.com'
        imap = Net::IMAP.new(server, ssl: true)
        keychain = Keychain.internet_passwords.where(server: server).first
        raise "No password in keychain" unless keychain
        imap.authenticate('PLAIN', keychain.account, keychain.password)
        imap.select("INBOX")
        imap.search(['HEADER', 'MESSAGE-ID', msg.id]).each do |nr|
          imap.move(nr, "Archive")
          BufferManager.flash "Archived #{msg.id}"
        end
        imap.close
      end
    end
  end
end

class Redwood::ThreadIndexMode
  def imap_archive
    thread = cursor_thread or return
    thread.imap_archive
    read_and_archive
  end
end

class Redwood::ThreadViewMode
  def imap_archive
    @thread.imap_archive
    self.archive_and_next
  end
end

.sup/hooks/keybindings.rb:

require 'pry'
modes["inbox-mode"].keymap.add! :imap_archive, "Archive on IMAP server", 'a'
modes["thread-view-mode"].keymap.add! :imap_archive, "Archive on IMAP server", 'a'
modes["search-results-mode"].keymap.add! :imap_archive, "Archive on IMAP server", 'a'

[guillaumecherel]

I've discovered notmuch (https://notmuchmail.org/) a few days ago. It was inspired by sup and does the job of indexing and searching for mails but offers (only) a command line interface. It's thus suited for using in scripts to automatically move emails matching certain tags (or other search criteria) to another directory. It doesn't have a graphical user interface, but you can find several as separate projects (including some for vim and emacs). I've only tried alot (https://github.com/pazz/alot) which covers all my needs.

So this issue is solved for me. Feel free to close it. I'm leaving it open because my solution is not using sup anymore, and in case someone wants to keep working on it with sup.

[klaxalk]

Hey, is there some update to this issue? I would appreciate the option to have a clean inbox on multiple machines, which, sadly, cannot be achieved without moving email between folders.