Closed napcae closed 3 years ago
@napcae, I think the problem here is fastmail's name of IMAP inbox folder is "Inbox" (i.e. capitalized) instead of Papermerge's default value of "INBOX" (all uppercase).
Try setting option PAPERMERGE_IMPORT_MAIL_INBOX
to:
PAPERMERGE_IMPORT_MAIL_INBOX = "Inbox"
i.e. Capitalized.
It is a valid bug anyway, because (Papermerge) application should provide proper feedback when it cannot select IMAP inbox folder. Thank you for opening this ticket!
Hey @ciur, thanks for taking a look. I don't think that's the problem though, I've started with a (imap) folder called "papermerge", i.e. PAPERMERGE_IMPORT_MAIL_INBOX = "papermerge"
and still had the same error.
I've just tried PAPERMERGE_IMPORT_MAIL_INBOX = "Inbox"
and the same error comes up:
papermerge_worker | [2021-02-13 10:29:18,251: ERROR/ForkPoolWorker-2] Task papermerge.core.management.commands.worker.import_from_email[b6313ea1-7c2f-4e11-970b-e615379e0b38] raised unexpected: TypeError('sequence item 0: expected str instance, bytes found')
Unless TypeError is a red herring, I'd suspect something different than capitalization be the problem, no?
Edit:
Listing imap folders shows that the default name is INBOX
as well:
irb(main):012:0> pp imap.list('%', '%')
[#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren],
delim="/",
name="INBOX">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren, :Archive],
delim="/",
name="Archive">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren],
delim="/",
name="redacted*">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren, :Drafts],
delim="/",
name="Drafts">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren, :Junk],
delim="/",
name="Junk Mail">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren, :Xnotes],
delim="/",
name="Notes">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren],
delim="/",
name="redacted*">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren, :Sent],
delim="/",
name="Sent Items">,
#<struct Net::IMAP::MailboxList
attr=[:Haschildren],
delim="/",
name="redacted*">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren, :Trash],
delim="/",
name="Trash">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren],
delim="/",
name="papermerge">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren],
delim="/",
name="*redacted*">,
#<struct Net::IMAP::MailboxList
attr=[:Hasnochildren],
delim="/",
name="travel">]
So I was digging a little bit more, turns out you need RW imap access. For testing purposes I had created a RO user which leads to the failure.
According to rfc3501, Section 6.4.5:
The \Seen flag is implicitly set; if this causes the flags to
change, they SHOULD be included as part of the FETCH responses.
Assuming fastmail implemented this per spec, that is the culprit.
I've verified my hypothesis by setting readonly=true
:
server.select_folder(settings.PAPERMERGE_IMPORT_MAIL_INBOX, readonly=True)
The importer won't crash but will import mails indefinitely since it won't set the read flag on the (imap) server.
How do you want to fix this? I'm not too familiar with python to fix this upstream(in imapclient), but an exception catch might be enough, wdyt @ciur ?
@napcae, I think that wrapping server.select_folder(...) into a try ... except
block with logging.error(<error + explanation what may be the cause of it and what user can do to fix it>)
will suffice.
Going with readonly=True
IMHO defeats the purpose of fetching documents from IMAP account.
If you go on with above fix please don't forget to update documentation. with your findings that IMAP account requires RW access and there will be XYZ warning otherwise.
Going with readonly=True IMHO defeats the purpose of fetching documents from IMAP account.
I'm aware, the explanation was for demonstration purpose only
One more thing, I mentioned it in my first post - the worker won't recognize mail configuration directives for mail, i.e. IMPORT_MAIL_HOST
, IMPORT_MAIL_USER
etc. without the PAPERMERGE
prefix. Is the documentation out of that or what is it that I'm missing here?
@napcae
One more thing, I mentioned it in my first post - the worker won't recognize mail configuration directives for mail, i.e. IMPORT_MAIL_HOST, IMPORT_MAIL_USER etc. without the PAPERMERGE prefix. Is the documentation out of that or what is it that I'm missing here?
Yes. There is difference where you place those settings. Settings can be either in:
In papermerge.conf.py
file configuration settings are without PAPERMERGE_
prefix, because all (well, 90%) of them are papermerge specific.
In django settings file however, there are all sort of settings - for celery (prefixed with CELERY), for allauth (prefixed with ACCOUNT). Respectively settings for specific for papermerge are prefixed as well.
You are right that above information would be great addition to the documentation. In fact I just updated and deployed documentation based on above explanation :)
Alright, thanks for clarification and updating the documentation. I've opened a PR as you can see above this comment. It won't fix the issue but at least it spits out some useful information for the user/administrator.
In case you experience issues with docker image provided by linuxserver.io/papermerge, please open bug report in their repository.
Description I've tried to setup a basic IMAP importer. Running on the supplied docker images, I've set
config/worker.production.py
(This is another bug maybe, if I set the vars without the PAPERMERGE_ prefix, the app is complaining in app.log that the vars are not set and therefore no IMAP importer will be configured.)
RAW Mail
``` Return-Path: <*redacated*@me.com> Received: from compute3.internal (compute3.nyi.internal [10.202.2.43]) by sloti35d2t02 (Cyrus 3.5.0-alpha0-141-gf094924a34-fm-20210210.001-gf094924a) with LMTPA; Thu, 11 Feb 2021 11:51:27 -0500 X-Cyrus-Session-Id: sloti35d2t02-1613062287-2734191-2-12295162176795636088 X-Sieve: CMU Sieve 3.0 X-Spam-known-sender: no X-Spam-sender-reputation: 500 (none) X-Spam-score: 0.0 X-Spam-hits: BAYES_05 -0.5, FREEMAIL_FROM 0.001, ME_SENDERREP_NEUTRAL 0.001, RCVD_IN_DNSWL_LOW -0.7, RCVD_IN_MSPIKE_H3 0.001, RCVD_IN_MSPIKE_WL 0.001, SPF_HELO_NONE 0.001, SPF_PASS -0.001, T_FREEMAIL_DOC_PDF 0.01, LANGUAGES unknown, BAYES_USED user, SA_VERSION 3.4.2 X-Spam-source: IP='17.58.63.179', Host='st43p00im-ztfb10063301.me.com', Country='US', FromHeader='com', MailFrom='com' X-Spam-charsets: plain='us-ascii', plain='us-ascii' X-Attached: Scanned Document 3.pdf X-Resolved-to: napcae+papermerge@fastmail.fm X-Delivered-to: papermerge@napcae.fastmail.fm X-Mail-from: *redacated*@me.com Received: from mx4 ([10.202.2.203]) by compute3.internal (LMTPProxy); Thu, 11 Feb 2021 11:51:27 -0500 Received: from mx4.messagingengine.com (localhost [127.0.0.1]) by mailmx.nyi.internal (Postfix) with ESMTP id 2D9BB7C00D7 forExpected An imported document via IMAP.
Actual No documents are imported:
Info: