Use uids in fetch

[dimitrisstr]

Maybe fetch should use uids instead of numbers. The following piece of code throws this exception imap_tools.errors.MailboxFetchError: Response status "OK" expected, but "NO" received. Data: [b'The specified message set is invalid.']

with MailBox(host).login(email, password) as mailbox:
    for msg in mailbox.fetch(criteria="UNSEEN"):
        mailbox.move(msg.uid, "folder")

[ikvk]

move signature: def move(self, uid_list: Union[str, Iterable[str]], ... I think uid_list is quite a telling name

Your mailbox may not support move by uid

Note that uid may return None (rare)

[dimitrisstr]

Sorry, I should have been more thorough. The issue is similar to issue 133. My inbox has 100 unseen emails and the loop throws an exception when it is trying to fetch the 51st email. I could copy emails in fetch iterations and delete them after the iteration. The problem is that someone may move an email manually from the UI while the script is running. That's why, i was wondering if the fetch method should use the uids method instead of the numbers method.

[ikvk]

As I sad at 133: Collect needed uids and call move once.

Show me how are you using numbers method. Move works with uids only.

[dimitrisstr]

I am referring to the implementation of the fetch method. It finds all the emails that meet the criteria using the numbers method.

[ikvk]

message_parts = "(BODY{}[{}] UID FLAGS RFC822.SIZE)".format( ... and then parse UID at MailMessage.uid

fetch already can search by uids with uid search criteria: mailbox.fetch(A(uid=['1', '2'])) https://github.com/ikvk/imap_tools#search-criteria

What exactly do you propose to do and how will it help?

[ikvk]

# from
nums = tuple((reversed if reverse else iter)(self.numbers(criteria, charset)))[limit_range]

# to
uids = tuple((reversed if reverse else iter)(self.uids(criteria, charset)))[limit_range]

?

[dimitrisstr]

Yes, this is exactly what i had in mind.

[ikvk]

Needs to read RFC here, i will think about it. And I hope you will help me with it.

[dimitrisstr]

I'd be glad to help you.

[ikvk]

@dimitrisstr

1. Suppose i retrived uids at fetch:

   message_parts = "(BODY{}[{}] UID FLAGS RFC822.SIZE)".format(
   '' if mark_seen else '.PEEK', 'HEADER' if headers_only else '')
   limit_range = slice(0, limit) if type(limit) is int else limit or slice(None)
   assert type(limit_range) is slice
   uids = tuple((reversed if reverse else iter)(self.uids(criteria, charset, True)))[limit_range]
   for fetch_item in (self._fetch_in_bulk if bulk else self._fetch_by_one)(uids, message_parts, reverse):  # noqa
   yield self.email_message_class(fetch_item)

How do you offer to fetch message data without message number by uid at _fetch_in_bulk and _fetch_by_one? At line fetch_result =

2. I think all your problem that you try to delete letters by on in a loop.

[dimitrisstr]

1. You can use self.client.uid('fetch', uid, message_parts). I think that the real problem is the miss_no_uid parameter of the fetch method. It will be useless, if fetch uses uids instead of message numbers.
2. My problem is that I try to automate some tasks, and at the same time, while the script is running, someone may move or delete an email using the UI. The workaround i use right now is to set the limit parameter, of the fetch method, to 1.

[ikvk]

time results:

from imap_tools import A, N
import datetime
import time

#  'OUTLOOK', 'ZIMBRA', 'MAIL_RU', 'YAHOO', 'YANDEX',
with get_test_mailbox('xxx') as mailbox:
    time_set = []
    for i in range(10):
        print('=', i)
        t = time.time()
        for msg in mailbox.fetch(bulk=0):
            print(msg.uid)
        time_set.append(time.time() - t)
    print(round(sum(time_set) / len(time_set), 2), time_set)

====
ZIMBRA old
    bulk  0.07  [0.08277750015258789, 0.07779216766357422, 0.08960175514221191, 0.07352757453918457, 0.0724496841430664, 0.07131838798522949, 0.08474612236022949, 0.07160234451293945, 0.07289767265319824, 0.09224295616149902]
    one   0.10  [0.14479851722717285, 0.13862943649291992, 0.08889079093933105, 0.10997748374938965, 0.09803938865661621, 0.10226082801818848, 0.10172724723815918, 0.08976054191589355, 0.10421180725097656, 0.09824419021606445]
ZIMBRA new
    bulk  0.14  [0.17185544967651367, 0.11617159843444824, 0.15271234512329102, 0.16122746467590332, 0.17380261421203613, 0.1385810375213623, 0.11998319625854492, 0.1419386863708496, 0.13717412948608398, 0.16089200973510742]
    one   0.15  [0.265552282333374, 0.131913423538208, 0.17278242111206055, 0.1416783332824707, 0.13575100898742676, 0.15034079551696777, 0.1308438777923584, 0.13962554931640625, 0.14082741737365723, 0.14168429374694824]

====
OUTLOOK old
    bulk  0.73  [1.3324766159057617, 0.7556638717651367, 0.6871931552886963, 0.6609692573547363, 0.6600816249847412, 0.6867294311523438, 0.6469390392303467, 0.6782865524291992, 0.6283712387084961, 0.6594040393829346]
    one   2.72  [3.3163089752197266, 2.588245391845703, 3.022836923599243, 2.5231645107269287, 2.5305676460266113, 2.5704076290130615, 2.655641794204712, 2.708867311477661, 2.6972944736480713, 2.673250436782837]
OUTLOOK new
    bulk  0.94  [1.075840711593628, 0.9262182712554932, 1.2316997051239014, 0.939857006072998, 0.905651330947876, 0.9089338779449463, 0.892751932144165, 0.8556170463562012, 0.8526890277862549, 0.8690805435180664]
    one   2.58  [3.2186834812164307, 2.5547666549682617, 2.4564359188079834, 2.58793044090271, 2.4789702892303467, 2.6506192684173584, 2.527388334274292, 2.484177827835083, 2.4634761810302734, 2.4538345336914062]

====
MAIL_RU old
    bulk  1.21  [1.8444347381591797, 1.0986638069152832, 1.7128126621246338, 0.9876015186309814, 1.705270767211914, 0.8997001647949219, 1.021353006362915, 0.8954658508300781, 1.0315186977386475, 0.9854748249053955]
    one   3.04  [2.5624523162841797, 3.688250780105591, 2.640326738357544, 3.7957658767700195, 2.673093557357788, 3.143564224243164, 2.849813222885132, 2.8817081451416016, 2.989767074584961, 3.2243306636810303]
MAIL_RU new
    bulk  1.24  [1.234104871749878, 1.215832233428955, 1.3136053085327148, 1.4261114597320557, 1.0973701477050781, 2.0126421451568604, 0.7960953712463379, 0.9327006340026855, 0.8965253829956055, 1.4738049507141113]
    one   3.64  [3.56408429145813, 3.852703332901001, 3.676325798034668, 3.8412628173828125, 3.524775505065918, 3.6137189865112305, 3.505016803741455, 3.5479702949523926, 3.8054487705230713, 3.5024352073669434]

====
YAHOO old
    bulk  1.19  [1.5369861125946045, 1.1241528987884521, 1.30910062789917, 1.0905780792236328, 1.112328290939331, 1.1532926559448242, 1.2369194030761719, 1.087550401687622, 1.1742920875549316, 1.1048543453216553]
    one   3.9   [3.7564921379089355, 3.724494218826294, 4.348383903503418, 3.8233118057250977, 4.414995431900024, 4.0000550746917725, 3.853823184967041, 3.75169038772583, 3.746474504470825, 3.5963141918182373]
YAHOO new
    bulk  1.55  [2.0416958332061768, 1.5042519569396973, 1.5103850364685059, 1.4582610130310059, 1.4732036590576172, 1.5990757942199707, 1.4518721103668213, 1.452500343322754, 1.5318374633789062, 1.4549624919891357]
    one   4.17  [4.310445547103882, 4.071575403213501, 4.031844854354858, 3.9776623249053955, 4.134377956390381, 4.157119512557983, 4.357776403427124, 3.9568369388580322, 4.412524938583374, 4.279238700866699]

====
YANDEX old
    bulk  0.9  [1.4435932636260986, 0.8275289535522461, 0.8417303562164307, 0.8620786666870117, 0.8423538208007812, 0.8458282947540283, 0.8372230529785156, 0.8444399833679199, 0.8451566696166992, 0.8469619750976562]
    one   3.88 [4.810968399047852, 4.374621152877808, 4.274755477905273, 3.695033073425293, 3.2208902835845947, 3.325171947479248, 3.3337655067443848, 3.3692147731781006, 4.0010669231414795, 4.4181671142578125]   
YANDEX new
    bulk  1.13 [1.9992485046386719, 0.9898438453674316, 0.9686620235443115, 0.9470534324645996, 0.9498946666717529, 1.489837408065796, 1.0160863399505615, 1.0047743320465088, 1.0090336799621582, 0.9598896503448486] 
    one   3.55 [5.225865840911865, 3.1782851219177246, 3.2470858097076416, 3.1306257247924805, 3.7210419178009033, 3.411531686782837, 3.234295129776001, 3.2834551334381104, 3.4541571140289307, 3.5949721336364746] 

[ikvk]

@dimitrisstr please check it at your task

https://github.com/ikvk/imap_tools/pull/203

[ikvk]

Added