Fetch operation progress not being called

[davetroy]

As a test, I'm doing the following:

        let fetchOp = session.fetchMessagesByUIDOperationWithFolder(folder, requestKind: requestFlags, uids: uids)

        fetchOp.progress = { (current: UInt32) in
            println("\(current)")
        }

        fetchOp.start { (error: NSError!, fetchedMessages: [AnyObject]!, vanished: MCOIndexSet!) -> Void in
etc...

The progress block is not being called, even though the fetchOp is running successfully and returning hundreds of messages. What could be the trouble? Swift issue?

[dinhvh]

Did you try with Obj-C?

[davetroy]

Not yet; Swift is definitely my preference for this project, and I haven't noticed any other issues with block bridging like this. I'll see if I can give that a try easily. But you think the setup above looks ok?

FWIW calling the block directly produces output:

fetchOp.progress(10)

[dinhvh]

It looks ok. I was just looking for an answer whether the same worked in Obj-C.
I'm not asking you to switch to Obj-C :) The setup looks good.

Hoa V. Dinh

On Saturday, August 30, 2014 at 6:37 AM, Dave Troy wrote:

Not yet; Swift is definitely my preference for this project, and I haven't noticed any other issues with block bridging like this. I'll see if I can give that a try easily. But you think the setup above looks ok?

— Reply to this email directly or view it on GitHub (https://github.com/MailCore/mailcore2/issues/851#issuecomment-53958634).

[davetroy]

I just wrote a version of essentially the same call in Obj-C and am getting the same result: 15 messages retrieved, but progress block is not called at all.

    MCOIMAPMessagesRequestKind flags = MCOIMAPMessagesRequestKindFullHeaders | MCOIMAPMessagesRequestKindExtraHeaders;

    MCOIMAPFetchMessagesOperation *fetchOp = [session fetchMessagesByUIDOperationWithFolder:@"[Gmail]/All Mail"
                                                                                requestKind:flags
                                                                                       uids:uids];
    fetchOp.extraHeaders = @[@"List-Id"];

    fetchOp.progress = ^(unsigned int current) {
        NSLog(@"%u", current);
    };

    [fetchOp start:^(NSError *error, NSArray *messages, MCOIndexSet *vanishedMessages) {
        NSLog(@"performed fetch, got %d messages", messages.count);
    }];

Any help appreciated!

[dinhvh]

Fixed with 7cd27c2. Could you confirm? Thanks!

[davetroy]

I tried this with both the Obj-C test harness and the Swift code and neither one seemed to work. I looked at the change and it seems quite subtle; also confirmed that I did have that commit in the code in my project (just to be sure) and it's just not working.

It's possible I'm doing something stupid, but I've checked the obvious candidates.

[dinhvh]

Could you add logs in MCOIMAPBaseOperation.mm, in

    virtual void itemProgress(mailcore::IMAPOperation * session, unsigned int current, unsigned int maximum) 

and in MCOIMAPFetchMessagesOperation.mm, in

- (void) itemProgress:(unsigned int)current maximum:(unsigned int)maximum

to see if those are called properly?

[davetroy]

I added logs to both of those itemProgress methods and neither one seems to have been called, even though the fetch completed successfully, returning 667 messages.

- (void) itemProgress:(unsigned int)current maximum:(unsigned int)maximum
{
    NSLog(@"base operation: %d", current);
}

and

- (void) itemProgress:(unsigned int)current maximum:(unsigned int)maximum
{
    NSLog(@"itemProgress: %d", current);
    if (_progress != NULL) {
        NSLog(@"itemProgress, _progress is not null: %d", current);
        _progress(current);
    }
}

None of these NSLogs produced any output. :( Other thoughts? Happy to try stuff.

[dinhvh]

Can you add a log here?

    virtual void itemProgress(mailcore::IMAPOperation * session, unsigned int current, unsigned int maximum) {
        [mOperation itemProgress:current maximum:maximum];
    }

[davetroy]

Oops, my mistake, put it in the wrong method in that file. Just added this:

    virtual void itemProgress(mailcore::IMAPOperation * session, unsigned int current, unsigned int maximum) {
        NSLog(@"base operation: %d", current);
        [mOperation itemProgress:current maximum:maximum];
    }

Still getting no output, with 673 messages reported retrieved.

[davetroy]

Anything you'd like for me to try on this? I'm trying to button up a project that relies on this and this is the main thing holding me up right now. Not sure what the issue could be.

[dinhvh]

There's a callback that is set on the low-level Libetpan object.
It will call the c++ callback. Could you check if that progress callback is at least called?

The C++ callback works for me for one of my projects.

Hoa V. Dinh

On Wednesday, September 3, 2014 at 6:08 AM, Dave Troy wrote:

Anything you'd like for me to try on this? I'm trying to button up a project that relies on this and this is the main thing holding me up right now. Not sure what the issue could be.

— Reply to this email directly or view it on GitHub (https://github.com/MailCore/mailcore2/issues/851#issuecomment-54293656).

[davetroy]

Thanks. I looked around to try to find that function in the libetpan library and didn't see it in a first pass. Do you know what file I should look in?

[dinhvh]

Search the caller to the C++ callback in mailcore2 code base.
I don't have access to the code right now.

Hoa V. Dinh

On Wednesday, September 3, 2014 at 10:14 AM, Dave Troy wrote:

Thanks. I looked around to try to find that function in the libetpan library and didn't see it in a first pass. Do you know what file I should look in?

— Reply to this email directly or view it on GitHub (https://github.com/MailCore/mailcore2/issues/851#issuecomment-54331619).

[davetroy]

Is this the place you mean? I added this printf below. Still not getting called.

void IMAPOperation::itemsProgressOnMainThread(void * ctx)
{
    if (isCancelled()) {
        release();
        return;
    }

    struct progressContext * context = (struct progressContext *) ctx;
    printf("progressOnMainThread");
    if (mImapCallback != NULL) {
        mImapCallback->itemProgress(this, context->current, context->maximum);
    }
    free(context);
    release();
}

Also did this:

void IMAPOperation::itemsProgress(IMAPSession * session, unsigned int current, unsigned int maximum)
{
    printf("progressOnBackgroundThread: %d", current);

    if (isCancelled())
        return;

    struct progressContext * context = (struct progressContext *) calloc(sizeof(* context), 1);
    context->current = current;
    context->maximum = maximum;
    retain();
    performMethodOnCallbackThread((Object::Method) &IMAPOperation::itemsProgressOnMainThread, context, true);
}

I set breakpoints inside both of those routines too. Neither is getting called.

[dinhvh]

Are you blocking the main thread while the emails are downloading?
Are you setting a dispatch queue other than the default to the IMAP session?

Hoa V. Dinh

On Wednesday, September 3, 2014 at 2:36 PM, Dave Troy wrote:

Is this the place you mean? I added this printf below. Still not getting called. void IMAPOperation::itemsProgressOnMainThread(void * ctx) { if (isCancelled()) { release(); return; } struct progressContext * context = (struct progressContext *) ctx; printf("progressOnMainThread"); if (mImapCallback != NULL) { mImapCallback->itemProgress(this, context->current, context->maximum); } free(context); release(); }

— Reply to this email directly or view it on GitHub (https://github.com/MailCore/mailcore2/issues/851#issuecomment-54371393).

[davetroy]

No on both counts. This is a really straightforward app that's just calling that fetchOp. Didn't change the dispatch queue.

[davetroy]

I think what I'll do is create a test project that attempts to demonstrate this problem and then post it. Maybe that will be helpful for both of us and reveal the problem.

[davetroy]

OK, I created a separate test project, and weirdly enough the appropriate callbacks are called, with all the NSLogs and printf's producing output (linking against the same copy of mailcore2).

Let me see if I can make the test project break by bringing it more in line with the function of the real app and then that should be revealing.

[dinhvh]

I'm closing it. Let me know if there's an issue with mailcore2.

[davetroy]

OK, I finally figured out what's going on here and it's weird but easy to trigger. If you call the fetch operation with FullHeaders vs. Headers flag set, the progress operation is not called.

// doesn't work
MCOIMAPMessagesRequestKind flags = MCOIMAPMessagesRequestKindFullHeaders | MCOIMAPMessagesRequestKindExtraHeaders;

// works
MCOIMAPMessagesRequestKind flags = MCOIMAPMessagesRequestKindHeaders | MCOIMAPMessagesRequestKindExtraHeaders;

I don't know why this is yet but I can reproduce it consistently.

[dinhvh]

There's obviously an issue. libetpan / mailimap_parser.c line 7472 would track it: it's the location where the callback is called. If you're able to recompile a custom version of libetpan with additional logging, you might be investigate more it.

[MIOSY]

Hi @dinhviethoa @davetroy @tokyovigilante @atomicbird @ajcollins , I'm copying/moving messages uid from INBOX folder to Trash folder using copyMessageOperation/moveMessageOperation and it was successful. Here I need progress of the copy/move operation, same as I can able to get progress for fetching but not for copy/move. Can someone help me to get the progress of the copy/move operation. here is my swift code below.

let localCopyMessageOperation = MCOIMAPSession.copyMessagesOperation(withFolder: "INBOX", uids: indexSet, destFolder: account.trashFolderPath)
        DispatchQueue.main.async { [weak self] in
            localCopyMessageOperation?.progress = {  //here getting error as copymessageoperation doesn't have a variable progress
                progress in
                var progressPercentage = Float(progress) / Float(self.totalDeletedMailscount)
                print("Progress percentage is \(progressPercentage)")
                self.pendingRing!.setProgress(CGFloat(progressPercentage), animated: false) //here deleted progress displayed in progress ring(PIE chart).
            }
        }

[schmidt9]

OK, I finally figured out what's going on here and it's weird but easy to trigger. If you call the fetch operation with FullHeaders vs. Headers flag set, the progress operation is not called.

// doesn't work
MCOIMAPMessagesRequestKind flags = MCOIMAPMessagesRequestKindFullHeaders | MCOIMAPMessagesRequestKindExtraHeaders;

// works
MCOIMAPMessagesRequestKind flags = MCOIMAPMessagesRequestKindHeaders | MCOIMAPMessagesRequestKindExtraHeaders;

I don't know why this is yet but I can reproduce it consistently.

Confirmed, still existing in v0.6.4 in Swift