Copy cannot be completed

[heisenberg2980]

I am trying to copy the emails between two different google accounts, and to reduce the risk of failures I am copying the emails from the original account to the local folder in TB, and then copying them from the local folder to the new account. The first step completes without any issue so all the emails are copied to the local folder, but the second step keeps failing after certain number of emails have been copied, no matter how many times I disable/enable the add-on and restart the process. Is there any log or report to know why the copy is failing?

[smitgd]

There might be something logged in the TB "Error Console", ctrl-shift-j. How many messages are in Local Folders when you do the copy? I currently have a local folder with 17247 messages that I could test this with but my gmail accounts are getting kind of full. Is that more or less than what you are copying?

[heisenberg2980]

The account I am trying to copy has less than that, the total number is 4620 and it says it has already copied 3305 (so 1315 messages pending):

This time it copied 375 out of the 1315:

But other times it says it has copied some messages, but then when I restart the process the exact same amount is pending.

Some errors like this appear in the error console while the copy is happening:

P.S.: I am planning to copy two more accounts that contain a higher number of messages, but I started with the smallest account to ensure it worked.

[heisenberg2980]

@smitgd after couple of iterations yesterday the process was stuck in 942 messages (it kept saying it has copied some messages, but after starting the copy again there were still 942 messages to copy):

I have tried again today and it says it has copied 485 messages with 457 failed (for some reason waiting a day makes a difference in the process, not sure why):

Now when I start the process again it says there are still 622 to copy, instead of 457:

[smitgd]

As an test I created a local folder with 4000 messages and copied it to gmail. What I saw was 15 messages copied OK and then it hung. After letting it timeout I got a pop-up message that the copy failed. This took quite a while to appear due to retries I assume. I'm not sure what caused the "hang". Maybe gmail refused the connection. Anyhow, on restart it copied the remaining 3985 messages successfully and now the source and destination agree on number of messages. I see similar errors in the error console as you report. I'll have to look closer to see what they mean. If you have problems and it's not working, you might consider restarting tb to reset the addon. I don't have a complete understanding of the addon since I just adapted an old one written a while back. Another thing is is you keep your source imap folder stored locally (called for "offline" use) there is probably no need to copy it to local folders first since the messages will be fetched from a the locally stored mbox file and not first from the source server, which is almost equivalent to "local folder" storage. Anyhow, I'll test it again and see if I can find something more specific and try again to duplicate what you see.

[heisenberg2980]

After many retries (around 10-12) I was able to copy the whole account. Now I have started with the second account with 10k messages and the first round it just copied 100 messages, but the second one copied 6k which isn´t that bad. I will keep retrying and hopefully it will copy all the messages, but it is pretty painful (as you mentioned, every time the process fails it takes a really long time before it times out), so it would be amazing if you can find the source of the problem.

In case you find it, just let me know if you need me to test any other version of the add-on.

[smitgd]

Thanks for the response. It looks like the problem is not on the gmail end but on the tb end. For my test, I can always fetch from local folders the 1st 15 messages and then the 16th can't be read from tb for yet unknown reasons. Then I start seeing the same errors as you report above: "no transfer callback..." until it times out. Then I start the copy again (without restarting tb) and it continues on and right now it's up to about 4300 messages copied to gmail. If it hangs I don't think you have to wait for the timeout. Just shutdown tb and start again (which only takes a few seconds) -- but be sure to close the error console and any other tb windows that may be open to get a full restart. Anyhow, I'll look closer and try to find out why the addon is having a problems sometimes getting the messages to send from tb. I was thinking it may have something to do with duplicate messages but that didn't seem to be the case...

[heisenberg2980]

After many attempts (maybe 30-40) I was able to copy my second account with 10k messages (there are 17 messages left that don´t want to be copied after 5-6 attempts, but I am not bothered), and now I will start with my third account with over 43k messages (wish me luck...). In the first try the process copied exactly 15 messages as you mentioned and then failed, so it seems there is something going on with those 15 first messages.

[smitgd]

I haven't yet found why it stops at 15 yet, but I'll try some more. Thanks for verifying that. Were the 17 that wouldn't copy possibly duplicates of messages that were already copied? I looked closer at how the addon determines duplicates and it basically saves a hash of the important header fields and compares them. So I ended up with 11 fewer messages at the destination than was in the source with my test but they really were duplicates. (Again, to be clear, I didn't write this addon from scratch but just adapted an old one that seemed to be abandoned to newer TB.)

[smitgd]

I now see why only 15 get copied on the first try. It's because the destination folder is empty (has no messages) so the attempt to get the "nextMsg()" was causing the "uncaught" exception we see in the log. The 15 is the maximum number of messages read from the source folder at a time (called "inflight") and then sent one at a time to the destination. Once there are some messages in the destination folder and you re-do the transfer, the 'enumerator" doesn't have a problem. I'm not sure why I never noticed this before. I've definitely copied messages into an empty folder before. Anyhow, I pushed the fix to the WL branch so you will need to pull or clone that branch to get the fix if you still want to try it. Thanks!

[smitgd]

I should also say that I don't think this will fix your need to re-do the transfer multiple times. I'm not sure what is causing that. From my location, I'm able to copy about 5000 messages to gmail without a re-do (except for the initial 15 message bug that should now be fixed).

[heisenberg2980]

Sure, I will try the new version, I know it is unlikely but maybe we are lucky and it also fix my other problem...

On a side note, to install this add-on I just downloaded the zip from github, unzip it and the zip the files inside the folder, and then use that zip file to "Install add-on from file" in TB, is that the way to do it? Just wondering if I am doing something wrong that could be also causing the issue (I don´t think so, as TB recognises and installs the add-on, but just in case)

[heisenberg2980]

After installing the new version I have restarted the process and I can see immediately around 34 of these errors in the log:

Couple of minutes later the process hasn´t failed yet and it is still copying (just around 500 messages copied though).

Do you see any of these errors in your log when you successfully copy 5k messages?

Also note the account I am trying to copy now is hotmail (but I don´t think that matters as one of the previous accounts were gmail and the other one was hotmail, and I was having the same issues)

[heisenberg2980]

I am not sure if it is related to the latest fix, but the first round with the new version has not been that bad, the process has failed after 7.8k messages copied, so around 36k to go, starting the second round now...

[smitgd]

Sure, I will try the new version, I know it is unlikely but maybe we are lucky and it also fix my other problem...

On a side note, to install this add-on I just downloaded the zip from github, unzip it and the zip the files inside the folder, and then use that zip file to "Install add-on from file" in TB, is that the way to do it? Just wondering if I am doing something wrong that could be also causing the issue (I don´t think so, as TB recognises and installs the add-on, but just in case)

I think you are referring to the "release" link: https://github.com/smitgd/addon-copyfolder-tb-esr68/releases/tag/v2.2.2

That doesn't currently now contain my fix for the 15 message bug. Right now you would have to do a git clone or git pull of the whole project to get the fix and switch to branch WL. I would have to make a new release for it to appear in the "release" link. I'll test a bit more and make a new release.

What you did to install it is OK. However, the release link contains the .xpi file for the project so you only need to install it, so you don't need to zip or unzip anything if you just download the .xpi. (The xpi is really just a zip of the project.)

[heisenberg2980]

Actually no, I was referring to the clone option where you can download the WL branch in a zip file, but I understand what you mean, once you create a new release with this branch the unzip-zip steps won´t be required anymore (not a big deal once you know how to do it, but maybe not so easy for other people not familiar with this).

Regarding the retries, they haven´t gone that well, the second and third retry failed after 150-200 messages (I think copying this account is going to be painful, but I will keep trying)

[smitgd]

Ok, I'm just a noob on on github features so I didn't even know you can just download a zip! I always just clone it using a command line "git clone".

After installing the new version I have restarted the process and I can see immediately around 34 of these errors in the log: ...

I see the same error but just one time per copy attempt before the copy begins. I spent some time trying to debug this today but never was able to determine anything. It looks like this happens before the copy starts and the addon tries to update and bring in any possible new mail to the source folder. My guess is that this error occurs because the source folder is Local Folders and nothing needs to be updated so the error is innocuous. I haven't yet tried it with an imap folder as the source.

What I don't understand is why you see it 34 times. Maybe you have restarted the copy 34 times without restarting tb?

[smitgd]

Regarding the retries, they haven´t gone that well, the second and third retry failed after 150-200 messages (I think copying this account is going to be painful, but I will keep trying)

I guess this is phase 2 of the problem. I wonder what's causing it to stop? I've been able to copy 5000 messages to gmail several times without a hang.

[heisenberg2980]

I have the feeling that my TB gets busy downloading and indexing messages that are pending to be sync, and that causes the copy to fail. I am not sure how to avoid that though, because every time I close and open TB there seems to be more messages to download and sync.

[smitgd]

Maybe I'm missing something but I thought you were copying from a "Local Folder". New messages shouldn't be appearing there on tb restart unless you are moving messages there manually or via a filter. Or maybe you are referring to the destination gmail folder receiving new messages? Also, you can turn off indexing temporarily near the bottom of "General" settings. FYI, I made one more commit to the WL branch and commented out the "gds: xfer good" log messages. It was causing the log to have a lot of "Quirks Mode" log message appear, whatever that means.

[smitgd]

I am not sure how to avoid that though, because every time I close and open TB there seems to be more messages to download and sync.

Maybe you need to temporarily set your tb accounts to not "check for new mail on startup", don't "check for new mail every X min" and don't "immediately get new mail" while doing the bulk copies. This is under the server setting for each imap account.

[heisenberg2980]

I am not sure how to avoid that though, because every time I close and open TB there seems to be more messages to download and sync.

Maybe you need to temporarily set your tb accounts to not "check for new mail on startup", don't "check for new mail every X min" and don't "immediately get new mail" while doing the bulk copies. This is under the server setting for each imap account.

Thanks for the advise, but I tried to untick the 3 options and restart TB, and for some reason TB is still downloading messages (maybe a bug?), so I guess I will have to wait until the messages of all folders have been downloaded to be able to have a clean copy.

[smitgd]

Thanks for the advise, but I tried to untick the 3 options and restart TB, and for some reason TB is still downloading messages (maybe a bug?), so I guess I will have to wait until the messages of all folders have been downloaded to be able to have a clean copy.

I forgot to mention that if you click on the folder, even with the 3 items unselected, tb will try to update the folder and download whatever the server has. So you have to not select the destination gmail folder to keep downloads from happening.

[heisenberg2980]

Finally!! After applying all the changes you mentioned to avoid TB to sync my destination account, and using your lastest version, I left the addon running overnight and it copied the last 24k messages successfully, hurrah!!

Something a bit weird is that even with all the options you mentioned unselected, this morning TB was still downloading messages from my destination account, but well, I don´t really care as long as the copy was able to complete without crashing.

Btw, you may want to include those tips in github so other people don´t have the same issue.

[smitgd]

Something a bit weird is that even with all the options you mentioned unselected, this morning TB was still downloading messages from my destination account,

The way it works is the addon copies (technically, imap appends) the source messages one at a time to the destination folder. This just stores them at the destination server (assuming it's an imap folder). Then only after you select (click on) the destination folder will TB download the headers and full messages to local storage (e.g., .msf (for headers) and mbox or maildir files).

Anyhow, glad to hear it finally worked OK. I'll update the readme to include these troubleshooting suggestions.