search

fmbiete / Z-Push-contrib

Z-Push fork with changes that I will try to contrib
GNU Affero General Public License v3.0
135 stars 62 forks source link

Sync of mailboxes with high counts of emails #118

Open fpPierre opened 9 years ago

fpPierre commented 9 years ago

Hello,

I do have the issue, that our mailboxes with enormous number of mails stops the initial sync after 10 to 20 percent of the mails ins box.

We do have mailboxes with folders which stores up to 10000 mails and 100000 mails in sum.

Has somebody an idea, how we can handle that with active sync?

fmbiete commented 9 years ago

Hi @fpPierre ,

I think that's a known problem with Z-Push. You will run into problems with huge mailboxes. Try setting this config option to true, maybe will help. But it's a very experimental code.

 // Users with many folders can use the 'partial foldersync' feature, where the server
// actively stops processing the folder list if it takes too long. Other requests are
// then redirected to the FolderSync to synchronize the remaining items.
// Device compatibility for this procedure is not fully understood.
// NOTE: THIS IS AN EXPERIMENTAL FEATURE WHICH COULD PREVENT YOUR MOBILES FROM SYNCHRONIZING.
define('USE_PARTIAL_FOLDERSYNC', false);
extremeshok commented 9 years ago

Any idea what defines a "huge" folder ? My thoughts one should be able to get a count of folders and or messages and then USE_PARTIAL_FOLDERSYNC should be automatically set. This would prevent issues with smaller mailboxes and prevent issues or mitigate them with larger mailboxes.

Ie. If folder_count > x or message_count > y { set USE_PARTIAL_FOLDERSYNC = true }

Sent from my iPhone

On 20 Dec 2014, at 8:31 PM, Francisco Miguel Biete notifications@github.com wrote:

Hi @fpPierre ,

I think that's a known problem with Z-Push. You will run into problems with huge mailboxes. Try setting this config option to true, maybe will help. But it's a very experimental code.

// Users with many folders can use the 'partial foldersync' feature, where the server // actively stops processing the folder list if it takes too long. Other requests are // then redirected to the FolderSync to synchronize the remaining items. // Device compatibility for this procedure is not fully understood. // NOTE: THIS IS AN EXPERIMENTAL FEATURE WHICH COULD PREVENT YOUR MOBILES FROM SYNCHRONIZING. define('', false); — Reply to this email directly or view it on GitHub.

fpPierre commented 9 years ago

I have activated it at our system. I will report the results.

Thanks both of you a lot.

fmbiete commented 9 years ago

Hi @extremeshok

If I remember correctly z-push core developers comments "huge" was a tricky definition. If a FolderSync ops takes longer that 15-30 seconds, it will fail (depends of the mobile device); so this should be a huge mailbox. Or if "backend list messages" ops for a folder take longer that 15-30 seconds for a folder.

In short it's not a matter of number of folders or messages, but how fast your IMAP/Zarafa/whatever will return the data for the list operations.

Regards,

extremeshok commented 9 years ago

Then it would make sense to auto enable it if the mailbox timeout for the specific server. Then the subsequent queries would be successful, even if the initial fails.

I'm only thinking of making zpush more robust and error free

Sent from my iPhone

On 21 Dec 2014, at 10:13 AM, Francisco Miguel Biete notifications@github.com wrote:

Hi @extremeshok

If I remember correctly z-push core developers comments "huge" was a tricky definition. If a FolderSync ops takes longer that 15-30 seconds, it will fail (depends of the mobile device); so this should be a huge mailbox. Or if "backend list messages" ops for a folder take longer that 15-30 seconds for a folder.

In short it's not a matter of number of folders or messages, but how fast your IMAP/Zarafa/whatever will return the data for the list operations.

Regards,

— Reply to this email directly or view it on GitHub.

fmbiete commented 9 years ago

Hi @extremeshok , Yeah, I also hope that it will become automatically enabled. But that's also pretty hard; we will need to use the user datastore or shared mem to record the tries.

extremeshok commented 9 years ago

store a json encoded flat file, or create a cache dir and create flatfiles per each account, which could hold the count.

Its rather simple, we did something similar with another project. One could even use sqlite as a datastore.

Unfortunately I don't have the time to push a patch

On 12/22/2014 12:21 PM, Francisco Miguel Biete wrote:

Hi @extremeshok https://github.com/extremeshok , Yeah, I also hope that it will become automatically enabled. But that's also pretty hard; we will need to use the user datastore or shared mem to record the tries.

— Reply to this email directly or view it on GitHub https://github.com/fmbiete/Z-Push-contrib/issues/118#issuecomment-67821883.