Jumoo / uSync.Complete.Issues

Public Issue tracker and roadmap for uSync.Complete
https://jumoo.co.uk/usync/complete/
2 stars 1 forks source link

Large transfers via Publisher #50

Closed benjaminc closed 3 years ago

benjaminc commented 4 years ago

Transferring large amounts of content via Publisher can cause timeouts on either the source or destination, or can exceed the maximum file sizes allowed. Additionally, if the network drops part of the way through, you can lose everything and have to restart. There are a few things that would need to be done to circumvent this.

First, make publishing a background process, with the UI simply checking on the status of the process. This would eliminate the request timeouts between the client and source system.

Second, once the publish package is constructed, break it into smaller pieces, the specific size should have a reasonable default of say 10 MB, but should be configurable on a per-publish-destination basis. Transfer the package in pieces, and include code to try a transfer multiple times, with a reasonable delay between, in case the connection is going up and down.

This could reuse the same code which I'm proposing for Exporter in a separate feature request. The publisher process could create an actual .usync file, then use the exporter's enhanced upload method to push it up, and finally trigger the import of the uploaded file.

KevinJump commented 4 years ago

Hi there is config that gets added to web.config to push the upload limits for publisher. see : https://docs.jumoo.co.uk/uSync/uSync.Complete/publisher/umbraco/

I would consider a 'large' publish to be more than 250 pages (this is conservative, we test with 1000's but acknowledge that's a controlled environment on fast hardware/fast disks). not sure what your experience is. it depends a lot on what settings you are using (and something we are looking to streamline).

In theory with everything else synced you don't need dependency checks, and parent checks etc, and that does make the syncs work a lot faster.

but i agree with all of the above, we will look into it.