Prioritized pages via rsync?

[coneybeare]

I have a problem which has only cropped up whilst on vacation with a very slow internet connection.

While syncing my site, the root index page is done first, while the /**/index.html files are done later, presumably in alphabetic order. This presents a problem when syncing takes 10 minutes, as the index pages point to 404 articles rather than the article page that syncs much later.

I want to be able to sync the leafs first, then the index pages and rss feeds etc...

My proposed solution would be to use rsync's "--files-from" option and have the file built automatically based on file tree depth, with the larger depth first. There is no current option I can pass to octopress deploy to accomplish this.

Thoughts?

[magikid]

I think this should be the default. It should be syncing from the bottom of the (directory) tree to the top to minimize 404s.

I haven't looked yet but does this also affect s3 syncing?

[imathis]

@coneybeare Thanks for sharing this with me. I've never written an Rsync script that does what you suggest, so I'm not sure how that would fit in with Octopress deploy. It sounds like you're saying you'd like for me to add support for the --files-from option and generate custom arguments for it. I'm happy to add --files-from but as much as possible, I'd like to avoid being clever on top of syncing technologies.

However, I think this is really a more sophisticated problem than what order to sync files. If your deployments take a long time over rsync, maybe it's better to rsync to a temporary directory, for example site_sync, and then run some commands over ssh after the sync completes like this:

mv live_site dead_site && mv site_sync live_site && mv dead_site site_sync

This way your site stability is reduced to the speed of directory renaming instead of your connection speed. Maybe you could create Rake task to accomplish this, or maybe it's something I can find a way to add to Octopress deploy. I'll need to think about it some more.

@magikid I'm not sure how s3 could manage something like this. Again, I don't want to get too clever, but if you have any suggestions for how to make this work better for s3, I'd love to hear them. The s3 support is built on the AWS SDK so any methods supported by that are available to me.

[coneybeare]

I like your deploy to a temp directory and mv idea, that sounds like a feasible solution that will take minimal effort to implement on a server I control. Thanks

 

 

On Saturday, May 9, 2015 at 11:12 AM, Brandon Mathis notifications@github.com, wrote:

@coneybeare Thanks for sharing this with me. I've never written an Rsync script that does what you suggest, so I'm not sure how that would fit in with Octopress deploy. It sounds like you're saying you'd like for me to add support for the --files-from option and generate custom arguments for it. I'm happy to add --files-from but as much as possible, I'd like to avoid being clever on top of syncing technologies.

However, I think this is really a more sophisticated problem than what order to sync files. If your deployments take a long time over rsync, maybe it's better to rsync to a temporary directory, for example site_sync, and then run some commands over ssh after the sync completes like this:

mv live_site dead_site && mv site_sync live_site && mv dead_site site_sync

This way your site stability is reduced to the speed of directory renaming instead of your connection speed. Maybe you could create Rake task to accomplish this, or maybe it's something I can find a way to add to Octopress deploy. I'll need to think about it some more.

@magikid I'm not sure how s3 could manage something like this. Again, I don't want to get too clever, but if you have any suggestions for how to make this work better for s3, I'd love to hear them. The s3 support is built on the AWS SDK so any methods supported by that are available to me.

— Reply to this email directly or view it on GitHub.

[imathis]

Cool, I'm going to close this for now. If someone wants to suggest a good way to improve s3 uploads. We'll track that in another issue.