DanEngelbrecht / golongtail

Command line front end for longtail synchronization tool
MIT License
26 stars 8 forks source link

Using self-containing archive or not #233

Closed mikeseese closed 1 year ago

mikeseese commented 1 year ago

This issue is more of a question. I'm trying to sync precompiled Unreal Engine editors, and I'm curious what your experience has been related to using a self-containing archive. I saw that golongtail has an option for packing while storing, but when does it make sense to use it vs not? My concern of using it is possibility of not being able to provide diffs better, but not using it may cause download times to be longer due to increased overhead by having multiple files. What is your recommendation, especially related to precompiled Unreal Engine editor packages?

mikeseese commented 1 year ago

Well I attempted using a normal put (without pack) and the subsequent get mostly maxed out my download bandwidth (1gbps), so perhaps packing won't make much of a difference in terms of network throughput for my use case, just minimizing the amount stored/downloaded at the cost of CPU time?

DanEngelbrecht commented 1 year ago

Yeah, longtail is pretty good at maxing out your bandwith. Having a store makes is good to reduce the overall size of multiple versions but adds a bit of complexity. Using and standalone archive is simpler but does not provide delta in the store. Applying a change (downsync/unpack) is still done with delta but for the archive case you need to download the full archive not saving any bandwidth.

So it depends on you use case…

mikeseese commented 1 year ago

Ok thanks for the clarification! That helps