Improve tar-Cache

[GoogleCodeExporter]

Currently we store the milestone tar.bz2 files in a cache. This saves a lot
of downloading time (and of course traffic).

If the slave has enough hd space it should be possible to store the
uncompressed .tar-files instead of .tar.bz2-files. This can save time and
memory at the cws_get step because the slave don't need to decompress data.

Original issue reported on code.google.com by christia...@web.de on 8 Aug 2008 at 1:31

[GoogleCodeExporter]

I have made a little test: 
only tar-ing the file without compression is over ten times faster than with bz2
compression.

Original comment by christia...@web.de on 18 Aug 2008 at 2:24

Attachments:

• test.sh

[GoogleCodeExporter]

On the other hand, the benefit of compressed tarballs is that the total disk 
usage is
lower.  Keep in mind that: 
  * compression is MUCH more expensive than decompression
  * disk I/O for uncompressed tarballs is 2x tar size (once for reading and once for
writing). 

It *may* have a performance benefit to have compressed tarballs.  However, the
maximum benefit varies with CPU speed and disk speed.  Of course, testing this 
is not
as simple as it may sound.  Things like buffer caches play a role. 

Original comment by ikel...@gmail.com on 20 Aug 2008 at 12:54

[GoogleCodeExporter]

Gregor came up with another idea of improving the tarball cache for slaves.

In his (and my) opinion it's not efficient that every single slave checkouts or
downloads the code to its very local cache. We should create a storage server 
that
contains the ready-to-build tarballs for all slaves. The only think we must 
ensure is
the upload concurrency, but I think we can handle that somehow.

Additionally the centralized cache simplifies the removal of broken masters.

Original comment by christia...@web.de on 20 Aug 2008 at 11:47

[GoogleCodeExporter]

We must reconsider the tar-cache concept due to the vcs switch to Subversion. 
Perhaps
an "svn switch" or "svn update" is faster and more efficient than downloading 
and
caching tarballs.

Original comment by christia...@web.de on 21 Aug 2008 at 2:50

[GoogleCodeExporter]

I have locally tested a "svn switch" from a cws to the corresponding master. 
This 
was surprisingly fast (it takes only a few minutes crunching the working copy 
on my 
machine, downloading the few different files was nothing).
So I will deferre this issue as a tarcache seems a lot slower than svn switch, 
which 
is a pleasing result :-)

Original comment by christia...@web.de on 23 Sep 2008 at 11:22

• Changed state: Deferred