search

frerich / clcache

A compiler cache for MSVC, much like ccache for gcc
Other
324 stars 83 forks source link

Hash server prevents deletion of directories #264

Closed TiloW closed 7 years ago

TiloW commented 7 years ago

As already pointed out by @frerich here, running the hash server prevents deletion of certain directories. Especially for the build directory itself this is a major problem.

While reading through #248 I realized this seems to be working as intended, i.e., one should start the server before the build and shut it down afterwards. This should probably mentioned in the readme and/or caveats section of the wiki. Does launching multiple cache servers in parallel work?

frerich commented 7 years ago

Why can't we ommit watching directories and simply use the modification time of the files instead?

I think this would not be a good approach for two reasons:

  1. Getting file meta information (e.g. the modification time or the file size) is notoriously inefficient on Windows, at least in comparison to how stat works on Linux. Maybe SSDs changed this to the better, but last time I tried it, getting file meta information was a fairly costly operation. For what it's worth, Git itself suffered from this on Windows, too (e.g. when deciding whether the filesystem is dirty) and a commit which addressed this explains that the Windows APIs are apparently much more tailored to get meta information for an entire directory of files than just individual files.

  2. Checking the modification time of the file means paying for the risk of a file change on every single cache access, i.e. it's a pessimistic invalidation scheme. Since files typically do not change though (it's rather the exception than the rule), this would mean doing a lot of work even though nothing changed.

Is there a way to watch files while allowing them to be removed?

I seem to recall that it's possible to watch directories further up the hierarchy (i.e. in the most extreme case the root directory) and then get recursive file modification notifications. This however means that you

  1. Have to manually figure out whether the modified file is of interest to us.
  2. Get a lot of notifications for unrelated reasons.

I personally like the blacklisting approach in which clcachesrv would get told a set of directories which shouldn't be watched.

frerich commented 7 years ago

I'm closing this for now since I don't see a good way to fix this problem; the clcachesrv program was augmented to support a new --exclude switch which can be used to exclude certain directories from being considered, the idea being that you typically want to exclude directories which will be removed at some point. I also documented this caveat for future reference.