dontchmod preference is conceptually broken

[gdt]

There are two problems with the dontchmod preference:

• It applies to both replicas, when "chmod doesn't work" is a per-replica property
• It is a detail about the current implementation detail, rather than a logical property

Instead:

• We should have chmodbrokenroot1 and chmodbrokenroot2 properties instead, which either avoid chmod calls in the corresponding replica, or print a warning on the first failure and then avoid them.
• It should be clear in the docs that if one sets chmodbrokenroot1 then one does not get to complain about how permissions end up on root1.
• A particular replica not supporting chmod should be separate from the concept of syncing permissions vs not. The point is that one can sync permissions but in the case of chmodbrokenrootN not actually succeed in setting them, vs not syncing permissions in which case files should be created according to umask, with no reason to expect chmod not to be called.

[gdt]

See also #660, incorporated by reference and closed as a dup.