Open varungandhi-src opened 1 year ago
There is an assertion being hit for a file with a particular path only appearing once, that's incorrect and should be removed.
One weird thing though, it appears that even the size of an index is not deterministic. Normally, any hash map iteration changes should just cause re-arrangements in the final result, and shouldn't affect the total file size. So something is definitely a little weird in the non-deterministic mode...
There isn't a good reason to think it will have a lot of overhead, the main concern is potentially memory use if some workers are very fast, and hence we need to read a lot of jobs. In this case, we should remove the
--deterministic
flag and replace it with a--allow-non-determinism
flag.