duperemove sees unchanged files as changed and eventually hangs after checksumming

[klausman]

Version: current HEAD, 9e97c827707e9d709180a12ddfa16527e36fc676

Command line: duperemove -r -d -h -v --debug /store

Symptoms:

• Reports unchanged files as changed, e.g.:

  process_extents: unable to get extent
  file /store/klausman/arma/repos/maps/@isla_duala/addons/isladuala3.pbo changed

  That file has not changed in months, if not years.

• After checksumming all files, it just sits idle:

  file /store/klausman/arma/repos/maps/@isla_duala/addons/isladuala3.pbo changed
  file /store/klausman/git/kernel.org/log changed
  file /store/prometheus/data-2.x/chunks_head/001717 changed
  file /store/prometheus/data-2.x/wal/00019695 changed
  [3530542] idle
  [3530543] idle
  [3530544] idle
  [3530545] idle
  [3530546] idle
  [3530547] idle
  [3530548] idle
  [3530549] idle
  [3530550] idle
  [3530551] idle
  [3530552] idle
  [3530553] idle
        Files scanned: 265405/265405 (100.00%)
        Bytes scanned: 3.7TB/3.7TB (100.00%)
        File listing: completed

  Nothing happens after this state is reached. Both strace and ltrace just show it updating the progress indicators. There is very little CPU usage and no I/O on the mounted filesystem.

I am unsure if the two symptoms are connected or not. I am pretty sure this is not issue #305 since that seems to be tied to a peculiar filename. I have not tried using the sync method described in issue #319, since checksumming close to 4TB of data is taking a long time as it is.

I have purposefully not used a hashfile, to make sure it isn't a race on the WAL used for that.