Hash function for bucket with zero collision. This can be used to de-duplicate material data and achieve bigger chunks sizes.
Workers that are idle can be utilized to de-duplicate gradually
Proposal:
Hash function is sum of each bucket value (entry start_index) in a bucket stored in a u128 (2^128 > 2^32 * 512) . When a entry is added/changed we simply update the entry hash value (no lazy hash calculation)
When a grid worker have no work it counts up a idle worker counter. If all workers become idle, then each worker is assigned a bucket and find duplicate entries using the hash function (since one worker have one bucket in this stage there will be no potential of RC and therefore is lockless). Testing should be done between this and the potential of multiple workers working on the same bucket to find the optimal solution. Each time changes are applied to the grid they will reset this check and brute force check the whole grid for duplicates again.
Hash function for bucket with zero collision. This can be used to de-duplicate material data and achieve bigger chunks sizes.
Workers that are idle can be utilized to de-duplicate gradually
Proposal:
Hash function is sum of each bucket value (entry start_index) in a bucket stored in a u128 (2^128 > 2^32 * 512) . When a entry is added/changed we simply update the entry hash value (no lazy hash calculation)
When a grid worker have no work it counts up a idle worker counter. If all workers become idle, then each worker is assigned a bucket and find duplicate entries using the hash function (since one worker have one bucket in this stage there will be no potential of RC and therefore is lockless). Testing should be done between this and the potential of multiple workers working on the same bucket to find the optimal solution. Each time changes are applied to the grid they will reset this check and brute force check the whole grid for duplicates again.