search

Rudxain / xorsum

Get XOR checksum with this command-line tool
https://crates.io/crates/xorsum
GNU Lesser General Public License v3.0
0 stars 1 forks source link

Multithreading #4

Closed Rudxain closed 3 months ago

Rudxain commented 2 years ago

This algorithm is infinitely parallelizable, we should take advantage of that

Rudxain commented 2 years ago

The problem is how should we go about it. Should each file be assigned to each logical core? Or should each file be hashed sequentially, but the data split among all logical cores?

Rudxain commented 2 years ago

Why not both? What if for each file it spawned a thread, and for each thread it spawned MORE threads equal to logical cores. That way, it will always use max parallelism, and the OS would be in charge of task scheduling. But maybe this alternative uses too much memory.

Related: https://github.com/BLAKE3-team/BLAKE3/pull/170#issuecomment-835407340

Rudxain commented 2 years ago

After reading https://github.com/BLAKE3-team/BLAKE3/issues/31 I'm now very hesitant of implementing this. I'll defer it, to give priority to other issues

Rudxain commented 2 years ago

I noticed they solved the Issue long ago, we should too

Rudxain commented 3 months ago

The algorithm isn't CPU-bound, so this is unnecessary complexity