Allow specifying piece size

[Ravencentric]

something like --piece-size, -ps 16MiB

[rndusr]

I'm leaning heavily towards no. There are already too many torrents with stupid piece sizes like 256 KiB for a 17 GiB torrent. I don't want to contribute to that.

If you really need a maximum limit, there is --max-piece-size.

Why do you think you need to specify the exact piece size?

[Ravencentric]

That's a valid concern, my only reason to ask for it is that currently, torf uses 8MiB till 80GiB which is nearly ~10k pieces. I believe torf should target 16MiB much earlier than that, keeping the piece count closer to ~1000.

[rndusr]

10k pieces is still less than 200KiB. Why is that an issue?

If you really care about this issue, feel free to rewrite torf.Torrent.calculate_piece_size(), find a very good reason why it is better than the existing one, and make a PR that explains everything in detail.

[Ravencentric]

private trackers recommend roughly ~1000 since too many pieces cause more protocol overhead, longer hashing times, and possibly higher load on the CPU and HDD with the upper cap being 16MiB to maintain compatibility with older clients

[rndusr]

"Private trackers" is not some kind of cohesive group. I've seen private trackers that don't care, private trackers that want custom piece sizes for torrent size ranges and private trackers with an upper limit of 8 MiB.

What exactly are the impacts of the protocol overhead, longer hashing times, etc? Is this theoretical or did someone do the math? Is there empiric data I can reproduce that backs up these theories?

[Ravencentric]

a bit late since I got busy, but I ran 10 tests each for creating a torrent and then verifying/rechecking it for both 8MiB and 16MiB on 77.67 GiB sized folder

these are the results I got (average):

8 MiB - 0:01:31 total | 874.02 MiB/s 16 MiB - 0:01:23 total | 958.26 MiB/s

[rndusr]

OK, thanks.

That's more of a difference than I expected, but 8 seconds isn't that much.

Maybe the best way to fix this would be to change torf.Torrent.calculate_piece_size(). But I've already changed it multiple times and this is getting boring. Everyone has a different opinion about piece count vs. piece size and nobody can prove their's is correct.

[cannibalChipper]

Since I had a similar request, I wanna chime in:

I'm leaning heavily towards no. There are already too many torrents with stupid piece sizes like 256 KiB for a 17 GiB torrent. I don't want to contribute to that.

That's true, but consider the fact that most people will just use auto. Anyone who will go beyond that and manually specify piece probably knows what they are doing and not having the option just stops these "power users" from further optimizing the torrent file

That's more of a difference than I expected, but 8 seconds isn't that much. Maybe the best way to fix this would be to change torf.Torrent.calculate_piece_size(). But I've already changed it multiple times and this is getting boring.

Thanks to your amazing work, Torf is the fastest torrent creator I've used. I think 8 seconds is quite a bit and will add up on multiple torrents. More speed is always better and will only make torf better.

Everyone has a different opinion about piece count vs. piece size and nobody can prove their's is correct.

In this case, it isn't about opinion, https://github.com/rndusr/torf-cli/issues/33#issuecomment-1507340954 demonstrates how it's better by numbers. i've also tested and got the same results.

I'll be happy with either of these solutions. Hope you can reconsider this issue.

[rndusr]

Thanks for the praise.

I'd still accept a PR that targets ~1000 pieces.

If you really need more optimization, we (you?) could add a CLI option that specifies the targeted number of pieces within an acceptable range. (Maybe 500 to 10,000?)

But allowing arbitrary piece sizes for any content size doesn't help anyone.