search

storj / uplink-c

Uplink C library
MIT License
25 stars 8 forks source link

Performance bottleneck: too small chunks from uplink_download_read #21

Open Erikvv opened 1 year ago

Erikvv commented 1 year ago

When calling uplink_download_read() it only gives back 7408 bytes. I've tested this with current main branch.

For some languages/platforms there is a big overhead calling native functions and the large number of calls becomes a performance bottleneck.

Erikvv commented 1 year ago

Jeff: seems like that should be easy to fix with a call to https://pkg.go.dev/io#ReadFull (or something similar) and some careful error handling..

kmozurkewich commented 1 year ago

Do we know where the 7408 value coming from?

Erikvv commented 1 year ago

Bill says: It likely has something to do with EncryptionParameters / Encryption block size and the math 29* 256 -16

https://github.com/storj/storj-private/blob/913718ef9776ec0eba9995e5deeb08da5508bcc3/satellite/metainfo/metainfo_test.go#L1008

https://github.com/storj/uplink/blob/5c11ad6846b6a602cfa489225a5d7d7eef883896/project.go#L74

wthorp commented 1 year ago

It looks like this was previously created as an issue but possibly not fully diagnosed before closing https://github.com/storj/storj/issues/3499

wthorp commented 5 months ago

I don't have a good setup for testing a potential fix: https://github.com/storj/uplink-c/tree/download-read-all

wthorp commented 5 months ago

TopperDEL tested w/ .NET and says there's no discernible performance impact in .NET

Erikvv commented 5 months ago

I will benchmark it in PHP this weekend