Uplink stuck during download

[ihaid]

When downloading files Uplink freezing on the beginning of download.

time uplink cp sj://perf/verybig.ISO Downloads/2GB.txt

Downloads/2GB.txt  67.11 MB / 1.69 GB [==>------------------------------------------------------------------------------------] 3.98%

➜  ~ uplink version
Release build
Version:            v1.98.2
Build timestamp:    20 Feb 24 20:26 EET
Git commit:         39ad43b61a965e8b3678e1c7b54e301c6d2a8a65

PATH                      VERSION
storj.io/storj            (devel)
storj.io/common           v0.0.0-20240213084453-c623afb8001a
storj.io/drpc             v0.0.33
storj.io/eventkit         v0.0.0-20240124163201-beae173bc798
storj.io/infectious       v0.0.2
storj.io/monkit-jaeger    v0.0.0-20230707083646-f15e6e8b7e8c
storj.io/picobuf          v0.0.3
storj.io/uplink           v1.12.3-0.20240207220713-2b8836cb2e03

[jtolio]

Does this happen reliably? Can you share a restricted access grant for a test file that this is happening for?

[ihaid]

Yes I can share in DM because it is prod It happens quite often

[ibrahimtan93]

hi @jtolio,

I have same issue. When mem cache full, uplink stops downloading.

[ihaid]

most likely this is timeout problem on the uplink side

[jtolio]

Appears (finally) fixed by https://review.dev.storj.io/c/storj/uplink/+/12743

[ihaid]

@jtolio Let’s try to merge it to include in the nearest release and then I will close it

[ihaid]

And we need to add some tests to it

[snichols]

I'm having a similar problem on Windows. Hopefully this fix lands soon as I'm consistently getting stuck trying to download ~8gb on this Windows box. I'm not using the uplink executable, just this simple chunk of code with v1.22.2:

src, err := storj.DownloadObject(context.Background(), xglauncher.StorjBucketName, key, nil)
if err != nil {
    return err
}
defer src.Close()
n, err := io.Copy(dst, src)
if err != nil {
    return err
}

I'm not sure if this is the same issue, but my resource monitor is showing 15 idle open TCP connections during the download with large variety of ips. This has worked flawlessly for me on my Linux box (same network).

I'll investigate adding my own retry / download resume code to compensate. That being said, this kind of problem makes me really nervous to use Storj in production.

[jtolio]

Something else may be happening, as our fix has been deployed for some time now. If you are using the library, it may be because we haven't tagged a new Uplink library release. If you're doing development, could you see if things work if you do go get storj.io/uplink@main to get the latest commit?

[snichols]

Sure, let me grab storj.io/uplink@main to see if that improves things.

[jtolio]

Confirmed that we haven't tagged a new release with the fix yet. I'll get that going.

[snichols]

Great. Doing a simple go get storj.io/uplink@main isn't playing nice with go modules:

go get go get storj.io/uplink@main
go: downloading storj.io/uplink v1.12.3-0.20240507000622-d7c4c594337f
go: malformed module path "get": missing dot in first path element

Perhaps I'm missing a secret handshake. Once the new tag drops, I'll upgrade and give it a go.

[onionjake]

go get go get storj.io/uplink@main

You have an extra go get in the command.

[snichols]

Copypasta FTL.

[snichols]

Alrighty, one download succeeded with the new code. I'll do some more tests and let you know if it breaks again. Thanks!

[jtolio]

storj.io/uplink@v1.13.0 released. Since the changes in v1.13.0 fixed all the known reproductions of this ticket, I'm going to close it. @snichols feel free to reopen if you find this happening again, and sorry for the tardy Uplink version release!