[bug] Downloading a big package fails with Incorrect Content-Range header

jblort commented 3 years ago

Environment Details (include every applicable attribute)

Operating System+version: Ubuntu 20.04
Compiler+version: clang 9
Conan version: 1.39.0
Python version: 3.8.10

Steps to reproduce (Include if Applicable)

Simply trying to run conan install android-ndk/r21d@ yields the error, the download fails, keep retrying and failing the same way.
The same error happens with a different android NDK package.

This seems similar to #7498 except the package is on Conan center in this case.

Logs (Executed commands with output) (Include/Attach if Applicable)

conan install android-ndk/r21d@
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=clang
compiler.libcxx=libstdc++11
compiler.version=9
os=Linux
os_build=Linux
[options]
[build_requires]
[env]
CC=/usr/bin/clang-9
CXX=/usr/bin/clang++-9
Installing package: android-ndk/r21d
Requirements
    android-ndk/r21d from 'conan-center' - Cache
Packages
    android-ndk/r21d:4db1be536558d833e52e862fd84d64d75c2b3656 - Download

Installing (downloading, building) binaries...
android-ndk/r21d: Retrieving package 4db1be536558d833e52e862fd84d64d75c2b3656 from remote 'conan-center' 
Downloading conanmanifest.txt completed [1028.25k]                                       
Downloading conaninfo.txt completed [0.20k]                                              
Downloading conan_package.tgz completed [1056295.44k]                                    
ERROR: Download failed, check server, possibly try again
Download failed, check server, possibly try again
Error in resumed download from https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/android-ndk/r21d/_/5013605c2f2c7c4cd448ea2d5c77e274/package/4db1be536558d833e52e862fd84d64d75c2b3656/f0d2c0a89bfe4745335cfea2ce9fd5d4/conan_package.tgz
Incorrect Content-Range header 
Waiting 0 seconds to retry...
Downloading conan_package.tgz completed [1057303.94k]                                    
ERROR: Download failed, check server, possibly try again
Download failed, check server, possibly try again
Error in resumed download from https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/android-ndk/r21d/_/5013605c2f2c7c4cd448ea2d5c77e274/package/4db1be536558d833e52e862fd84d64d75c2b3656/f0d2c0a89bfe4745335cfea2ce9fd5d4/conan_package.tgz
Incorrect Content-Range header 
Waiting 0 seconds to retry...

madebr commented 3 years ago

I have hit this problem as well: https://github.com/conan-io/conan-center-index/issues/6672

jblort commented 3 years ago

Ah I had missed this one. Closing this as this is the exact same issue.

madebr commented 3 years ago

You can keep this issue open. I don't know whether the issue is with Conan or Conan center.

lasote commented 3 years ago

I cannot reproduce nor in Ubuntu or OSX. I even tried to cut the connection and then restore and the download succeeded.

voodooattack commented 3 years ago

I'm having the same issue with llvm-core, managed to isolate it down to a specific revision:

$ conan download llvm-core/12.0.0@:1a58bb98343156787dc6776e659ce3b0244653ec
Downloading conanmanifest.txt completed [0.48k]                                          
Downloading conanfile.py completed [13.02k]                                              
Downloading conan_export.tgz completed [0.31k]                                           
Decompressing conan_export.tgz completed [0.00k]                                         
Downloading conan_sources.tgz completed [2.82k]                                          
Decompressing conan_sources.tgz completed [0.00k]                                        
llvm-core/12.0.0: Retrieving package 1a58bb98343156787dc6776e659ce3b0244653ec from remote 'conancenter' 
Downloading conanmanifest.txt completed [127.33k]                                        
Downloading conaninfo.txt completed [2.07k]                                              
Downloading conan_package.tgz completed [1060875.94k]                                    
ERROR: Download failed, check server, possibly try again
Download failed, check server, possibly try again
Error in resumed download from https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/llvm-core/12.0.0/_/ba41422a6e5174db3e923a3cb7d99466/package/1a58bb98343156787dc6776e659ce3b0244653ec/dd2408e34429c4a552e882aa3d47ed94/conan_package.tgz
Incorrect Content-Range header 
Waiting 0 seconds to retry...

It only fails for that revision on my system as far as I know. It downloads other revisions just fine if I omit that argument.

lasote commented 3 years ago

@voodooattack which operating system/conan version/python version are you using? Thanks

voodooattack commented 3 years ago

@voodooattack which operating system/conan version/python version are you using? Thanks

OS: Ubuntu 21.04 x86_64 Conan version: 1.39.0 Python version: 3.9.5

lasote commented 3 years ago

Cannot reproduce either 😢 I've tried with that configuration and downloading the revision you mentioned.

voodooattack commented 3 years ago

That is really odd. Is there any way I can help from my end?

jamasnga commented 2 years ago

What I have found is that this issue does not seem to occur when revisions_enabled=1 is set in the conan.conf. Had the same problem with conan install android-ndk/r21d@ and with revisions_enabled=1 the package could be downloaded every time. However I'm not sure if this is really the solution as the problem did not always occur without revisions_enabled set. Nevertheless, it seems to make a difference. Can anyone verify this?

OS: Ubuntu 20.04 Conan: 1.45.0 Python: 3.8.10

memsharded commented 2 years ago

There is a concurrency problem that maybe might happen when not using revisions: without revisions, the revision is always "0". If for some reason some other client decides to do an upload of a new build, that overwrites the "0" revision. Not sure to which extend Artifactory could have some concurrency limitation for this scenario, but it could be than using revisions practically removes this problem, because new builds happen under new revisions, and they never really overwrite the previous ones (that some other client could be downloading concurrently).

jamasnga commented 2 years ago

Attached is the console output: first without revisions_enabled then with revisions_enabled=1

issue_9412.txt

We also discovered this with an internal package we created and deployed to our own Artifactory (7.27.6). The package is ca. 3GB and was built with revisions_enabled=1. If the client does not have revisions_enabled=1 we run into the same problem. The download fails. With revisions_enabled=1 the download succeeds every time.

From my point of view this is definitely a bug. How can a user know if a package was built with package revisions enabled or not? He or she shouldn't have to.

I will perform an experiment with our own package and build it without package revisions enabled and see if this solves the problem on the client side.

memsharded commented 2 years ago

From my point of view this is definitely a bug. How can a user know if a package was built with package revisions enabled or not? He or she shouldn't have to.

yes it might be a bug, but as apparently it seems to be in the Artifactory side, it is something that we cannot fix it from here. And it will take definitely more time. As using revisions is recommended anyway, and Conan 2.0 (already in alpha.4) is always using revisions, I am not sure that such a fix would land in a useful time window.

jamasnga commented 2 years ago

As using revisions is recommended anyway, and Conan 2.0 (already in alpha.4) is always using revisions, I am not sure that such a fix would land in a useful time window.

OK, thanks for that info. We will simply enable package revisions from now on since that seems to solve the issue.

jblort commented 2 years ago

Not sure if this is still relevant, but I stumbled on the issue again, and tried to actually wget the package directly with:

wget https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/android-ndk/r21d/_/d6416b530a0cbadc7aa8f7f6f8942757/package/4db1be536558d833e52e862fd84d64d75c2b3656/4dea2f1dfa5d49d6209017f94958ab19/conan_package.tgz

This yielded the following output, which looks like there's an actual issue with the remote or with the package, independently from the conan client.

--2022-04-05 11:36:20--  https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/android-ndk/r21d/_/d6416b530a0cbadc7aa8f7f6f8942757/package/4db1be536558d833e52e862fd84d64d75c2b3656/4dea2f1dfa5d49d6209017f94958ab19/conan_package.tgz
Resolving center.conan.io (center.conan.io)... 34.107.244.196
Connecting to center.conan.io (center.conan.io)|34.107.244.196|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1238740245 (1,2G) [application/x-gzip]
Saving to: ‘conan_package.tgz’

conan_package.tgz              87%[=========================================>       ]   1,01G  4,85MB/s    in 4m 35s  

2022-04-05 11:40:55 (3,76 MB/s) - Connection closed at byte 1083928385. Retrying.

memsharded commented 2 years ago

Thanks @jblort If we can reproduce this against the ConanCenter server, removing the client side, that would help reporting something against Artifactory if necessary.

lasote commented 2 years ago

@jblort we are trying the wget command and it is working ok. Which OS are you using? any hint? thanks.

jblort commented 2 years ago

@lasote Hello! I'm using Ubuntu 20.04, and wget with the following version info for what it's worth: GNU Wget 1.20.3 built on linux-gnu.

In the meantime I've managed to circumvent that problem within conan by setting revisions_enabled=1 under the general settings of my conan.conf. It looks like it downloads a different package so I'm wondering if there isn't an issue particularly with this package here.

lasote commented 2 years ago

Thanks @jblort! No luck, I tried on docker with that specific ubuntu version and verified same wget version with same result. Also the conan install command :(

memsharded commented 2 years ago

In the meantime I've managed to circumvent that problem within conan by setting revisions_enabled=1 under the general settings of my conan.conf. It looks like it downloads a different package so I'm wondering if there isn't an issue particularly with this package here.

In both cases it should download the latest revision, so the possibility is that some PR created a new revision in the server and is no longer failing. Can you please @jblort try again without revisions, and verify it is still failing on your side?

jblort commented 2 years ago

Ok I just retried without revisions and it completed without error when running conan install android-ndk/r21d@. The ID of the package I downloaded matches the one that had an issue before, but it seems fine now... :thinking:

memsharded commented 2 years ago

Ok I just retried without revisions and it completed without error when running conan install android-ndk/r21d@. The ID of the package I downloaded matches the one that had an issue before, but it seems fine now... 🤔

Yes, that is the annoying thing about this kind of errors involving networking, they are really difficult to reproduce...

memsharded commented 2 years ago

I need to remove this from the milestone, as we haven't manage to reproduce so far. There are many things that can affect here, network issues, proxies, etc, so this might be challenging to solve. I am closing it, as there is little more we can do, but if anyone keeps persistently hitting this issue, or have any possible way to reproduce, please re-open or create a new ticket.

Many thanks to all for all the feedback!

elejke commented 1 year ago

I've managed to reproduce this multiple times with Ubuntu 22.04 with conan==1.59.0 with android-ndk/r21e, for android-ndk/r25 it works correctly (as it is about 500mb size, when r21e is ~1250mb)

conan install android-ndk/r21e@
Configuration:
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=11
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

android-ndk/r21e: WARN: Package binary is corrupted, removing: 4db1be536558d833e52e862fd84d64d75c2b3656
Installing package: android-ndk/r21e
Requirements
    android-ndk/r21e from 'conancenter' - Cache
Packages
    android-ndk/r21e:4db1be536558d833e52e862fd84d64d75c2b3656 - Download

Installing (downloading, building) binaries...
android-ndk/r21e: Retrieving package 4db1be536558d833e52e862fd84d64d75c2b3656 from remote 'conancenter' 
Downloading conanmanifest.txt completed [1068.21k]                                       
Downloading conaninfo.txt completed [0.29k]                                              
Downloading conan_package.tgz completed [1061341.32k]                                    
ERROR: Download failed, check server, possibly try again
Download failed, check server, possibly try again
Error in resumed download from https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/android-ndk/r21e/_/6de6a2d288f9ce3fa5bbd402d018652d/package/4db1be536558d833e52e862fd84d64d75c2b3656/86fc1b96570ed8e6f476bb21e809ad0d/conan_package.tgz
Incorrect Content-Range header 
Waiting 0 seconds to retry...

memsharded commented 1 year ago

I can verify that it fails in 1.59, in our gcc11 docker image, without using revisions:

Downloading conanmanifest.txt completed [1068.21k]
Downloading conaninfo.txt completed [0.29k]
Downloading conan_package.tgz completed [1060490.69k]
ERROR: Download failed, check server, possibly try again
Download failed, check server, possibly try again
Error in resumed download from https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/android-ndk/r21e/_/6de6a2d288f9ce3fa5bbd402d018652d/package/4db1be536558d833e52e862fd84d64d75c2b3656/86fc1b96570ed8e6f476bb21e809ad0d/conan_package.tgz
Incorrect Content-Range header
Waiting 0 seconds to retry...
Downloading conan_package.tgz completed [1057562.31k]
ERROR: Download failed, check server, possibly try again
Download failed, check server, possibly try again
Error in resumed download from https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/android-ndk/r21e/_/6de6a2d288f9ce3fa5bbd402d018652d/package/4db1be536558d833e52e862fd84d64d75c2b3656/86fc1b96570ed8e6f476bb21e809ad0d/conan_package.tgz
Incorrect Content-Range header

It finally succeed, but it seems a good start to reproduce. Next steps:

see if it fails with revisions enabled
see if it fails when downloading from other server, not ConanCenter (could be a server side, memory issue or something like that?)

memsharded commented 1 year ago

With revisions_enabled, I have been a bit more lucky, and even if the download struggles at approx 1Gb, it was able to resume with the correct header:

Downloading conan_package.tgz completed [1058831.94k]
Continuing download of conan_package.tgz:  92%|#########2| 1.06G/1.15G [00:12<00:01, 65.5MB/s]

If you could please @elejke confirm:

Do you have revisions enabled? If you don't, can you still reproduce if you enabled them?
Does it happens only when downloading from ConanCenter, or also when downloading from your own server?

elejke commented 1 year ago

With revisions_enabled=1 I have managed to download it, but there was a error, like there are no such package for this configuration and I shoud build=missing. So, I successfully "build from source" it and using now.

But this problem have been constantly reproducsing without revisions.

I didn't check for my own server but, I already have it there, so, I can check it if needed.

memsharded commented 1 year ago

But this problem have been constantly reproducsing without revisions.

Conan 2.0 uses the revisions API always, and using revisions has been the recommended approach in 1.X too. I don't think that there is something that different in API v2 that would fix it and I am afraid that we might see issues too, but if it is fixed with revisions, that would be great.

I didn't check for my own server but, I already have it there, so, I can check it if needed.

Yes, please, it would be super useful to know if it fails only from ConanCenter or other servers too.

elejke commented 1 year ago

Ah... I am wrong, it was for Darwin package and I still don't have exactly this revision of the package to check it for linux...

memsharded commented 1 year ago

I have done some progress:

I have built and uploaded r21e to my own Artifactory in the cloud
I can download it without any issues from the client.
I would say that this is not from Conan client side, but server side, and specifically ConanCenter-only, as normal Artifactory seems to work fine
I think it would be related to the ConanCenter infra, which is using Google CDN. From https://cloud.google.com/cdn/docs/caching, maybe this is related?

Cloud Storage supports byte range requests for most objects. However, Cloud Storage does not support byte range requests for objects with Content-Encoding: gzip metadata unless the client request includes an Accept- Encoding: gzip header. If you have Cloud Storage objects larger than 10 MB, make sure that they do not have Content-Encoding: gzip metadata. For information about how to edit object metadata, see Viewing and editing object metadata.

memsharded commented 7 months ago

Trying to follow up on this, sorry that we haven't been able to prioritize this enough.

I can still reproduce this with:

λ curl https://center.conan.io/artifactory/api/conan/conan-center/v1/files/_/android-ndk/r21d/_/d6416b530a0cbadc7aa8f7f6f8942757/package/4db1be536558d833e52e862fd84d64d75c2b3656/4dea2f1dfa5d49d6209017f94958ab19/conan_package.tgz --output my_conan_package.tgz
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 87 1181M   87 1034M    0     0  6479k      0  0:03:06  0:02:43  0:00:23 5843k
curl: (56) Failure when receiving data from the peer

So still an issue :(, and still an issue related specifically to ConanCenter only, which makes it more challenging :(

ConanCenter has a CDN in front of it, and files should be served by the CDN. Adding @davidsanfal just in case he knows some CDN limitation or something like that.

conan-io / conan