search

crops / extsdk-container

A container image that is able to setup and use an extensible sdk generated using openembedded-core.
GNU General Public License v2.0
35 stars 15 forks source link

curl fails to download after a certain time, but wget does not. #11

Open manojkiraneda opened 3 years ago

manojkiraneda commented 3 years ago

The container seems to use curl to download the .sh file and that is failing for some reason, but the wget on the same file seems to work. Can you guys help me to figure out what is the issue here ?

Even with wget, it seems like we are reconnection after certain time, do i need to enable some option in curl to get this working ?


----------------------------------------------------------------------------------------------------------------------
Attempting to download https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
#####                                                                      7.1%
curl: (18) transfer closed with 314315481 bytes remaining to read
Unable to download "https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh".

----------------------------------------------------------------------------------------------------------------------------
[Juliet@juliet ci-stage-area]$ wget https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
--2020-10-11 11:27:51--  https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
Resolving jenkins.openbmc.org (jenkins.openbmc.org)... 169.44.58.24
Connecting to jenkins.openbmc.org (jenkins.openbmc.org)|169.44.58.24|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 338280321 (323M) [application/x-sh]
Saving to: ‘oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh’

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s   6%[=====>                                                                                                ]  20.84M   324KB/s    in 71s     

2020-10-11 11:29:05 (303 KB/s) - Connection closed at byte 21854168. Retrying.

--2020-10-11 11:29:06--  (try: 2)  https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
Connecting to jenkins.openbmc.org (jenkins.openbmc.org)|169.44.58.24|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 338280321 (323M), 316426153 (302M) remaining [application/x-sh]
Saving to: ‘oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh’

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s  12%[++++++=====>                                                                                          ]  39.44M   392KB/s    in 46s     

2020-10-11 11:29:54 (410 KB/s) - Connection closed at byte 41353000. Retrying.

--2020-10-11 11:29:56--  (try: 3)  https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
Connecting to jenkins.openbmc.org (jenkins.openbmc.org)|169.44.58.24|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 338280321 (323M), 296927321 (283M) remaining [application/x-sh]
Saving to: ‘oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh’

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s  19%[++++++++++++======>
rewitt1 commented 3 years ago

Hi @manojkiraneda,

Unfortunately seems to be caused by some interaction between the client and the server when download speeds are slow. I limited the download speeds on my router, and even when using a web browser, the download still fails.

Here is an example of a similar issue, although I wouldn't go so far as to say it is the exact same.

Outside of contacting openbmc, I'm not sure of a solution, because it seems to occur independent of the client.

manojkiraneda commented 3 years ago

Hi @rewitt1

Thanks for looking at this issue. I agree that it's probably because of the server, but wget seems to work as it is doing a re-try, can we do similar settings with curl as well, so that we don't break out with a partial file in case of bad servers? That would really help.

I guess something like this might help.

Here is a snip of wget doing a retry & resuming from a closed connection from server.

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s   6%[=====>                                                                                                ]  20.84M   324KB/s    in 71s     

2020-10-11 11:29:05 (303 KB/s) - Connection closed at byte 21854168. Retrying.

--2020-10-11 11:29:06--  (try: 2)  https://jenkins.openbmc.org/job/latest-master-sdk/label=docker-builder,target=witherspoon/lastSuccessfulBuild/artifact/deploy/sdk/oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh
Connecting to jenkins.openbmc.org (jenkins.openbmc.org)|169.44.58.24|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 338280321 (323M), 316426153 (302M) remaining [application/x-sh]
Saving to: ‘oecore-x86_64-arm1176jzs-toolchain-nodistro.0.sh’

oecore-x86_64-arm1176jzs-toolchain-nodistro.0.s  12%[++++++=====>
rewitt1 commented 3 years ago

The most obvious ways I can think of fixing this would be one of the following:

  1. Add an extra argument to the entry point that tells curl to retry
  2. Add a --curl-arguments parameter that allows for specifying arguments to curl in addition to the url
  3. Replace --url with --curl-arguments

I'm not sure which of these I like best yet.

manojkiraneda commented 3 years ago

To solve only this problem, I would personally vote for 1 as retry only kicks-in when the connection is closed and has no effect in case of a better server.

But if you want to provide the user the facility to give his own parameters, then I would go for 2.

manojkiraneda commented 3 years ago

@rewitt1 sorry for asking, but when can i have this fix ? I was trying to use this container for our CI purpose and was blocked because of this.

rewitt1 commented 3 years ago

Hi @manojkiraneda,

I would like to give you an ETA, but I'm unsure of when I'll be able to get to it.

If you need a solution sooner, you can build the container image locally, substituting in the curl arguments you need.

manojkiraneda commented 3 years ago

sure thanks @rewitt1 will do.