Open thiagosfsilva opened 5 years ago
Encountered the same problem.
But a modified version of aria2 was found in a software called PanDownload, which does not have this problem. However, the author of the software did not disclose the source code of the modified aria2, which violated the GPL license.
I get the same error sometimes, but not always, when using aria2 as the external downloader to youtube-dl for videos on YouTube. Retrying the exact same commandline will then sometimes succeed completely but some other times only progress a bit and then abort with the same error message again. There is no apparent corruption of the downloaded files when completed. This has happened with the aria2c (1.35.0) from Devuan Linux, the Win64 binary (1.35.0) from the official releases on GitHub as well as a locally compiled from current git on Devuan Linux. Downloading the same videos from YouTube using youtube-dl but without aria2 as external downloader works without error (but is slower, so not preferable).
My logs, etc would show the equivalent to those above. Any further information needed to troubleshoot and find the error and a resolution?
I get the same problem, from time to time.... But when it happens it breaks my whole logic afterwards. The log info is the same as described above. Do you have any suggestion if some of the aria2c options from here https://aria2.github.io/manual/en/html/aria2c.html will maybe solve this problem?
Any help is appreciated
I'm seeing the same thing:
2020-09-02 15:55:43.089204 [INFO] [HttpConnection.cc:153] CUID#62 - Response received:
HTTP/1.1 206 Partial Content
Server: nginx/1.16.1 (Ubuntu)
Date: Wed, 02 Sep 2020 22:55:43 GMT
Content-Type: application/octet-stream
Content-Length: 2062988
Last-Modified: Fri, 17 Nov 2017 23:16:39 GMT
Connection: keep-alive
ETag: "5a0f6dd7-800000"
Strict-Transport-Security: max-age=15724800
Expires: Thu, 03 Sep 2020 04:55:43 GMT
Cache-Control: max-age=21600
Content-Range: bytes 0-2062987/8388608
2020-09-02 15:55:43.089204 [ERROR] [AbstractCommand.cc:345] CUID#62 - Download aborted. URI=https://archive.org/download/MAME0.37b5_MAME2000_Reference_Set_Update_2_ROMs_Samples/.____padding_file/163
Exception: [AbstractCommand.cc:345] errorCode=8 URI=https://ia801207.us.archive.org/1/items/MAME0.37b5_MAME2000_Reference_Set_Update_2_ROMs_Samples/.____padding_file/163
-> [HttpResponse.cc:86] errorCode=8 Invalid range header. Request: 0-2062987/2062988, Response: 0-2062987/8388608
2020-09-02 15:55:43.098016 [DEBUG] [AbstractCommand.cc:470] CUID#62 - Aborting download
same problem, first request is for the whole file, response:
ETag: "1292083385"
Content-Length: 44043948
Server: lighttpd/1.4.28
Something is missing, there is no "Accept-Ranges" header returned! second request for the second half ignores this fact and plows on:
Range: bytes=23068672-44043947
and the response doesnt match
Content-Range: bytes 23068672-44043947/44043969
Content-Length: 20975276
Server: lighttpd/1.4.28
comparing same file in a browser, browser actually sends
Range: bytes=0-
and receives
Accept-Ranges: bytes
Content-Length: 44043969
Content-Range: bytes 0-44043968/44043969
ETag: "2888719043"
Server: lighttpd/1.4.28
subsequent requests also include If-Range: "2888719043"
Aria2 should probably start sending "Range: bytes=0-" with first request, and "If-Range"
CUID#369 - Download aborted. URI=https://209zy.youkutv.cc/2020/12/02/bL5esVcJCK0BaZ97/out205.ts Exception: [AbstractCommand.cc:351] errorCode=8 URI=https://209zy.youkutv.cc/2020/12/02/bL5esVcJCK0BaZ97/out205.ts -> [HttpResponse.cc:86] errorCode=8 Invalid range header. Request: 1818302-1841271/1841272, Response: 0-1841271/1841272
Same problem. Now that two and a half years have passed, doesn't seem this problem is about to be fixed.
lighttpd 1.4.28 was released Aug 2010, almost 11 years ago. Please use the most recent lighttpd 1.4.59, which has better Range support than does lighttpd 1.4.28.
[...] Please use the most recent lighttpd [...]
This is probably a good idea to avoid other issues @gstrauss, but I doubt for example YouTube would consider moving from Google's own ESF server to the most recent lighttpd and unless Google's proprietary software would secretly be an outdated version of lighttpd, that does seem extremely unlikely to have any relevance to this aria2 issue.
@gstrauss i demonstrated the issue with an nginx 1.16.1 server. this isn't limited to old versions of lighttpd.
I have reproduced the issue with aira2c build from the master branch. I have tried to make a workaround. The issue is triggered quite reliably with master and not with the updated version. See the logs attached. outlog-EX_INVALID_RANGE_HEADER.log.gz outlog-master.log.gz
Encountered the same problem.
But a modified version of aria2 was found in a software called PanDownload, which does not have this problem. However, the author of the software did not disclose the source code of the modified aria2, which violated the GPL license.
where do I download this PanDownload app? can't find it in google
where do I download this PanDownload app? can't find it in google
You probably can't because it violated their silly little loicense, because fuck us who experience this issue.
bug still happens 🙄
[HttpResponse.cc:86] errorCode=8 Invalid range header. Request: 0-96669988/96669989, Response: 0-96669988/96670038
[HttpResponse.cc:86] errorCode=8 Invalid range header. Request: 0-2062987/2062988, Response: 0-2062987/8388608 2020-09-02 15:55:43.098016
[HttpResponse.cc:86] errorCode=8 Invalid range header. Request: 1818302-1841271/1841272, Response: 0-1841271/1841272
src/HttpRequest.cc:HttpRequest::isRangeSatisfied()
checks for an exact match to what is sent in the request. That is incorrect as a server might not support Range requests, might coalesce multiple ranges, might choose to return the entire file, might choose to return a larger range than requested, etc. Also, the file size may have changed. A robust client should use a strong validator, such as an ETag when making Range requests so that it knows that the Range requested is part of the same file (not modified) as other Range requests to the same file. Of course, when a log file is extended, ignoring the strong validator and requesting additional bytes via Range request may be appropriate. There is no one-size-fits-all.
aria2 does not handle Range responses that are not an exact match to what is sent in the request. Recovering properly can be difficult when requests have been pipelined. (Using HTTP/2 has much better control recovering than HTTP/1.1 using pipelining.)
Modifying src/HttpRequest.cc:HttpRequest::isRangeSatisfied()
to not fail on valid Range responses is not difficult, but might break assumptions elsewhere in the aria2 code, e.g. src/RequestGroup.cc:validateTotalLength()
.
Here is a completely UNTESTED guess. I am not a developer of aria2. caveat emptor. If you test this, make backups and do not test on important files! This code is a kludge and makes a modification inside a function beginning "is...", which is not expected from the function name and is generally not considered good practice.
--- a/src/HttpRequest.cc
+++ b/src/HttpRequest.cc
@@ -123,10 +123,19 @@ bool HttpRequest::isRangeSatisfied(const Range& range) const
if (!segment_) {
return true;
}
- return getStartByte() == range.startByte &&
- (getEndByte() == 0 || getEndByte() == range.endByte) &&
- (fileEntry_->getLength() == 0 ||
- fileEntry_->getLength() == range.entityLength);
+ if (getStartByte() < range.startByte ||
+ (getEndByte() != 0 && getEndByte() > range.endByte) ||
+ fileEntry_->getLength() > range.entityLength) { // (or truncate file?)
+ return false;
+ }
+ if (getStartByte() != range.startByte) {
+ //fileEntry_->setOffset(range.startByte);
+ return false;
+ }
+ if (fileEntry_->getLength() != 0 &&
+ fileEntry_->getLength() < range.entityLength) {
+ fileEntry_->setLength(range.entityLength);
+ }
+ return true;
}
namespace {
The above (commented out line) would VERY LIKELY corrupt files when the start of the range in the response does not match the exact start of the range requested because the write position in segment_
might need to be updated, too. (See the third example quoted in the top of this comment), so that is why I commented out the setOffset()
and do not handle the case where the server response is a range earlier than the startByte
, e.g. if the server returns the entire file instead of the requested range, or returns the range from the start of the file.
Still happens with 1.36.0 from time to time. I have a Linux server running nginx as a file server, when downloading from it and the task is almost finished, aria2 fails occasionally:
Jun 27 17:31:34 hostname aria2c[3304]: 06/27 17:31:34 [ERROR] CUID#24015 - Download aborted. URI=[hidden]
Jun 27 17:31:34 hostname aria2c[3304]: Exception: [AbstractCommand.cc:351] errorCode=8 URI=[hidden]
Jun 27 17:31:34 hostname aria2c[3304]: -> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 1156579328-1270874111/1422767233, Response: 0-1422767232/1422767233
Jun 27 17:31:39 hostname aria2c[3304]: [#bca86a 1.0GiB/1.3GiB(77%) CN:6 DL:10MiB ETA:28s]
Jun 27 17:31:39 hostname aria2c[3304]: [#bca86a 1.0GiB/1.3GiB(78%) CN:6 DL:11MiB ETA:26s]
Jun 27 17:31:39 hostname aria2c[3304]: [#bca86a 1.0GiB/1.3GiB(78%) CN:6 DL:10MiB ETA:27s]
Jun 27 17:31:39 hostname aria2c[3304]: [#bca86a 1.0GiB/1.3GiB(79%) CN:6 DL:10MiB ETA:26s]
Jun 27 17:31:39 hostname aria2c[3304]: [#bca86a 1.0GiB/1.3GiB(80%) CN:6 DL:10MiB ETA:24s]
Jun 27 17:31:39 hostname aria2c[3304]: 06/27 17:31:39 [ERROR] CUID#24014 - Download aborted. URI=[hidden]
Jun 27 17:31:39 hostname aria2c[3304]: Exception: [AbstractCommand.cc:351] errorCode=8 URI=[hidden]
Jun 27 17:31:39 hostname aria2c[3304]: -> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 1156579328-1270874111/1422767233, Response: 0-1422767232/1422767233
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.0GiB/1.3GiB(81%) CN:5 DL:10MiB ETA:23s]
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.0GiB/1.3GiB(81%) CN:5 DL:10MiB ETA:23s]
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.0GiB/1.3GiB(82%) CN:5 DL:10MiB ETA:22s]
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(83%) CN:5 DL:10MiB ETA:21s]
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(84%) CN:5 DL:9.7MiB ETA:22s]
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(84%) CN:5 DL:9.7MiB ETA:20s]
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(85%) CN:5 DL:10MiB ETA:17s]
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(86%) CN:5 DL:9.9MiB ETA:18s]
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(87%) CN:5 DL:10MiB ETA:16s]
Jun 27 17:31:48 hostname aria2c[3304]: *** Download Progress Summary as of Tue Jun 27 17:31:48 2023 ***
Jun 27 17:31:48 hostname aria2c[3304]: ===============================================================================
Jun 27 17:31:48 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(88%) CN:5 DL:10MiB ETA:15s]
Jun 27 17:31:48 hostname aria2c[3304]: FILE: /home/username/Downloads/filename
Jun 27 17:31:48 hostname aria2c[3304]: -------------------------------------------------------------------------------
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(88%) CN:5 DL:10MiB ETA:15s]
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(88%) CN:5 DL:10MiB ETA:15s]
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(89%) CN:5 DL:10MiB ETA:13s]
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.1GiB/1.3GiB(90%) CN:5 DL:10MiB ETA:12s]
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(91%) CN:5 DL:10MiB ETA:11s]
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(91%) CN:5 DL:10MiB ETA:10s]
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(92%) CN:5 DL:9.7MiB ETA:10s]
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(93%) CN:5 DL:10MiB ETA:8s]
Jun 27 17:31:57 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(94%) CN:5 DL:9.8MiB ETA:7s]
Jun 27 17:31:57 hostname aria2c[3304]: 06/27 17:31:57 [ERROR] CUID#23996 - Download aborted. URI=[hidden]
Jun 27 17:31:57 hostname aria2c[3304]: Exception: [AbstractCommand.cc:351] errorCode=8 URI=[hidden]
Jun 27 17:31:57 hostname aria2c[3304]: -> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 865075200-890241023/1422767233, Response: 0-1422767232/1422767233
Jun 27 17:31:57 hostname aria2c[3304]: 06/27 17:31:57 [ERROR] CUID#24016 - Download aborted. URI=[hidden]
Jun 27 17:31:57 hostname aria2c[3304]: Exception: [AbstractCommand.cc:351] errorCode=8 URI=[hidden]
Jun 27 17:31:57 hostname aria2c[3304]: -> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 852492288-865075199/1422767233, Response: 0-1422767232/1422767233
Jun 27 17:32:05 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(95%) CN:3 DL:10MiB ETA:6s]
Jun 27 17:32:05 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(95%) CN:3 DL:10MiB ETA:5s]
Jun 27 17:32:05 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(96%) CN:3 DL:8.8MiB ETA:6s]
Jun 27 17:32:05 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(96%) CN:3 DL:9.2MiB ETA:4s]
Jun 27 17:32:05 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(97%) CN:3 DL:8.4MiB ETA:4s]
Jun 27 17:32:05 hostname aria2c[3304]: [#bca86a 1.2GiB/1.3GiB(97%) CN:3 DL:7.9MiB ETA:3s]
Jun 27 17:32:05 hostname aria2c[3304]: [#bca86a 1.3GiB/1.3GiB(98%) CN:3 DL:7.7MiB ETA:2s]
Jun 27 17:32:05 hostname aria2c[3304]: [#bca86a 1.3GiB/1.3GiB(98%) CN:2 DL:7.5MiB ETA:2s]
Jun 27 17:32:05 hostname aria2c[3304]: 06/27 17:32:05 [ERROR] CUID#24023 - Download aborted. URI=[hidden]
Jun 27 17:32:05 hostname aria2c[3304]: Exception: [AbstractCommand.cc:351] errorCode=8 URI=[hidden]
Jun 27 17:32:05 hostname aria2c[3304]: -> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 849377328-857735167/1422767233, Response: 0-1422767232/1422767233
Jun 27 17:32:05 hostname aria2c[3304]: 06/27 17:32:05 [ERROR] CUID#24018 - Download aborted. URI=[hidden]
Jun 27 17:32:05 hostname aria2c[3304]: Exception: [AbstractCommand.cc:351] errorCode=8 URI=[hidden]
Jun 27 17:32:05 hostname aria2c[3304]: -> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 857735168-866123775/1422767233, Response: 0-1422767232/1422767233
Jun 27 17:32:06 hostname aria2c[3304]: 06/27 17:32:06 [NOTICE] Download GID#bca86a66f092f7ba not complete: /home/username/Downloads/filename
But if I restart the task, it will finish soon.
I had the issue with vanilla aria2 (1.36.0-1) from Ubuntu 22.04. After applying the patch https://github.com/aria2/aria2/pull/1869 and rebuilding the package, I was able to get 100+GB zip file without errors (with 10 threads and 1MB chunks). Can anyone try the patch? Maybe with few more confirmations the PR will get merged eventually...
I had the issue with vanilla aria2 (1.36.0-1) from Ubuntu 22.04. After applying the patch #1869 and rebuilding the package, I was able to get 100+GB zip file without errors (with 10 threads and 1MB chunks). Can anyone try the patch? Maybe with few more confirmations the PR will get merged eventually...
Hi would you mind sharing the compiled version for windows? I'm also experiencing this error. Thanks
@Maherz123 , Sorry I have no windows computer neither any experience in building things in windows. You could try dpkg-buldpackage in wsl2 in windows...
@Maherz123 , Sorry I have no windows computer neither any experience in building things in windows. You could try dpkg-buldpackage in wsl2 in windows...
It's ok I already found the version with this fix
@Maherz123 Where did you find the version with fix applied?
I see the same error ([HttpResponse.cc:86] errorCode=8 Invalid range header. Request: 513802240-1026555903/2052974093, Response: 0-2052974092/2052974093
) whenever I use larger than default value for max-connection-per-server
.
You maybe visiting the cloudflare website, just turn off the cache.
Same error today, torrent from archive.org, I did cancel with Ctrl-c
and re-run, still get stuck on 99%.
My Torrent download never completes?
Most likely, you have an out-of-date Torrent for the Item you are trying to download. The first thing to try is re-downloading the Items’ Torrent, and trying again.
Torrents for Items on the Internet Archive can become obsolete when the Item the Torrent is for changes. In that case, some or (more rarely) all of the files within the Torrent will fail to download completely.
This is because our Torrents rely heavily on webseeding (download directly from our servers, when no peers have the files you are seeking). When files on our servers have changed since the Torrent was made, they will not match expected ‘piece hashes’; some BitTorrent clients (e.g. Transmission) will attempt to re-download file pieces from changed files over and over, forever, assuming there was an error in transmission, when in fact the file has changed.
Torrents that never download at all most likely are the result of a different problem, lack of client support for Getright-style webseeding.
Aria does not support archive.org torrents or what?
Official package from Extra repo:
$ pacman -Qi aria2
Name : aria2
Version : 1.37.0-1
Description : Download utility that supports HTTP(S), FTP, BitTorrent, and Metalink
Architecture : x86_64
URL : https://aria2.github.io
Licenses : GPL
Groups : None
Provides : None
Depends On : gnutls libxml2 sqlite c-ares ca-certificates libssh2
Optional Deps : None
Required By : None
Optional For : yt-dlp
Conflicts With : None
Replaces : None
Installed Size : 5.34 MiB
Packager : Felix Yan <felixonmars@archlinux.org>
Build Date : Tue 21 Nov 2023 06:18:23 AM -03
Install Date : Wed 22 Nov 2023 10:32:40 AM -03
Install Reason : Explicitly installed
Install Script : No
Validated By : Signature
Error:
03/18 09:29:27 [ERROR] CUID#20 - Download aborted. URI=https://archive.org/download/shiki-jitsu-2000-eng-subs-hd/.____padding_file/0
Exception: [AbstractCommand.cc:351] errorCode=8 URI=https://ia902608.us.archive.org/33/items/shiki-jitsu-2000-eng-subs-hd/.____padding_file/0
-> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 0-1442461/1442462, Response: 0-1442461/8388608
03/18 09:29:27 [ERROR] CUID#27 - Download aborted. URI=http://ia902608.us.archive.org/33/items/shiki-jitsu-2000-eng-subs-hd/.____padding_file/0
Exception: [AbstractCommand.cc:351] errorCode=8 URI=http://ia902608.us.archive.org/33/items/shiki-jitsu-2000-eng-subs-hd/.____padding_file/0
-> [HttpResponse.cc:81] errorCode=8 Invalid range header. Request: 0-1442461/1442462, Response: 0-1442461/8388608
distro: Arch Linux x86_64
kernel: 6.1.52-1-lts
shell: bash 5.2.26
term: tmux
cpu: Intel i7-4790 (8) @ 3.600GHz
gpu: AMD ATI Radeon RX 470/480/570/570X/580/580X/590
You maybe visiting the cloudflare website, just turn off the cache.
can you tell me how to do it. Noob here!
You maybe visiting the cloudflare website, just turn off the cache.
can you tell me how to do it. Noob here!
If it's your own site, turn off caching.
@beck-8 Sadly not mine.
@tatsuhiro-t this issue seems to have gotten buried. aria2 has a deficient and incorrect implementation of HTTP Range. I posted a partial patch above in https://github.com/aria2/aria2/issues/1344#issuecomment-1570701152 Please review. The patch I posted lists some limitations documented, and unfortunately, those limitations are what recent posters have hit. Changes are likely needed elsewhere to fix aria2 deficient HTTP Range handling.
I had the issue with vanilla aria2 (1.36.0-1) from Ubuntu 22.04. After applying the patch #1869 and rebuilding the package, I was able to get 100+GB zip file without errors (with 10 threads and 1MB chunks). Can anyone try the patch? Maybe with few more confirmations the PR will get merged eventually...
If anyone needs the windows build for this patch, I built it using github actions. Download it at the artifacts section.
I have been using aria2 to download several large files (90-18Mb) from a server, and every time a few files return an error like the one below:
The command issued to download is:
I have tried re-downloading the files with error by limiting aria2c to a single connection and segment (-x 1 -s 1 -j 1), have tried using http instead of https, and have enabled / disabled -c and --auto-file-renaming=false, to no avail.
I am attaching two txt files, one with the pided terminal output and the other a logfile generated by aria2. My script loops over three different aria2c calls with different server API parameters, and the log file pertains to the first call. But the errors happen in every call, for a few files. Bear in mind that this was a re-run of the script, so aria skipped over the complete files, but was still not able to finish downloading the files that returned an error.
Any help is appreciated. Thank you.
output.txt log.txt