search

derniercri / snatch

A simple, fast and interruptable download accelerator, written in Rust
MIT License
676 stars 37 forks source link

Cannot download certain files #72

Closed equal-l2 closed 7 years ago

equal-l2 commented 7 years ago

Snatch built from devel HEAD (fac87df) cannot download these files even with -t1.

Output:

# Waiting a response from the remote server... OK !
# Remote content length: 384.57 KB
Downloading 1 chunks:
Chunk 0 ▎ |##############################################################################################################################################| 100.00 % 352.65 KB/s
thread '<unnamed>' panicked at 'Cannot download the chunk 0, due to error Invalid Status provided', src/download.rs:167
note: Run with `RUST_BACKTRACE=1` for a backtrace.
Done! Your download is available in 0.1.1.zip
Jean-Serge commented 7 years ago

Hello 😄

Which version of rust are you using ?

There is no issue from my side to download those files with rust 1.19.

equal-l2 commented 7 years ago

I've used stable (1.17) for build.

I'll try beta and nightly now.

equal-l2 commented 7 years ago

The error still occurs with beta(0.18-beta.3) and nightly(1.19.0-nightly 5f3966864).

I'm running this on Mac. Can it be a problem?

Jean-Serge commented 7 years ago

No I've just tried on a Mac with nightly without any problem. 🤔 Very strange

k0pernicus commented 7 years ago

Got the same problem of @equal-l2

This is my stack trace:

wpp00031:snatch antonin$ snatch https://github.com/derniercri/snatch/archive/0.1.1.zip -t 1
# Waiting a response from the remote server... OK !
# Remote content length: 384.57 KB
Downloading 1 chunks:
Chunk 0 ▎ |############################################################################################################| 100.00 % 472.07 KB/s
thread '<unnamed>' panicked at 'Cannot download the chunk 0, due to error Invalid Status provided', src/download.rs:167
note: Run with `RUST_BACKTRACE=1` for a backtrace.
Done! Your download is available in 0.1.1.zip

Also, the output file has to be deleted if the download was not ok.

Jean-Serge commented 7 years ago

Could you both try by putting the -t 1 option before URL ?

equal-l2 commented 7 years ago

The problem will occur without -t 1, at least for me.

Jean-Serge commented 7 years ago

I am only able to reproduce this bug when putting the -t 1 at the end of the command. This bug become stranger...

k0pernicus commented 7 years ago

Same thing with @equal-l2

Jean-Serge commented 7 years ago

@equal-l2 Can you tell me :

equal-l2 commented 7 years ago

@Jean-Serge

do you have the same issue using -t 2 ?

Yes.

how many thread does it use without this option ?

4 threads by default.

Jean-Serge commented 7 years ago

Can you send me the trace when using the -t 2 option ?

equal-l2 commented 7 years ago

Here.

# Waiting a response from the remote server... OK !
[WARNING] Cannot get the remote content length, using an HEADER request.
[WARNING] Trying to send an HTTP request, to get the remote content length...
Response { status: Ok, headers: Headers { Date: Fri, 26 May 2017 14:05:23 GMT
, Content-Type: application/zip
, Content-Length: 393798
, Access-Control-Allow-Origin: https://render.githubusercontent.com
, Content-Security-Policy: default-src 'none'; style-src 'unsafe-inline'
, Strict-Transport-Security: max-age=31536000
, Vary: Authorization,Accept-Encoding
Vary: Accept-Encoding
, X-Content-Type-Options: nosniff
, X-Frame-Options: deny
, X-XSS-Protection: 1; mode=block
, ETag: "c77b663f454ad7d29fd74de937d48c11ce794846"
, Content-Disposition: attachment; filename=snatch-0.1.1.zip
, X-Geo-Block-List:
, X-GitHub-Request-Id: DB47:24692:4A2E:84D2:59283623
, }, version: Http11, url: "https://codeload.github.com/derniercri/snatch/zip/0.1.1", status_raw: RawStatus(200, "OK"), message: Http11Message { is_proxied: false, method: None, stream: Wrapper { obj: Some(Reading(SizedReader(remaining=393798))) } } }
# Remote content length: 384.57 KB
Downloading 2 chunks:
0 / 196899 [---------------------------------------------------] 0.00 % 0.00/s
Chunk 1 ▎ |###########################################| 100.00 % 125.31 KB/s
thread '<unnamed>' panicked at 'Cannot download the chunk 1, due to error Invalid Status provided', src/download.rs:167
stack backtrace:
   0:        0x10ecd9b73 - std::sys::imp::backtrace::tracing::imp::unwind_backtrace::h884a721e113c3303
   1:        0x10ecdbec8 - std::panicking::default_hook::{{closure}}::h7a7d734b2824d103
   2:        0x10ecdbb0c - std::panicking::default_hook::h3eb11bd6cbfdc331
   3:        0x10ecde317 - std::panicking::rust_panic_with_hook::h8b9b25777425677b
   4:        0x10ecde1b4 - std::panicking::begin_panic::h69fe06a9d67b6165
   5:        0x10ecde122 - std::panicking::begin_panic_fmt::haf8695bfb6c52436
   6:        0x10e9a5835 - libsnatch::download::download_chunks::{{closure}}::hba634e8d9a54e7f0
   7:        0x10e96e84a - std::sys_common::backtrace::__rust_begin_short_backtrace::h9aa379327fb9c20e
   8:        0x10e9724d3 - std::thread::Builder::spawn::{{closure}}::{{closure}}::h26e5f94d0d3f8281
   9:        0x10e9a122a - <std::panic::AssertUnwindSafe<F> as core::ops::FnOnce<()>>::call_once::hb26a4adcb8ed6a76
  10:        0x10e9728a9 - std::panicking::try::do_call::h995fb58d60f03586
  11:        0x10ecdf44a - __rust_maybe_catch_panic
  12:        0x10e9726dc - std::panicking::try::h9d5ab9499fa7bb66
  13:        0x10e971795 - std::panic::catch_unwind::h8a7792b5472127dd
  14:        0x10e9722db - std::thread::Builder::spawn::{{closure}}::hdbd2a9b8fdfc6dc3
  15:        0x10e98c4e3 - <F as alloc::boxed::FnBox<A>>::call_box::h711bf8f596d161c0
  16:        0x10ecdb4f5 - std::sys::imp::thread::Thread::new::thread_start::h993b7e5cb7871f66
  17:     0x7fffab43f93a - _pthread_body
Downloading 2 chunks: 86 - _pthread_start
Chunk 0 ▎ |###########################################| 100.00 % 113.60 KB/s  liChunk 1 ▎ |###########################################| 100.00 % 125.31 KB/s
stack backtrace:
   0:        0x10ecd9b73 - std::sys::imp::backtrace::tracing::imp::unwind_backtrace::h884a721e113c3303
   1:        0x10ecdbec8 - std::panicking::default_hook::{{closure}}::h7a7d734b2824d103
   2:        0x10ecdbb0c - std::panicking::default_hook::h3eb11bd6cbfdc331
   3:        0x10ecde317 - std::panicking::rust_panic_with_hook::h8b9b25777425677b
   4:        0x10ecde1b4 - std::panicking::begin_panic::h69fe06a9d67b6165
   5:        0x10ecde122 - std::panicking::begin_panic_fmt::haf8695bfb6c52436
   6:        0x10e9a5835 - libsnatch::download::download_chunks::{{closure}}::hba634e8d9a54e7f0
   7:        0x10e96e84a - std::sys_common::backtrace::__rust_begin_short_backtrace::h9aa379327fb9c20e
   8:        0x10e9724d3 - std::thread::Builder::spawn::{{closure}}::{{closure}}::h26e5f94d0d3f8281
   9:        0x10e9a122a - <std::panic::AssertUnwindSafe<F> as core::ops::FnOnce<()>>::call_once::hb26a4adcb8ed6a76
  10:        0x10e9728a9 - std::panicking::try::do_call::h995fb58d60f03586
  11:        0x10ecdf44a - __rust_maybe_catch_panic
  12:        0x10e9726dc - std::panicking::try::h9d5ab9499fa7bb66
  13:        0x10e971795 - std::panic::catch_unwind::h8a7792b5472127dd
  14:        0x10e9722db - std::thread::Builder::spawn::{{closure}}::hdbd2a9b8fdfc6dc3
  15:        0x10e98c4e3 - <F as alloc::boxed::FnBox<A>>::call_box::h711bf8f596d161c0
  16:        0x10ecdb4f5 - std::sys::imp::thread::Thread::new::thread_start::h993b7e5cb7871f66
  17:     0x7fffab43f93a - _pthread_body
  18:     0x7fffab43f886 - _pthread_start
Done! Your download is available in 0.1.1.zip
equal-l2 commented 7 years ago

I've finally noticed these URLs don't support split download.

k0pernicus commented 7 years ago

Hello @equal-l2, I solved this issue few days ago actually - I forgot to describe the issue, sorry... I fixed the issue on my personal branch - a few refactoring and I will push a solution to download a file using only one thread, and a better support for servers that don't support the split behaviour.

equal-l2 commented 7 years ago

@k0pernicus I'm glad to hear that, thank you!

k0pernicus commented 7 years ago

No problem, I will push the code this week-end ;-)

k0pernicus commented 7 years ago

Hello @equal-l2, the solution has been merged (commit #73) ;-) You can now update your local project to download your files - a changelog has been added in the README. Please to close the issue if the solution is OK for you!

equal-l2 commented 7 years ago

It works perfectly now. Thank you again, @k0pernicus !