webseed unable to finish download

[GoogleCodeExporter]

What steps will reproduce the problem?
Create .torrent with non paded small sized files. Use only webseed (no 
announcers or DHT).

What is the expected output?
libTorrent unable to complete download (stuck at same %)

What version of the product are you using? On what operating system?
windows, trunk r9993

Please provide any additional information below.

attachment contains test .torrent and .log output for 
"client_test.exe -H -X -P 127.0.0.1:8888 testbin.torrent" (proxy_type==http)
.log contains infinit requests for same byte range.

https://www.dropbox.com/s/7acksh6tdx26onc/testbin.torrent
https://www.dropbox.com/s/mrcdutmkevg5lm0/%5B0.0.0.0%230%5D-%5B------%2380%5D.lo
g

Original issue reported on code.google.com by ajax16...@gmail.com on 9 Jun 2014 at 3:22

[GoogleCodeExporter]

this is a known issue. the web seed connection objects do not store partial 
blocks across sessions/connections.
If the web server does not support range requests (or perhaps in this case, the 
http proxy doesn't) AND there are blocks that span multiple files, those blocks 
cannot be downloaded from the web server.

One potential fix for this is for web seed connection objects to re-construct 
themselves immediately when they are closed, and pass along the partial piece 
they have. Or make the underlying bittorrent engine support storing partial 
pieces for other peer connections (including web seed connections) to pick up.

any idea of how to fix this without making it too complicated, or a patch to 
fix it, would be highly welcome!

Original comment by arvid.no...@gmail.com on 11 Jun 2014 at 4:37

[GoogleCodeExporter]

so all of possible faults:
- non keepalive capable webservers (intentional keepalive_disable or by 
useragent mask)
- useragent sensitive webservers. By default useragent was not send by 
libTorrent for consequental http requests. (in my test case all requests 
without "useragent" were served with 403)
- request disconnected by webserver keepalive request limit (several small 
sized files resided in one piece request create too many GET per connection 
which leads to early disconnect)
- proxy without pipeline support (consequental requests were ignored)

here is patch :
https://www.dropbox.com/s/ll8t3jq8m5j4le7/web_seed_restart.patch

it use your suggestion to recreate web seed 

Original comment by ajax16...@gmail.com on 12 Jun 2014 at 6:52

[GoogleCodeExporter]

this patch breaks the web_seed unit test. I'll take a look to see if it's 
something simple.

Original comment by arvid.no...@gmail.com on 12 Jun 2014 at 11:17

[GoogleCodeExporter]

I don't really understand why this works for you. the restart data you save 
only contains a single request. I would imagine it should contain all remaining 
requests to complete that one block at least, or maybe the whole piece.

also, when restarting a web seed, there's nothing allocating those blocks in 
the piece picker, so someone else might download them. Also, it's not clear to 
me why you save restart data and disconnect the peer (with no error). I would 
have expected to save the restart data _when_ the connection is closed, not 
closing it pre-emptively.

I also don't understand why you inserted that if (m_first_request) condition 
when issuing requests.

Original comment by arvid.no...@gmail.com on 12 Jun 2014 at 11:48

[GoogleCodeExporter]

given that your web server rejects requests without user-agent, I'm inclined to 
remove this behavior and always send it btw.

Original comment by arvid.no...@gmail.com on 12 Jun 2014 at 11:53

[GoogleCodeExporter]

it completely disables multi file GET per same http connection:
- strictly one GET range of multi file request at time (disables pipelined 
requests, but more compatible)
- if on_receive moves to next file index then disconnect and save partial block 
so next reconnected webseed will skip to new file index range. if only web seed 
connection available for that torrent then piece picker will request same piece 
again.

i think the better way is to save restart_data at same condition and instead of 
disconnect try to "send_buffer" with new request and if only it fails 
disconnect. i did not get how to "send_buffer" properly in on_receive (it seems 
it's to late to send any data).

Original comment by ajax16...@gmail.com on 13 Jun 2014 at 7:28

[GoogleCodeExporter]

I've made some modifications to your patch. This is what I have so far 
(including a new unit test for no keepalive support). It still doesn't work 
though. There's some bug causing it to not resume correctly and produce a 
corrupt piece. There's some debug logging in there right now too.

I also think there are some simplification opportunities around the new struct. 
Since there's only one web_peer_connection at a time for each web_seed_entry, 
it might be possible to not heap allocate it but just add those fields directly 
to web_seed_entry.

http://dpaste.com/334CJYD

Original comment by arvid.no...@gmail.com on 13 Jun 2014 at 2:57

[GoogleCodeExporter]

quick tested your variant, and just this fix:
        if (!m_requests.empty() && !m_file_requests.empty()
//          && m_parser.connection_close()
            && !m_piece.empty())

in case of wrong proxy connection_close are not set

Original comment by ajax16...@gmail.com on 13 Jun 2014 at 7:34

[GoogleCodeExporter]

I'm pretty confident in this patch. Could you please try it? all the unit tests 
pass with it.

http://dpaste.com/3824DKQ

Original comment by arvid.no...@gmail.com on 15 Jun 2014 at 3:52

[GoogleCodeExporter]

my test case also passed with OK, thanks a lot

Original comment by ajax16...@gmail.com on 15 Jun 2014 at 10:42

[GoogleCodeExporter]

checked in to trunk.

Original comment by arvid.no...@gmail.com on 15 Jun 2014 at 7:31

• Changed state: Fixed