search

effigies / BitTornado

UNMAINTAINED - John Hoffman's fork of the original bittorrent
Other
148 stars 45 forks source link

can not share data with the same btdownloadheadless.py client each other. #23

Open zlhgo opened 8 years ago

zlhgo commented 8 years ago

i am using the master branch on python 3.5

my torrent file is:

root@ubuntu-server:/tmp# btshowmetainfo.py debian-8.4.0-openstack-amd64.qcow2.torrent 
btshowmetainfo 20130326 - decode BitTorrent metainfo files

metainfo file.: debian-8.4.0-openstack-amd64.qcow2.torrent
info hash.....: 2ad139893ce44cbe94e93522d4563df5f6e0aabe
file name.....: debian-8.4.0-openstack-amd64.qcow2
file size.....: 481034240 (1835 * 262144 + 0)
announce url..: http://172.20.4.22:6969/announce

start a tracker server on http://172.20.4.22:6969/announce

start a bt client on the tracker server as seeder server.

~# btdownloadheadless.py debian-8.4.0-openstack-amd64.qcow2.torrent

saving:         debian-8.4.0-openstack-amd64.qcow2 (458.8 MB)
percent done:   0.0
time left:      Download Succeeded!
download to:    /home/zlh/ISO/debian-8.4.0-openstack-amd64.qcow2
download rate:  
upload rate:    0.0 kB/s
share rating:   0.000   (0.0 MB up / 0.0 MB down)
seed status:    0 seen recently, plus 0.000 distributed copies
peer status:    0 seen now, 0.0% done at 0.0 kB/s

start a another bt client on another computer, but they can not share data with each other.

~$ btdownloadheadless.py debian-8.4.0-openstack-amd64.qcow2.torrent
saving:         debian-8.4.0-openstack-amd64.qcow2 (458.8 MB)
percent done:   0.0
time left:      
download to:    /tmp/debian-8.4.0-openstack-amd64.qcow2
download rate:  0.0 kB/s
upload rate:    0.0 kB/s
share rating:   0.000   (0.0 MB up / 0.0 MB down)
seed status:    0 seen now, plus 0.000 distributed copies
peer status:    0 seen now, 0.0% done at 0.0 kB/s

but: if i use another btclient as the seeder server or client, they can share data with btdownloadheadless.py. i have tested the original python-bittorrent and deluge.

thanks,effigies, do you have the same problem on you computer?

effigies commented 8 years ago

Have you tested using the original BitTorrent (or BitTornado v0.3.18) on both ends of the connection? My experience is with very small swarms (mostly tried with n = 2), it will work, but for some reason it takes a while to initiate the connections, and that this is generally true with most clients.

I haven't tried with multiple trackers, so it's possible that there's a problem with bttrack sending on a full list of peers on small swarms.

zlhgo commented 8 years ago

BitTornado v0.3.18 has the same problem.

zlhgo commented 8 years ago

my OS is debian8, i have installed BitTorrent using apt-get install python-bittorrent, i don't k know whether the python-bittorrent is what you are saying the original BitTorrent. python-bittorrent works fine.

effigies commented 8 years ago

To be clear, you're running a server that has a tracker and an initial seeder, and a client that has a leecher? And these are the situations you've described?

Case Tracker Seeder Leecher Works?
1 BitTornado (master) BitTornado (master) BitTornado (master) No
2 BitTornado (master) BitTornado (master) BitTornado (v0.3.18) No
3 BitTornado (master) BitTornado (v0.3.18) BitTornado (v0.3.18) No
4 BitTornado (master) python-bittorrent python-bittorrent No
5 python-bittorrent python-bittorrent python-bittorrent Yes

Assuming this is correct:

What we need to know is whether the tracker works as expected, whether the client works as expected, and whether the bug was introduced between BitTorrent and BitTornado or between BitTornado v0.3.18 and master.

Case 5 indicates we can use python-bittorrent (yes, this is the original bittorrent, though possibly with Debian-introduced patches) as a benchmark. Can you test the following cases?

Case Tracker Seeder Leecher
6 BitTornado (v0.3.18) python-bittorrent python-bittorrent
7 python-bittorrent BitTornado (v0.3.18) python-bittorrent
8 python-bittorrent python-bittorrent BitTornado (v0.3.18)
9 python-bittorrent BitTornado (v0.3.18) BitTornado (v0.3.18)
10 python-bittorrent BitTornado (master) python-bittorrent
11 python-bittorrent python-bittorrent BitTornado (master)
12 python-bittorrent BitTornado (master) BitTornado (master)

Comparing case 6 and 4 will help determine where the tracker bug was introduced. Cases 7-9 will test the old BitTornado clients in different positions, and cases 10-12 will test the current clients.

What would be ideal in all of these cases is if you could give them up to ten minutes to connect, and estimate how long they take to connect. e.g. < 1min, > 10min (failure).

Sorry to push this work on you, but I don't have a ton of time, and knowing the conditions this fails in will help reduce the time it takes me to figure this out.

zlhgo commented 8 years ago

hi, effigies, i have tested all the situations.

Case Tracker Seeder Leecher Works? Comment
1 BitTornado (master) BitTornado (master) BitTornado (master) No
2 BitTornado (master) BitTornado (master) BitTornado (v0.3.18) No
3 BitTornado (master) BitTornado (v0.3.18) BitTornado (v0.3.18) No
4 BitTornado (master) python-bittorrent python-bittorrent Yes
6 BitTornado (v0.3.18) python-bittorrent python-bittorrent Yes Exception, See below
7 python-bittorrent BitTornado (v0.3.18) python-bittorrent Yes
8 python-bittorrent python-bittorrent BitTornado (v0.3.18) Yes Exception, See below
9 python-bittorrent BitTornado (v0.3.18) BitTornado (v0.3.18) Not Completed Download 97.1% ,then stopped, whether the seeder and leecher on same computer or different computer
10 python-bittorrent BitTornado (master) python-bittorrent Yes
11 python-bittorrent python-bittorrent BitTornado (master) Yes
12 python-bittorrent BitTornado (master) BitTornado (master) Unstable seeder and leecher on the same computer can download all the file,otherwise the download speed is very slow and can just download a bit,then stopped

all the exceptions are occurred after the download is completed: Case 6 tracker:

Traceback (most recent call last):
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/RawServer.py", line 142, in listen_forever
    self.sockethandler.handle_events(events)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/SocketHandler.py", line 319, in handle_events
    s.handler.data_came_in(s, data)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/HTTPHandler.py", line 155, in data_came_in
    if not c.data_came_in(data) and not c.closed:
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/HTTPHandler.py", line 46, in data_came_in
    self.next_func = self.next_func(val)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/HTTPHandler.py", line 78, in read_header
    r = self.handler.getfunc(self, self.path, self.headers)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/track.py", line 915, in get
    rsize = self.add_data(infohash, event, ip, paramslist)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/track.py", line 690, in add_data
    bc[1][myid] = bc[0][myid]
KeyError: 'M3-4-2--337c30979658'

Case 6 leecher:

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 801, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 754, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib/python2.7/dist-packages/BitTorrent/Rerequester.py", line 84, in rerequest
    h = urlopen(url)
  File "/usr/lib/python2.7/urllib2.py", line 154, in urlopen
    return opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 429, in open
    response = self._open(req, data)
  File "/usr/lib/python2.7/urllib2.py", line 447, in _open
    '_open', req)
  File "/usr/lib/python2.7/urllib2.py", line 407, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/dist-packages/BitTorrent/zurllib.py", line 33, in http_open
    fp = HTTPHandler.http_open(self,req)
  File "/usr/lib/python2.7/urllib2.py", line 1225, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/usr/lib/python2.7/urllib2.py", line 1198, in do_open
    r = h.getresponse(buffering=True)
  File "/usr/lib/python2.7/httplib.py", line 1136, in getresponse
    response.begin()
  File "/usr/lib/python2.7/httplib.py", line 453, in begin
    version, status, reason = self._read_status()
  File "/usr/lib/python2.7/httplib.py", line 417, in _read_status
    raise BadStatusLine(line)
BadStatusLine: ''

Case 8 leecher:

These errors occurred during execution:
[10:52:52] Traceback (most recent call last):
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/RawServer.py", line 142, in listen_forever
    self.sockethandler.handle_events(events)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/SocketHandler.py", line 319, in handle_events
    s.handler.data_came_in(s, data)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Encrypter.py", line 403, in data_came_in
    self.read(s)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Encrypter.py", line 488, in _read2
    x = self.next_func(m)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Encrypter.py", line 369, in read_message
    self.connecter.got_message(self, s)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Connecter.py", line 325, in got_message
    if c.download.got_piece(i, toint(message[5:9]), message[9:]):
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Downloader.py", line 182, in got_piece
    self._request_more()
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Downloader.py", line 189, in _request_more
    self.fix_download_endgame(new_unchoke)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Downloader.py", line 256, in fix_download_endgame
    del want[self.backlog - len(self.active_requests):]
TypeError: slice indices must be integers or None or have an __index__ method

[10:52:52] Traceback (most recent call last):
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/RawServer.py", line 142, in listen_forever
    self.sockethandler.handle_events(events)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/SocketHandler.py", line 319, in handle_events
    s.handler.data_came_in(s, data)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Encrypter.py", line 403, in data_came_in
    self.read(s)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Encrypter.py", line 493, in _read2
    self.close()
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Encrypter.py", line 382, in close
    self.sever()
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Encrypter.py", line 391, in sever
    self.connecter.connection_lost(self)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Connecter.py", line 222, in connection_lost
    c.download.disconnected()
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Downloader.py", line 88, in disconnected
    self._letgo()
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/Downloader.py", line 101, in _letgo
    self.downloader.storage.request_lost(index, begin, length)
  File "/home/zlh/Download/BitTornado-0.3.18/BitTornado/BT1/StorageWrapper.py", line 738, in request_lost
    assert not (begin, length) in self.inactive_requests[index]
AssertionError
effigies commented 8 years ago

Thanks for the info!

4 suggests that the current tracker is okay, at least sometimes, so I do think this is a client problem.

From 9 and 12, it looks like there were some issues that either emerged in the BitTornado fork from BitTorrent, or that were fixed in BitTorrent since the fork. It's also possible that some changes occurred to the libraries we depend on since then.

I'll look into this as I find time. It might be a bit as I have a lot going on right now, but hopefully I can get to it soon. If you're inspired to do some digging and want to submit a PR, I'd be glad to review it. In any case, thanks again for your work.

effigies commented 8 years ago

From playing with the last few issues/PRs, I think I've established that BitTornado clients are having trouble seeding. I can seed with rtorrent and download with btdownload{headless,curses}, but not vice-versa.

Hope to make some progress on this, soon.

(Tagging in @yang502.)

WuYiP commented 8 years ago

I also have faced the same problem about "can not share data with the same btdownloadheadless.py client each other in master branch",for that , I hope you can solve it soon ,thank you,effigies!

zlhgo commented 7 years ago

the latest code. 

./btdownloadheadless.py --security 0 --crypto_allowed 0  file.torrent

with https://github.com/feross/bittorrent-tracker

bittorrent-tracker --port 6969

can share data with the same btdownloadheadless.py client on same computer.

if use

 ./bttrack.py --dfile /tmp/bt_state --port 6969

the btdownloadheadless.py can not get peers from bttrack.py

so: maybe bttrack.py has a bit bug. :-) and maybe the BitTornado do not support crypt.

effigies commented 7 years ago

Thanks for the report. I'll have a look when I get a chance (not likely before February). Figuring out issues in bttrack is what #50 is aimed at, so hopefully once I get back to this project I'll have the tools to get to the bugs.

And yes, the crypto in BitTornado might not have translated cleanly in the conversion to Python 3.

Ademord commented 7 years ago

Has there any progress related to this? I am experiencing the same issue here and I would like to get bittornado running. Or would you suggest another way to set up a swarm? I tried out a chihaya tracker + bittornado but it gives me bad encoded data and the swarm dies after ~1min, with error TypeError BadConnection comparison (files over 100 MB), so I'm a bit stuck...

effigies commented 7 years ago

No. Sorry, I have way too much going on to put any effort into this repo, at this point. I can try to review PRs, if someone's interested in pushing forward, but the hold-up on #50 is that I don't have time to re-read and test it before merging.