search

torproject / stem

Python controller library for Tor
https://stem.torproject.org/
GNU Lesser General Public License v3.0
280 stars 76 forks source link

or_ports = controller.get_ports(Listener.OR) returns an empty list if the ORport is specified with an ip address #75

Closed toralf closed 4 years ago

toralf commented 4 years ago

With config values like

Nickname zwiebeltoralf
Address 5.9.158.75
OutboundBindAddress 5.9.158.75
OutboundBindAddress [2a01:4f8:190:514a::2]
DirPort 5.9.158.75:80
ORPort  5.9.158.75:443
DirPort [2a01:4f8:190:514a::2]:80   NoAdvertise
ORPort  [2a01:4f8:190:514a::2]:443

ControlPort 9051

I run into that issue. A circumvention is

  or_ports = controller.get_ports(Listener.OR)
  if not or_ports:
    or_ports = [ 443 ] if control_port == 9051 else [ 9001 ]
    print("warn: have to guess OR port(s):" + str(or_ports))

in my script (derived mostly from your example /usr/share/doc/stem-1.8.0/_static/example/relay_connections.py , my is here: https://github.com/toralf/torutils/blob/master/info.py)

atagar commented 4 years ago

Hi toralf. What does the following provide?

print(controller.get_info('net/listeners/or'))
atagar commented 4 years ago

Oh, and please run against stem's present master branch. Otherwise the results will be confusing. Likely the issue is that your ORPort is IPv6 and you don't have the patch added in ticket 74.

toralf commented 4 years ago

Hi toralf. What does the following provide?

print(controller.get_info('net/listeners/or'))

It returns

"5.9.158.75:443" "[2a01:4f8:190:514a::2]:443"

and

"5.9.158.75:9001" "[2a01:4f8:190:514a::2]:9001"

respectively.

toralf commented 4 years ago

Oh, and please run against stem's present master branch. Otherwise the results will be confusing. Likely the issue is that your ORPort is IPv6 and you don't have the patch added in ticket 74. Ok, I cloned the repo as root in $HOME and set

export PYTHONPATH=/root/stem/stem/util

and get then:

mr-fox ~/stem # python /usr/share/doc/stem-1.8.0/_static/example/relay_connections.py  --ctrlport 29051 --resolver lsof
Traceback (most recent call last):
File "/usr/share/doc/stem-1.8.0/_static/example/relay_connections.py", line 1, in <module>
import argparse
File "/usr/lib/python3.7/argparse.py", line 87, in <module>
import re as _re
File "/usr/lib/python3.7/re.py", line 124, in <module>
import enum
File "/root/stem/stem/util/enum.py", line 43, in <module>
from typing import Any, Iterator, List, Sequence, Tuple, Union
File "/usr/lib/python3.7/typing.py", line 1642, in <module>
Pattern = _alias(stdlib_re.Pattern, AnyStr)
AttributeError: module 're' has no attribute 'Pattern'
atagar commented 4 years ago

AttributeError: module 're' has no attribute 'Pattern'

Hi toralf. Unfortunately that's a problem within your python interpreter. I did a quick search for an upstream ticket but no luck. Unfortunately I won't be able to help you with that.

toralf commented 4 years ago

AttributeError: module 're' has no attribute 'Pattern'

Hi toralf. Unfortunately that's a problem within your python interpreter. I did a quick search for an upstream ticket but no luck. Unfortunately I won't be able to help you with that.

Well, it is a stem+Python3 issue, the reproducer is:

git clone https://github.com/torproject/stem.git && cd stem/stem/util'
Cloning into 'stem'...
remote: Enumerating objects: 191, done.
remote: Counting objects: 100% (191/191), done.
remote: Compressing objects: 100% (170/170), done.
remote: Total 30658 (delta 82), reused 129 (delta 20), pack-reused 30467
Receiving objects: 100% (30658/30658), 8.58 MiB | 3.93 MiB/s, done.
Resolving deltas: 100% (24369/24369), done.

mr-fox ~/stem/stem/util # eselect python list
Available Python interpreters, in order of preference:
  [1]   python3.7
  [2]   python2.7
  [3]   python3.8

mr-fox ~/stem/stem/util # eselect python set python2.7

mr-fox ~/stem/stem/util # python -c 'import argparse'

mr-fox ~/stem/stem/util # eselect python set python3.7

mr-fox ~/stem/stem/util # python -c 'import argparse'

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.7/argparse.py", line 87, in <module>
    import re as _re
  File "/usr/lib/python3.7/re.py", line 124, in <module>
    import enum
  File "/root/stem/stem/util/enum.py", line 43, in <module>
    from typing import Any, Iterator, List, Sequence, Tuple, Union
  File "/usr/lib/python3.7/typing.py", line 1642, in <module>
    Pattern = _alias(stdlib_re.Pattern, AnyStr)
AttributeError: module 're' has no attribute 'Pattern'

mr-fox ~/stem/stem/util # eselect python set python3.8

mr-fox ~/stem/stem/util # python -c 'import argparse'
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/usr/lib/python3.8/argparse.py", line 88, in <module>
    import re as _re
  File "/usr/lib/python3.8/re.py", line 124, in <module>
    import enum
  File "/root/stem/stem/util/enum.py", line 43, in <module>
    from typing import Any, Iterator, List, Sequence, Tuple, Union
  File "/usr/lib/python3.8/typing.py", line 1998, in <module>
    Pattern = _alias(stdlib_re.Pattern, AnyStr)
AttributeError: partially initialized module 're' has no attribute 'Pattern' (most likely due to a circular import)
toralf commented 4 years ago

And from another Gentoo dev: "<@sultan> toralf: enum.py in stem is the problem"

atagar commented 4 years ago

Oh, I see the issue. Don't 'cd' into the util directory, instead just use 'cd stem'. The above is breaking because re.py is confusing stem's enum module with the builtin.

You will probably still get an error (iirc I recall seeing an email concerning an iteration error) - that's the real error, not this.

toralf commented 4 years ago

Oh, I see the issue. Don't 'cd' into the util directory, instead just use 'cd stem'.

Indeed, now I do not get an Python error.

You will probably still get an error (iirc I recall seeing an email concerning an iteration error) - that's the real error, not this. Yep, it still gives:


mr-fox ~/stem # python /usr/share/doc/stem-1.8.0/_static/example/relay_connections.py  --ctrlport 29051 --resolver ss
0.4.5.0-alpha-dev   uptime: 1-08:51:39   flags: Fast, Guard, Running, Stable, V2Dir, Valid

Traceback (most recent call last): File "/usr/share/doc/stem-1.8.0/_static/example/relay_connections.py", line 130, in main() File "/usr/share/doc/stem-1.8.0/_static/example/relay_connections.py", line 66, in main for conn in get_connections(resolver = args.resolver, process_pid = pid): File "/usr/lib/python3.7/site-packages/stem/util/connection.py", line 348, in get_connections raise IOError('No results found using: %s' % resolver_command) OSError: No results found using: ss -nptu

despite

mr-fox ~/stem # ss -nptu | wc 17007 119024 2636085 mr-fox ~/stem # ss -nptu | head Netid State Recv-Q Send-Q Local Address:Port Peer Address:Port Process
tcp ESTAB 0 0 5.9.158.75:41637 107.150.36.162:443 users:(("tor",pid=2265,fd=1069))
tcp ESTAB 0 0 5.9.158.75:45005 199.249.230.110:443 users:(("tor",pid=2265,fd=3171))
...

toralf commented 4 years ago

And this is the real error now with HAED::

mr-fox ~ # export PYTHONPATH=/root/stem
mr-fox ~ # python /usr/share/doc/stem-1.8.0/_static/example/relay_connections.py  --ctrlport 29051 
 0.4.5.0-alpha-dev   uptime: 1-08:55:33   flags: Fast, Guard, Running, Stable, V2Dir, Valid

Traceback (most recent call last):
  File "/usr/share/doc/stem-1.8.0/_static/example/relay_connections.py", line 130, in <module>
    main()
  File "/usr/share/doc/stem-1.8.0/_static/example/relay_connections.py", line 50, in main
    for desc in controller.get_network_statuses():
TypeError: 'async_generator' object is not iterable
atagar commented 4 years ago

An email discussion between toralf and me led to test coverage of all our example code, as well as a fix for the async_generator stacktrace.

https://gitweb.torproject.org/stem.git/commit/?id=26bf1bb

toralf commented 4 years ago

I uninstalled Nyx and stem here at a hardened Gentoo Linux. With Tor version 0.4.5.0-alpha-dev (git-228ac47c2cc2625e) and Python 3. or /3.8 even latest Git's stem unable to derive the ORport 

mr-fox ~ # python 
Python 3.7.8 (default, Jul 22 2020, 16:33:41) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from stem.connection import connect
>>> from stem.control import Listener
>>> controller = connect(control_port = ('127.0.0.1', 9051))
>>> controller.get_ports(Listener.OR, [])
set()
>>> quit()

And your example shows it too (no inbound):

mr-fox ~ # python /root/stem/docs/_static/example/relay_connections.py
 0.4.5.0-alpha-dev   uptime: 6-09:24:53   flags: Fast, Guard, HSDir, Running, Stable, V2Dir, Valid

+------------------------------+------+------+
| Type                         | IPv4 | IPv6 |
+------------------------------+------+------+
| Inbound to our ControlPort   |    1 |    0 |
| Outbound to a relay          | 3688 |    0 |
| Outbound exit traffic        |   58 |    3 |
| Outbound uncategorized       | 4462 |  577 |
+------------------------------+------+------+
| Total                        | 8209 |  580 |
+------------------------------+------+------+

+------------------------------+------+------+
| Exit Port                    | IPv4 | IPv6 |
+------------------------------+------+------+
| 53 (DNS)                     |    0 |    1 |
| 853                          |    6 |    0 |
| 2083 (radsec)                |    5 |    0 |
| 5222 (Jabber)                |   34 |    2 |
| 5223 (Jabber)                |    7 |    0 |
| 5280                         |    2 |    0 |
| 6667 (IRC)                   |    3 |    0 |
| 7777                         |    1 |    0 |
+------------------------------+------+------+
| Total                        |   58 |    3 |
+------------------------------+------+------+

BTW I derived my scritp from that but I try to catch or_ports onyl once isntead for each of my ~8,000 connections to save runtime:

  or_ports = controller.get_ports(Listener.OR, [])
  if not or_ports:
    or_ports = { 443 } if control_port == 9051 else { 9001 }
    print('warn: have to guess OR port(s): ' + str(or_ports))

  for conn in get_connections(resolver = args.resolver, process_pid = pid):
    if conn.protocol == 'udp':
        continue

    if conn.local_port in or_ports:
...
atagar commented 4 years ago

Hi toralf, what is your torrc's ORPort line? On this ticket you said that GETINFO net/listeners/or provides...

5.9.158.75:443 [2a01:4f8:190:514a::2]:443

Since these aren't explicitly localhost get_ports() drops them, as documented. Please use get_listeners() instead.

toralf commented 4 years ago

Hi toralf, what is your torrc's ORPort line? On this ticket you said that GETINFO net/listeners/or provides...

 cat /etc/tor/torrc
# torrc
#
PIDFile /var/run/tor/tor.pid
DataDirectory /var/lib/tor/data

Nickname zwiebeltoralf
Address 5.9.158.75
OutboundBindAddress 5.9.158.75
OutboundBindAddress [2a01:4f8:190:514a::2]
DirPort 5.9.158.75:80
ORPort  5.9.158.75:443
DirPort [2a01:4f8:190:514a::2]:80   NoAdvertise
ORPort  [2a01:4f8:190:514a::2]:443

ControlPort 9051

Log notice file /var/log/tor/notice.log
Log warn   file /var/log/tor/warn.log

%include /etc/tor/torrc.d/

Since these aren't explicitly localhost get_ports() drops them, as documented. Please use get_listeners() instead. Yep, that works better. BTW, it seems, that 

if conn.local_port in controller.get_ports(Listener.OR, []):

is much faster if using a temp variable for "controller.get_ports(Listener.OR, [])"