Closed jchrisweaver closed 4 years ago
Hi @jchrisweaver ,
The most likely scenario is that port 1900 is in use, either by a different service or by a previous instance of Thespian that didn't fully shutdown.
On the mac, try:
$ netstat -vatn | grep 1900
tcp4 0 0 *.1900 *.* LISTEN 131072 131072 404 0
$ ps -awfx | grep 404
0 404 1 0 5:38PM ?? 0:00:03 python
$
The LISTEN on port 1900 shows something is active, and the 2nd to last column is the PID, which I then display to see what it is. In this case, it corresponds to the Thespian instance I currently have running.
If you do have something on port 1900 and it's Thespian, you can try $ python -m thespian.director stop
or alternatively python -c 'from thespian.actors import *; ActorSystem("multiprocTCPBase").shutdown()'
to stop it, but if the previous version is hung for some reason, you may need to kill the process (possibly kill -9 PID
).
Let me know if this doesn't help and we can dig deeper.
Thanks for the quick response!
I rebooted just for good measure. ;) After that, I verified the port wasn't in use and fired up lsof -P -n -r 1 -i :1900
Following the same steps again, I got the following:
=======
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python3 2030 chris 9u IPv4 0x8e543c41ae576a07 0t0 TCP *:1900 (LISTEN)
python3 2031 chris 9u IPv4 0x8e543c41ae576a07 0t0 TCP *:1900 (LISTEN)
python3 2031 chris 13u IPv4 0x8e543c41a7d6cf87 0t0 TCP 10.34.10.6:49929->10.34.10.6:1900 (SYN_SENT)
=======
The 10.34.10.6
is notable since my local network is 10.1.2.*
After some poking around, I turned off my software VPN software, tried the test again, and the previously failing test succeeded and the lsof
showed the following:
=======
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python3 2568 chris 9u IPv4 0x8e543c41afe5aa07 0t0 TCP *:1900 (LISTEN)
python3 2568 chris 11u IPv4 0x8e543c41a3dea627 0t0 TCP 10.1.2.125:1900->10.1.2.125:50338 (ESTABLISHED)
python3 2569 chris 9u IPv4 0x8e543c41afe5aa07 0t0 TCP *:1900 (LISTEN)
python3 2569 chris 13u IPv4 0x8e543c41b16f7367 0t0 TCP 10.1.2.125:50338->10.1.2.125:1900 (ESTABLISHED)
Apparently, the code is grabbing the IP from the tunnel interface that the vpn sets up. Here's ifconfig
snippet showing the utun2:
utun2: flags=8051<UP,POINTOPOINT,RUNNING,MULTICAST> mtu 1500
inet 10.34.10.6 --> 10.34.10.5 netmask 0xffffffff
So I can move forward but obviously I'd prefer to have my vpn enabled. ;) Is there a way to set the interface used in Thespian?
That's very interesting.
On startup, Thespian attempts to determine the best address to use to allow communication with possibly remote Thespian Actor System instances. There is a startup process that attempts to determine the set of local addresses, and then find a good candidate public address (basically, which address can reach 8.8.8.8
and use that local public address for the Actors. While there are potential issues with this, it generally has worked pretty well, but it appears that the tunnel interface doesn't allow local communications on the local tunnel address.
I've created an override_addr
branch that will let you set a THESPIAN_BASE_IPADDR
environment variable before startup to override the local public address used. If you are not communicating externally then it should be fine to set this to 127.0.0.1
, although you could also use 10.1.2.125
.
Please give this branch a try and let me know if this fixes your issue. If so, I will add documentation for this and try to make a new Thespian release with a day or so.
To use this branch:
$ git clone -b override_addr https://github.com/kquick/thespian thespian_oa
$ export PYTHONPATH=$(pwd)/thespian_oa:$PYTHONPATH
$ export THESPIAN_BASE_IPADDR='127.0.0.1'
$ ... startup thespian or esrally ...
Hi @jchrisweaver , any luck validating the override_attr
branch?
Just to followup @jchrisweaver , I've just released version 3.10.0 of Thespian which includes the functionality described above (THESPIAN_BASE_IPADDR
).
I'm going to close this issue based on this release, but please feel free to re-open it (or open a new issue) if this has not resolved your issues.
I come by way of trying to get ESRally to run to test various Elasticsearch configs.
I'm unable to run the simple example in the docs. Any suggestions on how to debug would be greatly appreciated.
Full trace
Steps to reproduce
pip install thespian
Here's the additional network-related info you've asked for in other dicussions:
Thanks for any assistance.