port to ephemeral Tor hidden services / python-stem

Can you please port unMessage to use ephemeral Tor hidden services?

This basically means, use the Tor ControlProtocol to create the hidden service. Not using torrc modifications. Yes, ephemeral Tor hidden services can also persist between reboots. After creating it with add_onion new, you get the hidden service private key, can store it, and restore it later. This can be done using python-stem. - https://packages.debian.org/stretch/python3-stem From that perspective, ricochet would be a project similar unMessage. (https://github.com/ricochet-im/ricochet) Further, onionshare is also an example using Tor ephemeral services.

Wishlist:

Use python-stem since that is perfectly compatible and tested with https://github.com/Whonix/control-port-filter-python.
Don't send more than 1 Tor ControlPort command at once. I.e. do not request 'status/circuit-established status/bootstrap-phase, do request 'status/circuit-established followed by 'status/bootstrap-phase'. That makes our (and Tails also I believe so) control protocol filtering easier.
Please do not listen on any random local port. onionshare listens on 17609 - 16759. zeronet listens on 15441. Have a look at https://en.wikipedia.org/wiki/List_of_TCP_and_UDP_port_numbers and see if you can find some unused port or port range. A fixed or limited list of ports makes our documentation to open a port in the Whonix-Workstation a lot saner.

Can you please port unMessage to use ephemeral Tor hidden services?

Absolutely. Thanks for the links and suggestions! I will let you know how that goes.

Please do not listen on any random local port.

unMessage's hidden service listens to 50000 by default, but we allow the user to choose the ports they wish to use (and that makes testing easier on the same machine). The hidden service then maps that port to localhost:50000 (but that will be changed for Whonix as discussed in #2).

That port is actually from the "Dynamic, private or ephemeral ports" and that does not seem right. We'll probably pick an unused port between 1024~49151 then. I will let you know which one.

Thanks Patrick!

Yesterday I was finally able to work on this again. Here are two scripts I wrote based on an old txtorcon example for ephemeral hidden services. They serve a simple html page, but one uses stem and the other uses txtorcon. As unMessage uses Twisted and txtorcon, I really wanted to use it instead of stem. If it does not work though, I think we will probably use stem then.

Both scripts worked on a non-Whonix system and I was hoping that just changing the CONTROL_PORT and LISTEN_IP would make them work on Whonix, but didn't. txtorcon simply gets stuck and I get no error. stem fails with an IncorrectSocketType('unable to use the control socket') error. I tried using 9051 and 9151. Are those the correct ports? Also, I might not be running Whonix 14. I am using Qubes 3.2 with the default Whonix templates. I know it is a Debian jessie, but Whonix 14 should be stretch, right? Is there a command to check the Whonix version?

Thanks!

Yes, it requires Whonix 14, which is not released, but the next version.

If it works on localhost, it should work with Whonix 14 as well with no changes required. Otherwise we'd have to fix our control-port-filter-python and/or anon-ws-disable-stacked-tor packages.

Once you are done (using as many Tor ControlProtocol commands as needed and a testers version available), one needs to write a tor-controlport-filter-merger whitelist profile. (Similar to https://github.com/Whonix/control-port-filter-python/tree/master/usr/share/tor-controlport-filter-merger/examples.) (Probably not hard. I could do that.)

I know it is a Debian jessie, but Whonix 14 should be stretch, right?

Right.

On Mon, Feb 20, 2017 at 07:13:18AM -0800, Patrick Schleizer wrote:

Yes, it requires Whonix 14, which is not released, but the next version.

If it works on localhost, it should work with Whonix 14 as well with no changes required. Otherwise we'd have to fix our control-port-filter-python and/or anon-ws-disable-stacked-tor packages.

By "If it works on localhost", you mean "If it works on a non-Whonix system connecting to the control port on localhost"?

From my understanding the gateway's Tor process is accessible to the workstation without any authentication (or maybe with a password that is available in TOR_CONTROL_PASSWD) and then the Control Port Filter is responsible by allowing/denying commands.

Once you are done (using as many Tor ControlProtocol commands as needed and a testers version available), one needs to write a tor-controlport-filter-merger whitelist profile. (Similar to https://github.com/Whonix/control-port-filter-python/tree/master/usr/share/tor-controlport-filter-merger/examples.) (Probably not hard. I could do that.)

Is there a way to test it right now? Is there a dev Whonix version I could use to do that? Or just making sure I have an updated control port filter running is enough?

I would then listen on the IP returned by qubesdb-read /qubes-ip, connect to the port in TOR_CONTROL_PORT (on localhost), and create an ephemeral hidden service, mapped to that IP and to unMessage's public port. Is that right?

Thanks @adrelanos!

Felipe Dau:

By "If it works on localhost", you mean "If it works on a non-Whonix system connecting to the control port on localhost"?

Yes.

From my understanding the gateway's Tor process is accessible to the workstation without any authentication

Yes.

(or maybe with a password that is available in TOR_CONTROL_PASSWD) and then the Control Port Filter is responsible by allowing/denying commands.

Yes.

Once you are done (using as many Tor ControlProtocol commands as needed and a testers version available), one needs to write a tor-controlport-filter-merger whitelist profile. (Similar to https://github.com/Whonix/control-port-filter-python/tree/master/usr/share/tor-controlport-filter-merger/examples.) (Probably not hard. I could do that.)

Is there a way to test it right now?

That's a rather difficult process until it reaches testers-only quality.

https://www.whonix.org/wiki/Upgrading_Whonix_13_to_Whonix_14

Or just making sure I have an updated control port filter running is enough?

It's not possible to backport the filter to Whonix 13. Too many related changes.

I would then listen on the IP returned by qubesdb-read /qubes-ip,

Yes.

connect to the port in TOR_CONTROL_PORT (on localhost), and create an ephemeral hidden service, mapped to that IP and to unMessage's public port. Is that right?

That would work. Tor control protocol access is super simple and sorted out in Whonix 14.

System Tor's /var/run/tor/control Tor control unix domain socket or system Tor's default ControlPort 127.0.0.1 9051 would work out of the box.

(system Tor refers to Tor from packages.debian.org.)

Here is a trivial script that uses python3-stem that works as is in the workstation.

https://github.com/Whonix/anon-shared-helper-scripts/blob/master/usr/lib/anon-shared-helper-scripts/tor_circuit_established_check.py

anon-ws-disable-stacked-tor / control-port-filter-python have been designed to require as few changes by upstream application as possible. For example onionshare has a Tor config menu.

https://cloud.githubusercontent.com/assets/156128/21556064/8ead0338-cdd2-11e6-918c-d4ca61724b52.png

Any of these settings work in Whonix as is.

The only struggle for getting applications using Tor ephemeral hidden services running in Whonix is the listener IP and port range. For that we haven't found a generic solution. Either workarounds in Whonix and sometimes even cooperation with the application developers required. If you like to read more about that, you could have a look at this ticket.

https://phabricator.whonix.org/T561

Thank you for working on this! :)

On Mon, Feb 20, 2017 at 02:48:04PM -0800, Patrick Schleizer wrote:

The only struggle for getting applications using Tor ephemeral hidden services running in Whonix is the listener IP and port range. For that we haven't found a generic solution. Either workarounds in Whonix and sometimes even cooperation with the application developers required. If you like to read more about that, you could have a look at this ticket.

https://phabricator.whonix.org/T561

Thanks @adrelanos, I believe that might be the only thing left to do in order to make it work. If you can, take a look into Connect to Tor and use Ephemeral Onion Services The changes are based on what we discussed here.

That's a rather difficult process until it reaches testers-only quality.

https://www.whonix.org/wiki/Upgrading_Whonix_13_to_Whonix_14

The next thing I am going to do is following this link and hopefully find a way to test it on Whonix.

Thank you very much!

Hi @adrelanos,

I followed https://www.whonix.org/wiki/Upgrading_Whonix_13_to_Whonix_14 and it looks like I was able to update a gateway and workstation to Whonix 14.

To test, I executed this simple script that uses txtorcon to connect to the Tor process and then closes the connection:

#!/usr/bin/env python
from twisted.internet import defer, reactor
from twisted.internet.endpoints import TCP4ClientEndpoint
from twisted.internet.task import react

import txtorcon

WAIT_TIME = 5

@defer.inlineCallbacks
def connect(r):
    d = defer.Deferred()

    print 'Connecting'
    ep = TCP4ClientEndpoint(r, '127.0.0.1', 9151)
    tor = yield txtorcon.connect(r, ep)
    print 'Connected'

    def close():
        print 'Closing'
        tor.quit()
        print 'Closed'

        d.callback(None)

    r.callLater(WAIT_TIME, close)
    print 'Waiting for %d seconds' % WAIT_TIME

    yield d

react(connect)

(I tested on a non-Whonix systems and the script runs successfully)

As expected, I got a 510 error when I ran on Whonix because the commands were filtered. I created /usr/local/etc/tor-controlport-filter-merger.d/40_unmessage.yml with:

---
- exe-paths:
    - '*'
  users:
    - '*'
  hosts:
    - '*'
  commands:
    GETCONF:
      - '*'
    GETINFO:
      - '*'

Restarted the CPF with service tor-controlport-filter restart and I got no errors when I ran the script again, but txtorcon got stuck before connecting and it looks like I'll have to debug it. I tried to log the commands with:

@@ -374,6 +374,8 @@ class FilteredControlPortProxySession:
                 self.debug_log('ignoring received empty (or whitespace-only) '
                                + 'line')
                 continue
+            with open('/home/user/cpf.log', 'a') as f:
+                f.write(line)
             match = re.match(
                 r'(?P<cmd>\S+)(?P<cmd_arg_sep>\s*)(?P<arg_str>[^\r\n]*)\r?\n$',
                 line

but didn't work as well. (I also gave all users +rw permissions to that file)

Do you have any suggestions on how to debug it from the filter's side?

Thanks!

Thank you very much for your interest and dedication, @felipedau!

Sorry, I've been very busy and didn't get to test unMessage in Whonix 14 yet. It's still in my inbox and not forgotten. I appreciate all your efforts! Just briefly...

with open('/home/user/cpf.log', 'a') as f:

I doubt cpfpy (control-port-filter-python) code level changes are required to debug this. It already has logging.

To debug this...

On Whonix-Gateway.

sudo journalctl -f -u tor-controlport-filter

Keep watching the log as you start unMessage. Then you'll see any eventually rejected Tor control commands.

You'll also need to open a port in Whonix-Workstation firewall (or open all ports). unMessage instructions would be very, very similar to onionshare and ricochet which are documented.

Does that help? Please feel free to ask me anything.

Thank you very much for your interest and dedication, @felipedau!

Sorry, I've been very busy and didn't get to test unMessage in Whonix 14 yet. It's still in my inbox and not forgotten. I appreciate all your efforts! Just briefly...

Hey you are already spending time by discussing it with me! All I can do is just thank you :D

I doubt cpfpy (control-port-filter-python) code level changes are required to debug this. It already has logging.

To debug this...

On Whonix-Gateway.

sudo journalctl -f -u tor-controlport-filter

Keep watching the log as you start unMessage. Then you'll see any eventually rejected Tor control commands.

That's embarrassing. I think it's time for me to actually read the code - I should have noticed I could use that logger :O Thanks!

I got a friendlier output from journalctl -e:

Mar 07 10:02:03 host tor-controlport-filter[2090]: PROTOCOLINFO 1
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): -> PROTOCOLINFO 1
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): <- 250-PROTOCOLINFO 
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): <- 250-AUTH METHODS=
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): <- 250-VERSION Tor="
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): <- 250 OK
Mar 07 10:02:03 host tor-controlport-filter[2090]: AUTHENTICATE
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): -> AUTHENTICATE
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): <- 250 OK
Mar 07 10:02:03 host Tor[950]: New control connection opened.
Mar 07 10:02:03 host tor-controlport-filter[2090]: SETEVENTS
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): -> SETEVENTS
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): <- 250 OK
Mar 07 10:02:03 host tor-controlport-filter[2090]: GETCONF __owningcontrollerprocess
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated): -> GETCONF __owningc
Mar 07 10:02:03 host tor-controlport-filter[2090]: 10.137.3.43:50978 (filter: 30_autogenerated) disconnected: client

From txtorcon I got:

250-PROTOCOLINFO 1
250-AUTH METHODS=NULL
250-VERSION Tor="0.2.9.9 (git-1d8323c042800718)"
250 OK
AUTHENTICATE
250 OK
GETINFO signal/names

So apparently txtorcon is crashing somewhere. I will let you know once I debug it more.

You'll also need to open a port in Whonix-Workstation firewall (or open all ports). unMessage instructions would be very, very similar to onionshare and ricochet which are documented.
https://www.whonix.org/wiki/Next#onionshare
https://www.whonix.org/wiki/Next#ricochet
Does that help? Please feel free to ask me anything.

I will certainly do that, but I bet that is not the issue right now - probably will when I try to add a hs. I am currently just trying to connect to the process.

Thanks, I'll keep you updated!

I did not manage to witness unMessage to even talk to cpfpy. Please tell me if I should post the unMessage python exceptions.

Just tested, - '*' for commands will throw a cpfpy python exception. For development purposes, please try the following config. We can of course write a real profile once a prototype is functinal.

---
- exe-paths:
    - '*'
  users:
    - '*'
  hosts:
    - '*'
  commands:
    GETCONF:
     - '.+'
    GETINFO:
     - '.+'
  events:
    STREAM:

On Tue, Mar 07, 2017 at 07:29:41PM -0800, Patrick Schleizer wrote:

I did not manage to witness unMessage to even talk to cpfpy. Please tell me if I should post the unMessage python exceptions.

I just made a new commit that allows unMessage to listen on another interface, until we decide on the convention we are going to use.

You should run it with:

unmessage-cli -n <peer_name> --local-server-ip $(qubesdb-read /qubes-ip) --connect-to-tor -c 9151 -t 9050

Although I think you could try to use the script I posted above.

Just tested, - '*' for commands will throw a cpfpy python exception. For development purposes, please try the following config. We can of course write a real profile once a prototype is functinal.
---
- exe-paths:
    - '*'
  users:
    - '*'
  hosts:
    - '*'
  commands:
    GETCONF:
     - '.+'
    GETINFO:
     - '.+'
  events:
    STREAM:

Thanks!

And this is what happens for me.

unmessage-cli -n n --local-server-ip $(qubesdb-read /qubes-ip) --connect-to-tor -c 9151 -t 9050

 ! unMessage

 * Peer failed
 ! Peer failed: [Failure instance: Traceback: <type 'exceptions.AttributeError'>: 'EPollRe
actor' object has no attribute 'connect'
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:651:_runCallbacks
/usr/local/lib/python2.7/dist-packages/unmessage-0.1.1+39.g710f2ea-py2.7.egg/unmessage/pee
r.py:568:endpoint_listening
/usr/local/lib/python2.7/dist-packages/unmessage-0.1.1+39.g710f2ea-py2.7.egg/unmessage/pee
r.py:653:_start_tor
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1445:unwindGenerator
--- <exception caught here> ---
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1299:_inlineCallbacks
build/bdist.linux-x86_64/egg/txtorcon/torcontrolprotocol.py:56:connect
]
 * Running reactor

>

On Wed, Mar 08, 2017 at 06:56:22AM -0800, Patrick Schleizer wrote:

And this is what happens for me.

unmessage-cli -n n --local-server-ip $(qubesdb-read /qubes-ip) --connect-to-tor -c 9151 -t 9050

 ! unMessage

 * Peer failed
 ! Peer failed: [Failure instance: Traceback: <type 'exceptions.AttributeError'>: 'EPollRe
actor' object has no attribute 'connect'
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:651:_runCallbacks
/usr/local/lib/python2.7/dist-packages/unmessage-0.1.1+39.g710f2ea-py2.7.egg/unmessage/pee
r.py:568:endpoint_listening
/usr/local/lib/python2.7/dist-packages/unmessage-0.1.1+39.g710f2ea-py2.7.egg/unmessage/pee
r.py:653:_start_tor
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1445:unwindGenerator
--- <exception caught here> ---
/usr/lib/python2.7/dist-packages/twisted/internet/defer.py:1299:_inlineCallbacks
build/bdist.linux-x86_64/egg/txtorcon/torcontrolprotocol.py:56:connect
]
 * Running reactor

>

This version is using a newer txtorcon that has not been released yet, so you will have to manually install from https://github.com/meejah/txtorcon. Sorry, I just mentioned this in the PR and forgot about here :/

txtorcon has also dropped txsocksx but unMessage still uses it. Until I remove it from the code, please make sure to pip install txsocksx.

Finally, sometimes when unMessage fails to start, the user information is not saved completely and it will crash the next time you launch it because it could not load that info properly, so until I fix that as well, you will have to rm -fr ~/.config/unMessage/ :/

I hope now you will be able to find the real issue but please do not spend too much time on that! Once I am able to run new tests, I will let you know what I learned.

Thank you very much @adrelanos!

Cloned https://github.com/meejah/txtorcon. Have run all of them.

python setup.py build
python setup.py install
sudo python setup.py build
sudo python setup.py install

Do I still have some dependency issue?

unmessage-cli -n n --local-server-ip $(qubesdb-read /qubes-ip) --connect-to-tor -c 9151 -t 9050
Traceback (most recent call last):
  File "/usr/local/bin/unmessage-cli", line 11, in <module>
    load_entry_point('unmessage==0.1.1+39.g710f2ea', 'console_scripts', 'unmessage-cli')()
  File "/home/user/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 560, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/user/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2648, in load_entry_point
    return ep.load()
  File "/home/user/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2302, in load
    return self.resolve()
  File "/home/user/.local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2308, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/local/lib/python2.7/dist-packages/unmessage-0.1.1+39.g710f2ea-py2.7.egg/unmessage/__main__.py", line 5, in <module>
    from . import cli
  File "/usr/local/lib/python2.7/dist-packages/unmessage-0.1.1+39.g710f2ea-py2.7.egg/unmessage/cli.py", line 12, in <module>
    from . import peer
  File "/usr/local/lib/python2.7/dist-packages/unmessage-0.1.1+39.g710f2ea-py2.7.egg/unmessage/peer.py", line 15, in <module>
    import txtorcon
  File "build/bdist.linux-x86_64/egg/txtorcon/__init__.py", line 16, in <module>
  File "build/bdist.linux-x86_64/egg/txtorcon/controller.py", line 32, in <module>
  File "build/bdist.linux-x86_64/egg/txtorcon/torconfig.py", line 34, in <module>
  File "/home/user/.local/lib/python2.7/site-packages/twisted/python/deprecate.py", line 293, in deprecationDecorator
    function, version, None, replacement)
  File "/home/user/.local/lib/python2.7/site-packages/twisted/python/deprecate.py", line 238, in getDeprecationWarningString
    _fullyQualifiedName(callableThing), version, format, replacement)
  File "/home/user/.local/lib/python2.7/site-packages/twisted/python/deprecate.py", line 198, in _getDeprecationWarningString
    'version': getVersionString(version)}
  File "/home/user/.local/lib/python2.7/site-packages/incremental/__init__.py", line 518, in getVersionString
    result = '%s %s' % (version.package, version.short())
  File "/home/user/.local/lib/python2.7/site-packages/incremental/__init__.py", line 221, in short
    gitver = self._getGitVersion()
  File "/home/user/.local/lib/python2.7/site-packages/incremental/__init__.py", line 411, in _getGitVersion
    if ".git" in os.listdir(upOne):
OSError: [Errno 20] Not a directory: '/usr/local/lib/python2.7/dist-packages/txtorcon-0.18.0-py2.7.egg'

On Wed, Mar 08, 2017 at 02:47:18PM -0800, Patrick Schleizer wrote:

Cloned https://github.com/meejah/txtorcon. Have run all of them.
python setup.py build
python setup.py install
sudo python setup.py build
sudo python setup.py install
Do I still have some dependency issue?

Hmm, doesn't look like it. I've never seen anything like that. Could you try to use pip to install it? These are the dependencies you can install with the package manager:

sudo apt-get install build-essential gcc libffi-dev libopus0 \
    libsodium-dev libssl-dev portaudio19-dev python-dev python-tk tor

I would suggest you to create a virtual environment:

sudo pip install virtualenv
virtualenv test-env
source test-env/bin/activate

Install them, one at a time, using the env's pip (without sudo)

pip install -e /path/to/txtorcon/
pip install -e /path/to/unmessage/

-e Installs in editable mode so that you can update the repo without reinstalling the package. pip list should display:

txtorcon (0.18.0, /path/to/txtorcon)
unmessage (0.1.1+39.g710f2ea, /path/to/unmessage)

I hope that works! (If it does though, we must find out why the regular setup fails)

Thanks!

Did that.

pip list

DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.                         
appdirs (1.4.3)
attrs (16.3.0)
Automat (0.5.0)
cffi (1.9.1)
constantly (15.1.0)
cryptography (1.7.2)
enum34 (1.1.6)
idna (2.5)
incremental (16.10.1)
ipaddress (1.0.18)
opuslib (1.1.0)
packaging (16.8)
passlib (1.7.1)
pip (9.0.1)
pyasn1 (0.2.3)
pyasn1-modules (0.0.8)
PyAudio (0.2.10)
pyaxo (0.7.7)
pycparser (2.17)
PyNaCl (1.0.1)
pyOpenSSL (16.2.0)
pyparsing (2.2.0)
pyperclip (1.5.27)
service-identity (16.0.0)
setuptools (34.3.1)
six (1.10.0)
Twisted (17.1.0)
txtorcon (0.18.0, /home/user/txtorcon)
unmessage (0.1.1+39.g710f2ea, /home/user/unmessage)
wheel (0.29.0)
zope.interface (4.3.3)

Same issue as in my last comment.

On Wed, Mar 08, 2017 at 03:36:28PM -0800, Patrick Schleizer wrote:

Same issue as in my last comment.

Hmm, if it is the exact same error, then unMessage is importing the txtorcon you had previously installed. When using a virtual environment, the files should be at /path/to/env/lib/python2.7/site-packages/ and not /home/user/.local/lib/python2.7/site-packages/. I know this is just a detail, but in our case it is important.

I tested in my vm (which does launch unMessage) and on that section of incremental's code upOne is '/path/to/txtorcon/' which is really a directory and not a file.

I noticed from the list that txsocksx is not installed - I do not know how unMessage did not crash on line 14 (import txsocksx.errors), but if at some point it fails to import it, pip install txsocksx solves it.

I think you might have to uninstall those packages or just create a new vm.

I hope it works :) Thanks!

New VM. Please advice.

user@host:~$ pip
Could not find the database of available applications, run update-command-not-found as root to fix this
pip: command not found
user@host:~$ sudo apt-get install build-essential gcc libffi-dev libopus0 \ libsodium-dev libssl-dev portaudio19-dev python-dev python-tk tor python-pip
Reading package lists... Done
Building dependency tree       
Reading state information... Done
E: Unable to locate package  libsodium-dev
user@host:~$ sudo apt-get install build-essential gcc libffi-dev libopus0 libsodium-dev libssl-dev portaudio19-dev python-dev python-tk tor python-pip  
Reading package lists... Done
Building dependency tree       
Reading state information... Done
build-essential is already the newest version (12.3).
gcc is already the newest version (4:6.3.0-1).
gcc set to manually installed.
libopus0 is already the newest version (1.2~alpha2-1).
libopus0 set to manually installed.
tor is already the newest version (0.2.9.9-1).
tor set to manually installed.
The following additional packages will be installed:
  blt libasound2-dev libexpat1-dev libjack-dev libjack0 libportaudiocpp0 libpython-all-dev libpython-dev libpython2.7-dev libssl-doc libtk8.6 python-all
  python-all-dev python-crypto python-keyring python-keyrings.alt python-pip-whl python-secretstorage python-wheel python2.7-dev tk8.6-blt2.5 uuid-dev
Suggested packages:                                                                                                                                           
  blt-demo libasound2-doc jackd1 tk8.6 portaudio19-doc python-crypto-dbg python-crypto-doc python-fs python-gdata python-kde4 python-keyczar                  
  python-secretstorage-doc tix python-tk-dbg                                                                                                                  
The following packages will be REMOVED:                                                                                                                       
  libjack-jackd2-0                                                                                                                                            
The following NEW packages will be installed:                                                                                                                 
  blt libasound2-dev libexpat1-dev libffi-dev libjack-dev libjack0 libportaudiocpp0 libpython-all-dev libpython-dev libpython2.7-dev libsodium-dev
  libssl-dev libssl-doc libtk8.6 portaudio19-dev python-all python-all-dev python-crypto python-dev python-keyring python-keyrings.alt python-pip
  python-pip-whl python-secretstorage python-tk python-wheel python2.7-dev tk8.6-blt2.5 uuid-dev
0 upgraded, 29 newly installed, 1 to remove and 0 not upgraded.
Need to get 36.1 MB of archives.
After this operation, 67.9 MB of additional disk space will be used.
Do you want to continue? [Y/n] 
Get:1 http://ftp.us.debian.org/debian stretch/main amd64 libjack0 amd64 1:0.125.0-2 [56.4 kB]
Get:2 http://ftp.us.debian.org/debian stretch/main amd64 libtk8.6 amd64 8.6.6-1+b1 [754 kB]
Get:3 http://ftp.us.debian.org/debian stretch/main amd64 tk8.6-blt2.5 amd64 2.5.3+dfsg-3 [585 kB]
Get:4 http://ftp.us.debian.org/debian stretch/main amd64 blt amd64 2.5.3+dfsg-3 [14.6 kB]
Get:5 http://ftp.us.debian.org/debian stretch/main amd64 libasound2-dev amd64 1.1.3-5 [262 kB]
Get:6 http://ftp.us.debian.org/debian stretch/main amd64 libexpat1-dev amd64 2.2.0-2 [134 kB]                                                                
Get:7 http://ftp.us.debian.org/debian stretch/main amd64 libffi-dev amd64 3.2.1-6 [161 kB]                                                                   
Get:8 http://ftp.us.debian.org/debian stretch/main amd64 uuid-dev amd64 2.29.1-1 [83.7 kB]                                                                   
Get:9 http://ftp.us.debian.org/debian stretch/main amd64 libjack-dev amd64 1:0.125.0-2 [223 kB]                                                              
Get:10 http://ftp.us.debian.org/debian stretch/main amd64 libportaudiocpp0 amd64 19.6.0-1 [18.8 kB]                                                          
Get:11 http://ftp.us.debian.org/debian stretch/main amd64 libpython2.7-dev amd64 2.7.13-2 [28.2 MB]                                                          
Get:12 http://ftp.us.debian.org/debian stretch/main amd64 libpython-dev amd64 2.7.13-2 [20.1 kB]                                                             
Get:13 http://ftp.us.debian.org/debian stretch/main amd64 libpython-all-dev amd64 2.7.13-2 [960 B]                                                           
Get:14 http://ftp.us.debian.org/debian stretch/main amd64 libsodium-dev amd64 1.0.11-1 [170 kB]                                                              
Get:15 http://ftp.us.debian.org/debian stretch/main amd64 libssl-dev amd64 1.1.0e-1 [1,572 kB]                                                               
Get:16 http://ftp.us.debian.org/debian stretch/main amd64 libssl-doc all 1.1.0e-1 [1,425 kB]                                                                 
Get:17 http://ftp.us.debian.org/debian stretch/main amd64 portaudio19-dev amd64 19.6.0-1 [108 kB]                                                            
Get:18 http://ftp.us.debian.org/debian stretch/main amd64 python-all amd64 2.7.13-2 [942 B]                                                                  
Get:19 http://ftp.us.debian.org/debian stretch/main amd64 python2.7-dev amd64 2.7.13-2 [281 kB]                                                              
Get:20 http://ftp.us.debian.org/debian stretch/main amd64 python-dev amd64 2.7.13-2 [1,126 B]                                                                
Get:21 http://ftp.us.debian.org/debian stretch/main amd64 python-all-dev amd64 2.7.13-2 [962 B]                                                              
Get:22 http://ftp.us.debian.org/debian stretch/main amd64 python-crypto amd64 2.6.1-7 [259 kB]                                                               
Get:23 http://ftp.us.debian.org/debian stretch/main amd64 python-secretstorage all 2.3.1-2 [13.8 kB]                                                         
Get:24 http://ftp.us.debian.org/debian stretch/main amd64 python-keyring all 10.1-1 [40.7 kB]                                                                
Get:25 http://ftp.us.debian.org/debian stretch/main amd64 python-keyrings.alt all 1.3-1 [16.4 kB]                                                            
Get:26 http://ftp.us.debian.org/debian stretch/main amd64 python-pip-whl all 9.0.1-2 [1,406 kB]                                                              
Get:27 http://ftp.us.debian.org/debian stretch/main amd64 python-pip all 9.0.1-2 [179 kB]                                                                    
Get:28 http://ftp.us.debian.org/debian stretch/main amd64 python-tk amd64 2.7.13-1 [27.4 kB]                                                                 
Get:29 http://ftp.us.debian.org/debian stretch/main amd64 python-wheel all 0.29.0-2 [51.7 kB]                                                                
Fetched 36.1 MB in 1min 17s (463 kB/s)                                                                                                                       
dpkg: libjack-jackd2-0:amd64: dependency problems, but removing anyway as you requested:
 mpv depends on libjack-jackd2-0 (>= 1.9.10+20150825) | libjack-0.125; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.125 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.125 is to be removed.
 gstreamer1.0-plugins-good:amd64 depends on libjack-jackd2-0 (>= 1.9.10+20150825) | libjack-0.125; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.125 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.125 is to be removed.
 libasound2-plugins:amd64 depends on libjack-jackd2-0 (>= 1.9.5~dfsg-14) | libjack-0.116; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.116 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.116 is to be removed.
 libportaudio2:amd64 depends on libjack-jackd2-0 (>= 1.9.10+20150825) | libjack-0.125; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.125 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.125 is to be removed.
 libavdevice57:amd64 depends on libjack-jackd2-0 (>= 1.9.10+20150825) | libjack-0.125; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.125 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.125 is to be removed.
 libasound2-plugins:amd64 depends on libjack-jackd2-0 (>= 1.9.5~dfsg-14) | libjack-0.116; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.116 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.116 is to be removed.
 mpv depends on libjack-jackd2-0 (>= 1.9.10+20150825) | libjack-0.125; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.125 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.125 is to be removed.
 gstreamer1.0-plugins-good:amd64 depends on libjack-jackd2-0 (>= 1.9.10+20150825) | libjack-0.125; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.125 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.125 is to be removed.
 libportaudio2:amd64 depends on libjack-jackd2-0 (>= 1.9.10+20150825) | libjack-0.125; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.125 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.125 is to be removed.
 libavdevice57:amd64 depends on libjack-jackd2-0 (>= 1.9.10+20150825) | libjack-0.125; however:
  Package libjack-jackd2-0:amd64 is to be removed.
  Package libjack-0.125 is not installed.
  Package libjack-jackd2-0:amd64 which provides libjack-0.125 is to be removed.

(Reading database ... 160897 files and directories currently installed.)
Removing libjack-jackd2-0:amd64 (1.9.10+20150825git1ed50c92~dfsg-4+b1) ...
Selecting previously unselected package libjack0:amd64.
(Reading database ... 160885 files and directories currently installed.)
Preparing to unpack .../00-libjack0_1%3a0.125.0-2_amd64.deb ...
Unpacking libjack0:amd64 (1:0.125.0-2) ...
Selecting previously unselected package libtk8.6:amd64.
Preparing to unpack .../01-libtk8.6_8.6.6-1+b1_amd64.deb ...
Unpacking libtk8.6:amd64 (8.6.6-1+b1) ...
Selecting previously unselected package tk8.6-blt2.5.
Preparing to unpack .../02-tk8.6-blt2.5_2.5.3+dfsg-3_amd64.deb ...
Unpacking tk8.6-blt2.5 (2.5.3+dfsg-3) ...
Selecting previously unselected package blt.
Preparing to unpack .../03-blt_2.5.3+dfsg-3_amd64.deb ...
Unpacking blt (2.5.3+dfsg-3) ...
Selecting previously unselected package libasound2-dev:amd64.
Preparing to unpack .../04-libasound2-dev_1.1.3-5_amd64.deb ...
Unpacking libasound2-dev:amd64 (1.1.3-5) ...
Selecting previously unselected package libexpat1-dev:amd64.
Preparing to unpack .../05-libexpat1-dev_2.2.0-2_amd64.deb ...
Unpacking libexpat1-dev:amd64 (2.2.0-2) ...
Selecting previously unselected package libffi-dev:amd64.
Preparing to unpack .../06-libffi-dev_3.2.1-6_amd64.deb ...
Unpacking libffi-dev:amd64 (3.2.1-6) ...
Selecting previously unselected package uuid-dev:amd64.
Preparing to unpack .../07-uuid-dev_2.29.1-1_amd64.deb ...
Unpacking uuid-dev:amd64 (2.29.1-1) ...
Selecting previously unselected package libjack-dev.
Preparing to unpack .../08-libjack-dev_1%3a0.125.0-2_amd64.deb ...
Unpacking libjack-dev (1:0.125.0-2) ...
Selecting previously unselected package libportaudiocpp0:amd64.
Preparing to unpack .../09-libportaudiocpp0_19.6.0-1_amd64.deb ...
Unpacking libportaudiocpp0:amd64 (19.6.0-1) ...
Selecting previously unselected package libpython2.7-dev:amd64.
Preparing to unpack .../10-libpython2.7-dev_2.7.13-2_amd64.deb ...
Unpacking libpython2.7-dev:amd64 (2.7.13-2) ...
Selecting previously unselected package libpython-dev:amd64.
Preparing to unpack .../11-libpython-dev_2.7.13-2_amd64.deb ...
Unpacking libpython-dev:amd64 (2.7.13-2) ...
Selecting previously unselected package libpython-all-dev:amd64.
Preparing to unpack .../12-libpython-all-dev_2.7.13-2_amd64.deb ...
Unpacking libpython-all-dev:amd64 (2.7.13-2) ...
Selecting previously unselected package libsodium-dev:amd64.
Preparing to unpack .../13-libsodium-dev_1.0.11-1_amd64.deb ...
Unpacking libsodium-dev:amd64 (1.0.11-1) ...
Selecting previously unselected package libssl-dev:amd64.
Preparing to unpack .../14-libssl-dev_1.1.0e-1_amd64.deb ...
Unpacking libssl-dev:amd64 (1.1.0e-1) ...
Selecting previously unselected package libssl-doc.
Preparing to unpack .../15-libssl-doc_1.1.0e-1_all.deb ...
Unpacking libssl-doc (1.1.0e-1) ...
Selecting previously unselected package portaudio19-dev:amd64.
Preparing to unpack .../16-portaudio19-dev_19.6.0-1_amd64.deb ...
Unpacking portaudio19-dev:amd64 (19.6.0-1) ...
Selecting previously unselected package python-all.
Preparing to unpack .../17-python-all_2.7.13-2_amd64.deb ...
Unpacking python-all (2.7.13-2) ...
Selecting previously unselected package python2.7-dev.
Preparing to unpack .../18-python2.7-dev_2.7.13-2_amd64.deb ...
Unpacking python2.7-dev (2.7.13-2) ...
Selecting previously unselected package python-dev.
Preparing to unpack .../19-python-dev_2.7.13-2_amd64.deb ...
Unpacking python-dev (2.7.13-2) ...
Selecting previously unselected package python-all-dev.
Preparing to unpack .../20-python-all-dev_2.7.13-2_amd64.deb ...
Unpacking python-all-dev (2.7.13-2) ...
Selecting previously unselected package python-crypto.
Preparing to unpack .../21-python-crypto_2.6.1-7_amd64.deb ...
Unpacking python-crypto (2.6.1-7) ...
Selecting previously unselected package python-secretstorage.
Preparing to unpack .../22-python-secretstorage_2.3.1-2_all.deb ...
Unpacking python-secretstorage (2.3.1-2) ...
Selecting previously unselected package python-keyring.
Preparing to unpack .../23-python-keyring_10.1-1_all.deb ...
Unpacking python-keyring (10.1-1) ...
Selecting previously unselected package python-keyrings.alt.
Preparing to unpack .../24-python-keyrings.alt_1.3-1_all.deb ...
Unpacking python-keyrings.alt (1.3-1) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../25-python-pip-whl_9.0.1-2_all.deb ...
Unpacking python-pip-whl (9.0.1-2) ...
Selecting previously unselected package python-pip.
Preparing to unpack .../26-python-pip_9.0.1-2_all.deb ...
Unpacking python-pip (9.0.1-2) ...
Selecting previously unselected package python-tk.
Preparing to unpack .../27-python-tk_2.7.13-1_amd64.deb ...
Unpacking python-tk (2.7.13-1) ...
Selecting previously unselected package python-wheel.
Preparing to unpack .../28-python-wheel_0.29.0-2_all.deb ...
Unpacking python-wheel (0.29.0-2) ...
Setting up python-secretstorage (2.3.1-2) ...
Setting up python-pip-whl (9.0.1-2) ...
Setting up libasound2-dev:amd64 (1.1.3-5) ...
Setting up libportaudiocpp0:amd64 (19.6.0-1) ...
Setting up python-crypto (2.6.1-7) ...
Setting up python-keyring (10.1-1) ...
Setting up libjack0:amd64 (1:0.125.0-2) ...
Setting up libssl-dev:amd64 (1.1.0e-1) ...
Setting up libtk8.6:amd64 (8.6.6-1+b1) ...
Setting up python-wheel (0.29.0-2) ...
Setting up uuid-dev:amd64 (2.29.1-1) ...
Setting up libffi-dev:amd64 (3.2.1-6) ...
Setting up python-keyrings.alt (1.3-1) ...
Setting up libsodium-dev:amd64 (1.0.11-1) ...
Processing triggers for libc-bin (2.24-9) ...
Setting up tk8.6-blt2.5 (2.5.3+dfsg-3) ...
Setting up libexpat1-dev:amd64 (2.2.0-2) ...
Processing triggers for man-db (2.7.6.1-2) ...
Setting up libpython2.7-dev:amd64 (2.7.13-2) ...
Setting up libssl-doc (1.1.0e-1) ...
Setting up python-pip (9.0.1-2) ...
Setting up python2.7-dev (2.7.13-2) ...
Setting up python-all (2.7.13-2) ...
Setting up blt (2.5.3+dfsg-3) ...
Setting up libpython-dev:amd64 (2.7.13-2) ...
Setting up libjack-dev (1:0.125.0-2) ...
Setting up portaudio19-dev:amd64 (19.6.0-1) ...
Setting up python-dev (2.7.13-2) ...
Setting up libpython-all-dev:amd64 (2.7.13-2) ...
Setting up python-all-dev (2.7.13-2) ...
Setting up python-tk (2.7.13-1) ...
Processing triggers for libc-bin (2.24-9) ...
user@host:~$ sudo pip install virtualenv
Collecting virtualenv
  Downloading virtualenv-15.1.0-py2.py3-none-any.whl (1.8MB)
    100% || 1.8MB 335kB/s 
Installing collected packages: virtualenv
Successfully installed virtualenv-15.1.0
user@host:~$ virtualenv test-env
New python executable in /home/user/test-env/bin/python
Installing setuptools, pip, wheel...done.
user@host:~$ source test-env/bin/activate
(test-env) user@host:~$ pip install -e txtorcon         
Obtaining file:///home/user/txtorcon
Collecting Twisted[tls]>=11.1.0 (from txtorcon==0.18.0)
  Downloading Twisted-17.1.0.tar.bz2 (3.0MB)
    100% || 3.0MB 261kB/s 
Collecting ipaddress>=1.0.16 (from txtorcon==0.18.0)
  Downloading ipaddress-1.0.18-py2-none-any.whl
Collecting zope.interface>=3.6.1 (from txtorcon==0.18.0)
  Downloading zope.interface-4.3.3.tar.gz (150kB)
    100% || 153kB 422kB/s 
Collecting incremental (from txtorcon==0.18.0)
  Downloading incremental-16.10.1-py2.py3-none-any.whl
Collecting automat (from txtorcon==0.18.0)
  Downloading Automat-0.5.0-py2.py3-none-any.whl
Collecting constantly>=15.1 (from Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading constantly-15.1.0-py2.py3-none-any.whl
Collecting pyopenssl>=16.0.0 (from Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading pyOpenSSL-16.2.0-py2.py3-none-any.whl (43kB)
    100% || 51kB 367kB/s 
Collecting service_identity (from Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading service_identity-16.0.0-py2.py3-none-any.whl
Collecting idna>=0.6 (from Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading idna-2.5-py2.py3-none-any.whl (55kB)
    100% || 61kB 677kB/s 
Requirement already satisfied: setuptools in ./test-env/lib/python2.7/site-packages (from zope.interface>=3.6.1->txtorcon==0.18.0)
Requirement already satisfied: six in ./test-env/lib/python2.7/site-packages (from automat->txtorcon==0.18.0)
Collecting attrs (from automat->txtorcon==0.18.0)
  Downloading attrs-16.3.0-py2.py3-none-any.whl
Collecting cryptography>=1.3.4 (from pyopenssl>=16.0.0->Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading cryptography-1.7.2.tar.gz (420kB)
    100% || 430kB 1.0MB/s 
Collecting pyasn1-modules (from service_identity->Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting pyasn1 (from service_identity->Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading pyasn1-0.2.3-py2.py3-none-any.whl (53kB)
    100% || 61kB 1.1MB/s 
Requirement already satisfied: appdirs>=1.4.0 in ./test-env/lib/python2.7/site-packages (from setuptools->zope.interface>=3.6.1->txtorcon==0.18.0)
Requirement already satisfied: packaging>=16.8 in ./test-env/lib/python2.7/site-packages (from setuptools->zope.interface>=3.6.1->txtorcon==0.18.0)
Collecting enum34 (from cryptography>=1.3.4->pyopenssl>=16.0.0->Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading enum34-1.1.6-py2-none-any.whl
Collecting cffi>=1.4.1 (from cryptography>=1.3.4->pyopenssl>=16.0.0->Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading cffi-1.9.1-cp27-cp27mu-manylinux1_x86_64.whl (387kB)
    100% || 389kB 504kB/s 
Requirement already satisfied: pyparsing in ./test-env/lib/python2.7/site-packages (from packaging>=16.8->setuptools->zope.interface>=3.6.1->txtorcon==0.18.0)
Collecting pycparser (from cffi>=1.4.1->cryptography>=1.3.4->pyopenssl>=16.0.0->Twisted[tls]>=11.1.0->txtorcon==0.18.0)
  Downloading pycparser-2.17.tar.gz (231kB)
    100% || 235kB 678kB/s 
Building wheels for collected packages: Twisted, zope.interface, cryptography, pycparser
  Running setup.py bdist_wheel for Twisted ... done
  Stored in directory: /home/user/.cache/pip/wheels/65/e3/44/cd3da92c03926aabc80e658e11d6e64619abce3ef44c1c34df
  Running setup.py bdist_wheel for zope.interface ... done
  Stored in directory: /home/user/.cache/pip/wheels/00/aa/8b/f1d1eb398423e59894b45ee151344e243808156c2d182c9f4e
  Running setup.py bdist_wheel for cryptography ... done
  Stored in directory: /home/user/.cache/pip/wheels/3e/09/be/ca344cd1e68fd4d4d8872a25287e884a6f73557710efa51019
  Running setup.py bdist_wheel for pycparser ... done
  Stored in directory: /home/user/.cache/pip/wheels/a8/0b/41/dc95621f9d3a0da7bc191b8a71f0e8182ffd3cc5f33ac55005
Successfully built Twisted zope.interface cryptography pycparser
Installing collected packages: zope.interface, constantly, incremental, attrs, automat, idna, pyasn1, enum34, ipaddress, pycparser, cffi, cryptography, pyopenssl, pyasn1-modules, service-identity, Twisted, txtorcon
  Running setup.py develop for txtorcon
Successfully installed Twisted-17.1.0 attrs-16.3.0 automat-0.5.0 cffi-1.9.1 constantly-15.1.0 cryptography-1.7.2 enum34-1.1.6 idna-2.5 incremental-16.10.1 ipaddress-1.0.18 pyasn1-0.2.3 pyasn1-modules-0.0.8 pycparser-2.17 pyopenssl-16.2.0 service-identity-16.0.0 txtorcon zope.interface-4.3.3
(test-env) user@host:~$ 
(test-env) user@host:~$ 
(test-env) user@host:~$ pip install -e unmessage         
Obtaining file:///home/user/unmessage
Collecting opuslib>=1.1.0 (from unmessage==0.1.1+39.g710f2ea)
  Downloading opuslib-1.1.0.tar.gz
Collecting PyAudio>=0.2.10 (from unmessage==0.1.1+39.g710f2ea)
  Downloading PyAudio-0.2.10.tar.gz (287kB)
    100% || 296kB 436kB/s 
Collecting pyaxo>=0.7.3 (from unmessage==0.1.1+39.g710f2ea)
  Downloading pyaxo-0.7.7.tar.gz (63kB)
    100% || 71kB 120kB/s 
Collecting PyNaCl>=1.0.1 (from unmessage==0.1.1+39.g710f2ea)
  Downloading PyNaCl-1.0.1.tar.gz (2.7MB)
    100% || 2.7MB 226kB/s 
Collecting pyperclip>=1.5.27 (from unmessage==0.1.1+39.g710f2ea)
  Downloading pyperclip-1.5.27.zip
Requirement already satisfied: Twisted>=16.6.0 in ./test-env/lib/python2.7/site-packages (from unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: txtorcon>=0.17.0 in ./txtorcon (from unmessage==0.1.1+39.g710f2ea)
Collecting passlib>=1.6.1 (from pyaxo>=0.7.3->unmessage==0.1.1+39.g710f2ea)
  Downloading passlib-1.7.1-py2.py3-none-any.whl (498kB)
    100% || 501kB 402kB/s 
Requirement already satisfied: six in ./test-env/lib/python2.7/site-packages (from PyNaCl>=1.0.1->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: cffi>=1.1.0 in ./test-env/lib/python2.7/site-packages (from PyNaCl>=1.0.1->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: Automat>=0.3.0 in ./test-env/lib/python2.7/site-packages (from Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: constantly>=15.1 in ./test-env/lib/python2.7/site-packages (from Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: incremental>=16.10.1 in ./test-env/lib/python2.7/site-packages (from Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: zope.interface>=3.6.0 in ./test-env/lib/python2.7/site-packages (from Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: ipaddress>=1.0.16 in ./test-env/lib/python2.7/site-packages (from txtorcon>=0.17.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: pycparser in ./test-env/lib/python2.7/site-packages (from cffi>=1.1.0->PyNaCl>=1.0.1->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: attrs in ./test-env/lib/python2.7/site-packages (from Automat>=0.3.0->Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: setuptools in ./test-env/lib/python2.7/site-packages (from zope.interface>=3.6.0->Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: appdirs>=1.4.0 in ./test-env/lib/python2.7/site-packages (from setuptools->zope.interface>=3.6.0->Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: packaging>=16.8 in ./test-env/lib/python2.7/site-packages (from setuptools->zope.interface>=3.6.0->Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Requirement already satisfied: pyparsing in ./test-env/lib/python2.7/site-packages (from packaging>=16.8->setuptools->zope.interface>=3.6.0->Twisted>=16.6.0->unmessage==0.1.1+39.g710f2ea)
Building wheels for collected packages: opuslib, PyAudio, pyaxo, PyNaCl, pyperclip
  Running setup.py bdist_wheel for opuslib ... done
  Stored in directory: /home/user/.cache/pip/wheels/71/a0/e5/8bf54ee0a68ffa1c2b6ea9d2309e3233465de85d059f678e37
  Running setup.py bdist_wheel for PyAudio ... done
  Stored in directory: /home/user/.cache/pip/wheels/fc/01/de/06cb78cefa82ecbdee4a65b717976e03ad75c5867e3be38ff0
  Running setup.py bdist_wheel for pyaxo ... done
  Stored in directory: /home/user/.cache/pip/wheels/ae/b8/c5/f07d82c958b7dbe9bc89f95c16e5e59e0da8ac007918efd80c
  Running setup.py bdist_wheel for PyNaCl ... done
  Stored in directory: /home/user/.cache/pip/wheels/5c/e3/22/01d05a856963e55f2377627dfb192503c4383fb4b1e3419535
  Running setup.py bdist_wheel for pyperclip ... done
  Stored in directory: /home/user/.cache/pip/wheels/0b/fe/d7/1ab1ec7a91dc707d04b872214f6ce617f1b04a027de12fd4fe
Successfully built opuslib PyAudio pyaxo PyNaCl pyperclip
Installing collected packages: opuslib, PyAudio, passlib, PyNaCl, pyaxo, pyperclip, unmessage
  Running setup.py develop for unmessage
Successfully installed PyAudio-0.2.10 PyNaCl-1.0.1 opuslib-1.1.0 passlib-1.7.1 pyaxo-0.7.7 pyperclip-1.5.27 unmessage
(test-env) user@host:~$ 
(test-env) user@host:~$ 
(test-env) user@host:~$ unmessage-cli -n n --local-server-ip $(qubesdb-read /qubes-ip) --connect-to-tor -c 9151 -t 9050
Traceback (most recent call last):
  File "/home/user/test-env/bin/unmessage-cli", line 11, in <module>
    load_entry_point('unmessage', 'console_scripts', 'unmessage-cli')()
  File "/home/user/test-env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 560, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/home/user/test-env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2648, in load_entry_point
    return ep.load()
  File "/home/user/test-env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2302, in load
    return self.resolve()
  File "/home/user/test-env/local/lib/python2.7/site-packages/pkg_resources/__init__.py", line 2308, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/home/user/unmessage/unmessage/__main__.py", line 5, in <module>
    from . import cli
  File "/home/user/unmessage/unmessage/cli.py", line 12, in <module>
    from . import peer
  File "/home/user/unmessage/unmessage/peer.py", line 14, in <module>
    import txsocksx.errors
ImportError: No module named txsocksx.errors
(test-env) user@host:~$ 
(test-env) user@host:~$ 
(test-env) user@host:~$ pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
appdirs (1.4.3)
attrs (16.3.0)
Automat (0.5.0)
cffi (1.9.1)
constantly (15.1.0)
cryptography (1.7.2)
enum34 (1.1.6)
idna (2.5)
incremental (16.10.1)
ipaddress (1.0.18)
opuslib (1.1.0)
packaging (16.8)
passlib (1.7.1)
pip (9.0.1)
pyasn1 (0.2.3)
pyasn1-modules (0.0.8)
PyAudio (0.2.10)
pyaxo (0.7.7)
pycparser (2.17)
PyNaCl (1.0.1)
pyOpenSSL (16.2.0)
pyparsing (2.2.0)
pyperclip (1.5.27)
service-identity (16.0.0)
setuptools (34.3.1)
six (1.10.0)
Twisted (17.1.0)
txtorcon (0.18.0, /home/user/txtorcon)
unmessage (0.1.1+39.g710f2ea, /home/user/unmessage)
wheel (0.29.0)
zope.interface (4.3.3)
(test-env) user@host:~$

On Wed, Mar 08, 2017 at 05:28:28PM -0800, Patrick Schleizer wrote:

File "/home/user/unmessage/unmessage/peer.py", line 14, in import txsocksx.errors ImportError: No module named txsocksx.errors

This is the error I said that I had no idea why it did not manifest on the first time, but now it did. pip install txsocksx will fix that.

(I won't promisse, but I'll try to remove this dependency from unMessage tonight)

Once that's installed, the next line (import txtorcon) is the one that caused the first error you reported. I believe that this time it won't manifest. If the CLI successfully launches, unMessage will likely get stuck at:

 * Starting peer
 * Configuring local server
 * Running local server
 * Connecting to existing Tor
 * Running reactor

cpfpy should have received the commands. If it does not progress within a couple seconds, you can just /quit because txtorcon failed to connect.

If you get:

 * Starting peer
 * Configuring local server
 * Peer failed
 ! Peer failed: [Failure instance: Traceback: <type 'exceptions.TypeError'>: an integer is required

the peer's info got corrupted. You should rm -fr ~/.config/unMessage/ and relaunch unMessage.

I'm really sorry for all this - I hope to fix these issues within the week.

Thanks!

No problem. Right. Seeing denied messages in cpfpy log now. Will work on a profile.

Do you have a ticket on how to build from source code yet? In these comments you gave a pretty good guide. Perhaps copy it over to a readme?

This profile goes further.

---
- exe-paths:
    - '*'
  users:
    - '*'
  hosts:
    - '*'
  commands:
    GETCONF:
     - '.+'
    GETINFO:
     - '.+'
    USEFEATURE:
     - '.+'
  events:
    STREAM:

Might be the first application using txtorcon with cpfpy.

 ! unMessage

 * Starting peer
 * Configuring local server
 * Running local server
 * Connecting to existing Tor
 * Running reactor
 * Controlling Tor process
 * Waiting for the Onion Service
 * Peer failed
 ! Peer failed: [Failure instance: Traceback (failure with no frames): <class 'txtorcon.torcontrolprotocol.TorProtocolError'>: 510 Command filtered
]

unMessage gets 510 Command filtered which must come from cpfpy. However, cpfpy's log does not include that it replied 510 Command filtered. We might have a bug in cpfpy where it replies 510 Command filtered but fails to which incoming command it denied and fails to log that it replied 510 Command filtered.

On Wed, Mar 08, 2017 at 06:23:02PM -0800, Patrick Schleizer wrote:

No problem. Right. Seeing denied messages now. Will work on a profile.

Thanks!

Do you have a ticket on how to build from source code yet?

I honestly have never built unMessage. As we are constantly working on the code, we use pip install -e. When we tested on fresh installations for the releases, we also used pip because it's the easiest way to install it and we assumed most users would prefer that.

I am going to create a ticket for that!

In these comments you gave a pretty good guide. Perhaps copy it over to a readme?

That's a great idea. Although most of the issues we had here can be fixed and are related to this newer version, the docs should definitely be improved. I think I am going to use one of @HulaHoopWhonix's tickets as this is related to #5.

We still have to find out why we cannot use unMessage installed with python setup.py install. I just created a fresh vm and was able to reproduce the error you first reported. I might actually talk with incremental's dev, because that does not seem to be txtorcon's or Twisted's fault.

On Wed, Mar 08, 2017 at 06:44:16PM -0800, Patrick Schleizer wrote:

This profile goes further.

---
- exe-paths:
    - '*'
  users:
    - '*'
  hosts:
    - '*'
  commands:
    GETCONF:
     - '.+'
    GETINFO:
     - '.+'
    USEFEATURE:
     - '.+'
  events:
    STREAM:

Thanks @adrelanos, I am going to use that for my tests!

Might be the first application using txtorcon with cpfpy.

Indeed. I talked to @meejah (txtorcon's dev) these days and he was not aware of txtorcon being used with a cpf.

unMessage gets 510 Command filtered which must come from cpfpy. However, cpfpy's log does not include that it replied 510 Command filtered. We might have a bug in cpfpy where it replies 510 Command filtered but fails to which incoming command it denied and fails to log that it replied 510 Command filtered.

If we made a filter that allows any command, for now, would it work?

Is there something special that txtorcon should be doing?

If you can point me to a direction, it will make my life a lot easier to make txtorcon compatible with cpfpy.

Thanks!

If we made a filter that allows any command, for now, would it work?

Yes.

Use cpfpy with --complain. That allows all commands. Only logs them and lets them through. Do you know how to modify the cpfpy systemd unit to add this command line switch?

Just now tested. Works for me. No more python exceptions. Haven't actually chatted with anyone yet.

Is there something special that txtorcon should be doing?

Using as few Tor control protocol commands as possible. Not asking for so many GETCONF etc. At the moment it looks like on startup it asks Tor at startup for all states that the Tor control protocol provides. (It might not actually ask for all of them, but it looks like it asks for hundredths of them. That makes filtering and writing the profile very hard.

On Wed, Mar 08, 2017 at 07:58:32PM -0800, Patrick Schleizer wrote:

If we made a filter that allows any command, for now, would it work?

Yes.

Just now tested. Works for me. No more python exceptions. Haven't actually chatted with anyone yet.

Cool!

Use cpfpy with --complain. That allows all commands. Do you know how to modify the cpfpy systemd unit to add this command line switch?

I do not, but I'll take a look into how to do that. It looks like my Whonix 14 workstation template broke. I'm getting time sync issues and I tried to manually set the time, restart Tor, sdwdate, etc, but nothing worked :/

Using as few Tor control protocol commands as possible. Not asking for so many GETCONF etc. At the moment it looks like on startup it asks Tor at startup for all states that the Tor control protocol provides. (It might not actually ask for all of them, but it looks like it asks for hundredths of them.

Hmm got it. I will see what I can do about that.

Thanks @adrelanos!

Regarding all the GETCONFs, etc...That will be TorConfig starting; most of the information in txtorcon is derived by querying Tor (e.g. what config-items are available, etc). Some users find TorConfig "too magic", which is fair.

So, making "something different" might be interesting, especially with this use-case in mind (i.e. minimizing GETCONF requests). There's one weird corner case though, where I think you basically have to query all the HiddenServiceDir options because it's a "Virtual" kind and order-matters (so if you don't, then you can't do a SETCONF properly). Anyway, this new-thing could then take the place of TorConfig in the startup/launching code (most easily in the new launch).

This also might point to needed an API change in the new Tor object before I release it though: currently you can always (and synchronously) get a TorConfig instance via the .config property; this should maybe be create_config instead so it's explicit (and also because in the "new thing" case above there won't be one always).

If you want to play around a bit at a lower-level immediately, you could start with a TorControlProtocol by directly connecting to an endpoint using a TorProtocolFactory as the factory. This will only do some GETINFOs to determine what signals and events are available and the version.

Felipe Dau:

Use cpfpy with --complain. That allows all commands. Do you know how to modify the cpfpy systemd unit to add this command line switch?

I do not, but I'll take a look into how to do that.

In file: /lib/systemd/system/tor-controlport-filter.service.d/30_cpfpy.conf

Change from:

ExecStart=/usr/lib/tor-controlport-filter --debug --listen-interface eth1

to;

ExecStart=/usr/lib/tor-controlport-filter --debug --listen-interface eth1 --complain

sudo systemctl daemon-reload

sudo systemctl restart tor-controlport-filter

It looks like my Whonix 14 workstation template broke. I'm getting time sync issues and I tried to manually set the time, restart Tor, sdwdate, etc, but nothing worked :/

Please check '/var/log/sdwdate.log'.

Anyhow. By running "sudo whonix_firewall" it should should always enable full network access.

You'll need to allow the HS_DESC event for sure, too (so that txtorcon knows when the descriptor upload has succeeded to at least one dirauth after calling ADD_ONION). Note that it only listens for this if you use the ephemeral-onion stuff, so it won't show up in the filter's log until you do that...

Chatting worked for @felipedau and me.

Here is the cpfpy profile that I am using.

Note: this should not be used anywhere in production since it basically makes cpfpy not filter anything.

/etc/tor-controlport-filter-merger.d/40_test.yml

---
- exe-paths:
    - '*'
  users:
    - '*'
  hosts:
    - '*'
  commands:
    GETCONF:
     - '.+'
    GETINFO:
     - '.+'
    USEFEATURE:
     - '.+'
    ADD_ONION:
     - '.+'
    DEL_ONION:
     - '.+'
  events:
    STREAM:
    HS_DESC:

On Thu, Mar 09, 2017 at 07:13:29PM -0800, Patrick Schleizer wrote:

Chatting worked for @felipedau and me.

Here is the cpfpy profile that I am using.

Note: this should not be used anywhere in production since it basically makes cpfpy not filter anything.

That's great! Does that mean there is no need to use --complain then?

Thanks!

As I expect to have #24 merged in the next few days, I am going to close this issue when that happens and center all the remaining Whonix tasks in #2.

AnemoneLabs / unmessage

port to ephemeral Tor hidden services / python-stem #4