create_config, not .config

[meejah]

We should make the API for "get a TorConfig instance" async and call it .create_config, not .config on the Tor class.

Also, .connect and .launch should not use TorConfig at all. It is already "not really" exposed in the API (there is a _tor_config for tests (and .global_tor()?) but it shouldn't be used and the docs say that).

The use-case for this is cutting down on the number of GETCONF calls for "filtering proxies" (such as Whonix and SubGraph use to limit access to the tor control port). This is also the only "implied" object created, so it would be nice to eliminate it, making all additional "helper" object things that are created from Tor methods.

Thanks to @felipedau for bringing this use-case up. See also: https://github.com/AnemoneLabs/unmessage/issues/4#issuecomment-285561506

[meejah]

Hmm, there might be a couple problems with that:

• you can never use SETCONF safely without doing GETCONF first (for example if you "SETCONF SocksPort=1234" without any of the other ones configured before, you'll clobber any other SOCKS ports.
• the Tor controller object does use SETCONF for SOCKSPorts; this is for the APIs that allow you to specify a socks-port (e.g. the .socks_endpoint() factory-method).
• ...so "just use SETCONF" isn't very good, because it's "pretty low level" and easy to get wrong

So perhaps we really do want 2 variants of TorConfig: one that tries to do "minimal" GETCONF queries and one that doesn't.

But perhaps a different alternative is to just try/except around the GETCONF we have and just set it to some sentinel value (that causes an exception later if you try and change the value). This would let it work slightly better against a filtering proxy, but also would make it harder to know if (or which) GETCONF calls you'd have to whitelist for your application...

[meejah]

Other options:

• have a "don't query all values" option to TorConfig. This would necessarily have to then query at lease all options you changed (or produce an error) if you ever tried to .save().
• it would also be an error to attempt to access any option that wasn't queried

Yet another option that (maybe) directly addresses the use-case:

• try/except around the get_conf() and if we get a 500 error, assume it's filtered and mark that option as "not really available". So then it would be an error to ask for it, and an error to set it.

It would be useful to have some feedback from e.g. @adrelanos or other Whonix (or SubGraph) contributors on what would actually make TorConfig play more nicely with control-port filtering software.

[adrelanos]

meejah:

• have a "don't query all values" option to TorConfig.

Sounds good!

Yet another option that (maybe) directly addresses the use-case:

• try/except around the get_conf() and if we get a 500 error, assume it's filtered and mark that option as "not really available". So then it would be an error to ask for it, and an error to set it.

I like that less since that would still create lots of noise in control-port-filter-python logs.

[meejah]

Thanks for the feedback @adrelanos :)

I'll have to explore the "don't query everything" option a bit further since there might be a lot of sharp corners. Essentially: txtorcon does manipulate the config for certain things. I might just get rid of the "pass any SOCKS port you want and we'll create one" convenience-thing, but it still needs to ask for available SOCKS ports etc.

Additionally: is there any way for txtorcon to reliably tell "I'm running against a filtering proxy"? Then e.g. I could trigger the "minimal GETCONF" mode automatically and/or provide better error-messages when things do go wrong (e.g. "option not available" vs. "we think we're talking to a filtering proxy and it won't let us do 'GETCONF whatever'")

[meejah]

One suggestion: the answer to PROTOCOLINFO could include something in the VERSION lines without violating the control-spec I think.

e.g. something like:

<PROTOCOLINFO 1
>PROTOCOLINFO 1
>AUTH METHODS=NULL
>VERSION Tor="0.2.9.9" Proxy="control-port-filter-python 1.0.0"

It should be easy for a filtering-proxy to add the "Proxy=" kwarg since they're probably manipulating this response anyway (e.g. yours is to change the methods to NULL, didn't look at roflcoptr yet).

[felipedau]

So perhaps we really do want 2 variants of TorConfig: one that tries to do "minimal" GETCONF queries and one that doesn't.

I agree, that's a good idea. Do you think we can solve this issue mostly by changing the config code? I mean, could we just give Tor a different config and it might not even notice because it behaves like the regular one?

• have a "don't query all values" option to TorConfig. This would necessarily have to then query at lease all options you changed (or produce an error) if you ever tried to .save(). [...] Additionally: is there any way for txtorcon to reliably tell "I'm running against a filtering proxy"? Then e.g. I could trigger the "minimal GETCONF" mode automatically and/or provide better error-messages when things do go wrong (e.g. "option not available" vs. "we think we're talking to a filtering proxy and it won't let us do 'GETCONF whatever'")

I think that, right now, the first option could be implemented as it does not requires the filters to change. However, I do think that having such a convention for the filters is desirable and having txtorcon adapting to such situation would be great!

• it would also be an error to attempt to access any option that wasn't queried [...]
• try/except around the get_conf() and if we get a 500 error, assume it's filtered and mark that option as "not really available". So then it would be an error to ask for it, and an error to set it.

I think these would be nice improvements/optimizations to have... which maybe both configs can benefit from?

It would be useful to have some feedback from e.g. @adrelanos or other Whonix (or SubGraph) contributors on what would actually make TorConfig play more nicely with control-port filtering software.

@adrelanos would probably be able to tell us if these changes would work for other cpfs. I would like to mention that finding a solution to this will also make it possible to use txtorcon on Tails with its cpf, which is the one Whonix forked from.

I hope to dedicate more time to this issue this weekend.

Thanks @adrelanos and @meejah!

[adrelanos]

Felipe Dau:

So perhaps we really do want 2 variants of TorConfig: one that tries to do "minimal" GETCONF queries and one that doesn't.

I agree, that's a good idea. Do you think we can solve this issue mostly by changing the config code?

I mean, could we just give Tor a different config and it might not even notice because it behaves like the regular one?

I don't understand. Could you rephrase that please?

@adrelanos would probably be able to tell us if these changes would work for other cpfs. I would like to mention that finding a solution to this will also make it possible to use txtorcon on Tails with its cpf, which is the one Whonix forked from.

Tails and Whonix filter are only different wrt how cpfpy configuration files are parsed. Not noticeable from application level.

[adrelanos]

meejah:

Additionally: is there any way for txtorcon to reliably tell "I'm running against a filtering proxy"?

I don't think there currently is, but would probably not be hard to implement.

However, I guess it's best if at application level you won't have to distinguish talking to real Tor control port vs tor-controlport-filter. That would add another level of complexity.

Ideally we could solve all in the filter. By asking you "please don't ask for all existing GETCONFs even if you do not need them" as well as "ideally one Tor control protocol command at once (GETINFO version followed by GETINNFO circuit-established is better than a combined GETINFO version circuit-established because then the filter rules look cleaner" is already a lot to ask. And nice of you considering to implement that!

"ask for all existing GETCONFs" seems most useful for Tor controllers such as 'arm' / 'nyx'. Not so much for applications with a more limited scope as in creating, deleting onions and looking up related status's.

//cc @fred-a-kemp (author of tor-controlport-filter by Tails)

[felipedau]

I don't understand. Could you rephrase that please?

Sure. The TorConfig class is responsible by making the calls to the control port and that's what we wish to change. What I'm asking is if we would have to adapt the Tor class to know how to deal with this "light-weight" TorConfig or if just by having an interface which both configs implement would suffice.

Tails and Whonix filter are only different wrt how cpfpy configuration files are parsed. Not noticeable from application level.

Oh, good to hear that! Thanks :)

[felipedau]

Additionally: is there any way for txtorcon to reliably tell "I'm running against a filtering proxy"? Then e.g. I could trigger the "minimal GETCONF" mode automatically and/or provide better error-messages when things do go wrong [...] One suggestion: the answer to PROTOCOLINFO could include something in the VERSION lines without violating the control-spec I think.

What if we came up with a command that is not part of the protocol which we would get Unrecognized command without a filter but OK with a filter and then enable the "minimal" mode?

have a "don't query all values" option to TorConfig. This would necessarily have to then query at lease all options you changed (or produce an error) if you ever tried to .save().

What if there was a TorConfig.is_minimal boolean attribute (set either manually or automatically) which would be checked by TorConfig.bootstrap to decide if self.protocol.get_info_raw("config/names") should be called or get only the minimal set of configs we need?

[adrelanos]

I still think detecting the filter and acting different is the wrong approach.

Why not go minimal by default, and ask for other stuff when needed? Then all the filtering could remain abstracted in the filter.

Another idea I had in mind is tricking txtorcon. When it asks config/names, only reply with a limited amount of names. Do you think that would work or is a sane approach?

Felipe Dau:

Additionally: is there any way for txtorcon to reliably tell "I'm running against a filtering proxy"? Then e.g. I could trigger the "minimal GETCONF" mode automatically and/or provide better error-messages when things do go wrong [...] One suggestion: the answer to PROTOCOLINFO could include something in the VERSION lines without violating the control-spec I think.

What if we came up with a command that is not part of the protocol which we would get Unrecognized command without a filter but OK with a filter and then enable the "minimal" mode?

Sure.

You'd probably also wanted to make sure it also works when using tor-controlport-filter --complain.

[meejah]

Why not go minimal by default, and ask for other stuff when needed? Then all the filtering could remain abstracted in the filter.

That breaks the existing API of TorConfig.

This is why I was presenting it as "maybe I could make a different implementation" because current users of TorConfig would all break if I made the API a deferred-using one. That said, some people would prefer a "more explicit" API like this.

Another idea I had in mind is tricking txtorcon. When it asks config/names, only reply with a limited amount of names. Do you think that would work or is a sane approach?

Hmm, very interesting!

Yes, that should work (at least as far as: txtorcon will then only ask for those config-items). However, some things will break if certain items aren't in there (e.g. without SOCKSPort it can't ask for that, and does do that sometimes). I guess you would just make up this list from any whitelisted GETCONF items? I'll try hacking the filter to do this and see how it goes. Great idea!

This is definitely way better than "special case" behavior in txtorcon (i.e. "am I connecting to a filter"). It also seems "more correct" to me as far as the control-spec.

[adrelanos]

meejah:

Why not go minimal by default, and ask for other stuff when needed? Then all the filtering could remain abstracted in the filter.

That breaks the existing API of TorConfig.

This is why I was presenting it as "maybe I could make a different implementation" because current users of TorConfig would all break if I made the API a deferred-using one. That said, some people would prefer a "more explicit" API like this.

I see. Yes, if you could implement a new "minimal" TorConfig that would be awesome. Next step would then be contacting the application developers who use the "full" TorConfig and ask them to port to the new "minimal" TorConfig if that is possible in their use case.

(I imagine porting to a new "minimal" TorConfig will be possible for most if not all applications that matter in case of Whonix, i.e. @unMessage, @GlobaLeaks are not Tor controllers so they need fewer Tor control protocol commands, therefore could use the "minimal" TorConfig.

That seems to me on a coding level like a clean implementation. Thank you for your interest to implement this!

Another idea I had in mind is tricking txtorcon. When it asks config/names, only reply with a limited amount of names. Do you think that would work or is a sane approach?

Hmm, very interesting!

Yes, that should work (at least as far as: txtorcon will then only ask for those config-items).

Great!

However, some things will break if certain items aren't in there (e.g. without SOCKSPort it can't ask for that, and does do that sometimes).

We could leave SOCKSPort inside config/names, no problem. We can even rewrite the reply that the client is getting for SOCKSPort.

Example of doing that here:

https://github.com/Whonix/control-port-filter-python/blob/master/etc/tor-controlport-filter-merger.d/30_whonix.yml

Btw other tor controlport filter examples here:

https://github.com/Whonix/control-port-filter-python/tree/master/usr/share/tor-controlport-filter-merger/examples

I guess you would just make up this list from any whitelisted GETCONF items?

Probably yes. Whitelisting all GETCONF items that are really required. And then manually add them to a list for config/names and add that to tor controlport filter profile.

(I mean, I wouldn't auto generate the config/names list within the python code. That level of complexity should not be needed.)

I'll try hacking the filter to do this and see how it goes. Great idea!

This is definitely way better than "special case" behavior in txtorcon (i.e. "am I connecting to a filter"). It also seems "more correct" to me as far as the control-spec.

Great!

[felipedau]

I guess you would just make up this list from any whitelisted GETCONF items?

Probably yes. Whitelisting all GETCONF items that are really required. And then manually add them to a list for config/names and add that to tor controlport filter profile.

That's a great idea! Would that be the default behavior of the filter? Do you think people might not like this feature because they would be confused thinking configs are not available, but are actually just "pre-filtered"? Well, that until they learn to deal with cpfs...

This is why I was presenting it as "maybe I could make a different implementation" because current users of TorConfig would all break if I made the API a deferred-using one. That said, some people would prefer a "more explicit" API like this.

And do you think it is worth supporting the minimal/deferred approach? I think it is and agree with @adrelanos that minimal should be used and request on demand, but maybe too much work while the trick presented by him solves the problem (for now?).

If the "minimal" feature should proceed, then it could be optional while full would still be the default mode and announce that txtorcon x.y.z will swap them and full would become optional and minimal the default. And as suggested by @adrelanos, contact the people we know are using it.

[adrelanos]

The default filtering approach in Whonix will be to enable the minimum for client use only (no hidden services). i.e.:

https://github.com/Whonix/control-port-filter-python/blob/master/etc/tor-controlport-filter-merger.d/30_whonix.yml

Users that wish to use applications that require other Tor control protocol commands such as onionshare, unMessage etc. will have to follow Whonix documentation and enable the profile as of Whonix 14.

For Non-Qubes-Whonix 15 I am thinking about a first boot whonix setup wizard question "allow onions?" (phrasing this is still todo) and then allow all applications we have profiles for (copy them for the user from the examples to the actual config folder) to simplify things.

Qubes-Whonix 15 I am thinking about an shell wrappers in front of onionshare, unMessage etc. that on start of onionshare etc. ask "enable onionshare profile in sys-whonix" and therefore simplify that for the user.

Btw that's one reason for the stackable wrappers draft. https://phabricator.whonix.org/T634

Would that be the default behavior of the filter?

The behavior when the unMessage tor controlport filter profile got enabled.

Do you think people might not like this feature because they would be confused thinking configs are not available, but are actually just "pre-filtered"?

For any application that needs more GETCONFS, they'd need to enable a tor controlport filter profile for that application. That should then be a higher priority configuration file (lexically later file name) and overwrite the small config/names list with a longer one. Exotic combinations of random profiles may not be possible without profile modifications.

Profiles will always be per application. A one fits all may not be possible without sacrifice of security. Perhaps we could move away from per-application profiles and make that

• a) minimal, client only
• b) extended client only (not sure if there are any use cases yet)
• c) few add_onion related only
• d) more add_onion related commands
• e) even more relaxed
• f) everything

Or simpler, just two modes:

• a) minimal, client only
• b) more add_onion related commands

[felipedau]

For Non-Qubes-Whonix 15 I am thinking about a first boot whonix setup wizard question "allow onions?" (phrasing this is still todo) and then allow all applications we have profiles for (copy them for the user from the examples to the actual config folder) to simplify things.

It would be great to have an app for that!

Qubes-Whonix 15 I am thinking about an shell wrappers in front of onionshare, unMessage etc. that on start of onionshare etc. ask "enable onionshare profile in sys-whonix" and therefore simplify that for the user.

Btw that's one reason for the stackable wrappers draft. https://phabricator.whonix.org/T634

I really like this idea! A great UX improvement.

Would that be the default behavior of the filter?

The behavior when the unMessage tor controlport filter profile got enabled.

The behavior I meant was that the filter's reply to config/names would always be based on the profiles currently enabled. Is that the case?

Profiles will always be per application. A one fits all may not be possible without sacrifice of security. Perhaps we could move away from per-application profiles and make that

a) minimal, client only
b) extended client only (not sure if there are any use cases yet)
c) few add_onion related only
d) more add_onion related commands
e) even more relaxed
f) everything

Or simpler, just two modes:

a) minimal, client only
b) more add_onion related commands

I also like this idea - and customizing it with an application as you suggested above would be awesome. The only difficulty I see would be handling the different ports each app uses. How would that work?

Thanks!

[adrelanos]

Felipe Dau:

Would that be the default behavior of the filter?

The behavior when the unMessage tor controlport filter profile got enabled.

The behavior I meant was that the filter's reply to config/names would always be based on the profiles currently enabled. Is that the case?

The reply to config/names would always be dependent on the activated tor-controlport-filter profile(s), kinda yes. But maybe not as you might think. Easy to talk past each other here. No code changes in tor-controlport-filter. Implemented as tor-controlport-filter profile only.

Here is the pseudo profile part for unmessage.yml (untested):

---
- exe-paths:
    - '*'
  users:
    - '*'
  hosts:
    - '*'
  commands:
    GETINFO:
      - 'config/names'
        - pattern:     '.*'
          replacement: 'TODO'

No more automatic than that.

Then indeed multiple txtorcon based applications requiring different replies for config/names would be tricky to implement in the profile. (In Whonix) the higher priority config file's config/names would overwrite the lower priority config file. If it disabled anything, it would break the application. I guess there we hit the boundaries of merging diverse complex filter profiles.

(Such a conflict could of course be manually solved by adding yet another higher priority profile that allows the merged set of config/names.)

Perhaps it would indeed be a nice feature to automatically reply to config/names the white listed commands under GETINFO. I.e. to not have 'config/names' in the profile, but have this handled automatically by tor-controlport-filter.

What do you think about this, @fred-a-kemp? (anonym, author of tor-controlport-filter)

Profiles will always be per application. A one fits all may not be possible without sacrifice of security. Perhaps we could move away from per-application profiles and make that

a) minimal, client only b) extended client only (not sure if there are any use cases yet) c) few add_onion related only d) more add_onion related commands e) even more relaxed f) everything

Or simpler, just two modes:

a) minimal, client only b) more add_onion related commands

I also like this idea - and customizing it with an application as you suggested above would be awesome. The only difficulty I see would be handling the different ports each app uses. How would that work?

In Tails: a lot better than in Whonix. They can use

- exe-paths:
    - '/usr/bin/unmessage'

etc. Something that cannot be used in Whonix due to gateway / workstation split model, where the exe-paths information gets lost. (That's why it's set to * in Whonix.)

In Whonix: where we merge all profiles from /etc/tor-controlport-filter-merger.d/ to /etc/tor-controlport-filter.d/30_autogenerated.yml it's a bit more complex.

onionshare

    ADD_ONION:
      - pattern:     'NEW:BEST Port=80,(176[0-5][0-9])'
        replacement: 'NEW:BEST Port=80,{client-address}:{} Flags=DiscardPK'

and zeronet

    ADD_ONION:
      - pattern:     'NEW:(\S+) Port=15441,15441'
        replacement: 'NEW:{} Port=15441,{client-address}:15441'

get merged to

    ADD_ONION:
      - pattern:     'NEW:BEST Port=80,(176[0-5][0-9])'
        replacement: 'NEW:BEST Port=80,{client-address}:{} Flags=DiscardPK'
    ADD_ONION:
      - pattern:     'NEW:(\S+) Port=15441,15441'
        replacement: 'NEW:{} Port=15441,{client-address}:15441'

Lucky we these won't conflict. But yeah, if in future loads of applications do all sorts of things, we could end up in a situation where one pattern / replacement could be incompatible with another. Kinda complex to explain, but that's a reason why I am kindly asking application developers for distinct port ranges.

[anonym]

Perhaps it would indeed be a nice feature to automatically reply to config/names the white listed commands under GETINFO. I.e. to not have 'config/names' in the profile, but have this handled automatically by tor-controlport-filter.

What do you think about this, @fred-a-kemp? (anonym, author of tor-controlport-filter)

Yeah, this makes sense. IMHO we should only special-case exactly GETINFO config/names, and not arbitrary GETINFO calls where config/names is just one of the arguments -- in this latter case some of the other arguments might be filtered, so our answer should be partially filtered, and I have a hard time coming up with something that feels safe and sane for that.

So, let's only special-case exactly GETINFO config/names (and always allow it, no matter the filter file). Sorry for my hostility to GitHub, but you'll have to fetch my fix from the feature/getinfo-config-names branch in the new upstream repo (note the name change!) found at: https://git-tails.immerda.ch/onion-grater (web interface is broken, but git clone works). Let me know what you think!

[meejah]

Perhaps it would indeed be a nice feature to automatically reply to config/names the white listed commands under GETINFO. I.e. to not have 'config/names' in the profile, but have this handled automatically by tor-controlport-filter.

To me this sounds like a good way as well.

Similarly, I think you should always allow "GETINFO info/names" and auto-reply with anything whitelisted under 'GETINFO'

For txtorcon specifically, everything is built up from info/names and config/names so downstream applications should get plausible errors then (e.g. "some/key" is not available).

[adrelanos]

fred-a-kemp:

So, let's only special-case exactly GETINFO config/names (and always allow it, no matter the filter file). Sorry for my hostility to GitHub, but you'll have to fetch my fix from the feature/getinfo-config-names branch in the new upstream repo (note the name change!) found at: https://git-tails.immerda.ch/onion-grater (web interface is broken, but git clone works). Let me know what you think!

Great!

Looks like you branched off an outdated branch? (missing python -u, --listen-interface, etc.)

[anonym]

Patrick Schleizer:

fred-a-kemp:

So, let's only special-case exactly GETINFO config/names (and always allow it, no matter the filter file). Sorry for my hostility to GitHub, but you'll have to fetch my fix from the feature/getinfo-config-names branch in the new upstream repo (note the name change!) found at: https://git-tails.immerda.ch/onion-grater (web interface is broken, but git clone works). Let me know what you think!

Great!

Looks like you branched off an outdated branch? (missing python -u, --listen-interface, etc.)

Right. I force pushed the branch rebased on master, so please fetch and adjust!

[adrelanos]

---
- commands:
    GETCONF:
    - SocksPort
    GETINFO:
    - events/names
    - signal/names
    - config/names
    - status/circuit-established
    - version
    - pattern: net/listeners/socks
      response:
      - pattern: 250-net/listeners/socks=".*"
        replacement: 250-net/listeners/socks="127.0.0.1:9150"
    - consensus/valid-after
    - consensus/valid-until
    - consensus/fresh-until
    SIGNAL:
    - NEWNYM
    USEFEATURE:
    - EXTENDED_EVENTS
  confs:
    __owningcontrollerprocess: null
  events:
    CONF_CHANGED:
      suppress: true
    SIGNAL:
      suppress: true
    STREAM: null
  exe-paths:
  - '*'
  hosts:
  - '*'
  name: 'merged_filter_files:  40_unmessage.yml 30_whonix.yml'
  users:
  - '*'

Your patch seems to work, thank you @fred-a-kemp.

Mar 29 12:43:33 host systemd[1]: Stopping Tor control port filter proxy...
Mar 29 12:43:33 host systemd[1]: Stopped Tor control port filter proxy.
Mar 29 12:43:33 host systemd[1]: Starting Tor control port filter proxy...
Mar 29 12:43:33 host tor-controlport-filter[7358]: IP address for interface eth1 : 10.137.11.1
Mar 29 12:43:33 host tor-controlport-filter[7358]: Tor control port filter started, listening on 10.137.11.1:9051
Mar 29 12:43:33 host tor-controlport-filter[7358]: Serving Thread started
Mar 29 12:43:34 host systemd[1]: Started Tor control port filter proxy.
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated) connected: loaded filter: 30_autogenerated
Mar 29 12:43:39 host tor-controlport-filter[7358]: Final rules:
Mar 29 12:43:39 host tor-controlport-filter[7358]: commands:
Mar 29 12:43:39 host tor-controlport-filter[7358]:   GETCONF:
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: SocksPort}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: (__owningcontrollerprocess)}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   GETINFO:
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: events/names}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: signal/names}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: config/names}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: status/circuit-established}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: version}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - pattern: net/listeners/socks
Mar 29 12:43:39 host tor-controlport-filter[7358]:     response:
Mar 29 12:43:39 host tor-controlport-filter[7358]:     - {pattern: 250-net/listeners/socks=".*", replacement: '250-net/listeners/socks="127.0.0.1:9150"'}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: consensus/valid-after}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: consensus/valid-until}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: consensus/fresh-until}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   SIGNAL:
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: NEWNYM}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   USEFEATURE:
Mar 29 12:43:39 host tor-controlport-filter[7358]:   - {pattern: EXTENDED_EVENTS}
Mar 29 12:43:39 host tor-controlport-filter[7358]: events:
Mar 29 12:43:39 host tor-controlport-filter[7358]:   CONF_CHANGED: {suppress: true}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   SIGNAL: {suppress: true}
Mar 29 12:43:39 host tor-controlport-filter[7358]:   STREAM: {}
Mar 29 12:43:39 host tor-controlport-filter[7358]: restrict-stream-events: false
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): -> PROTOCOLINFO 1
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- 250-PROTOCOLINFO 1
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- 250-AUTH METHODS=NULL
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- 250-VERSION Tor="0.2.9.10 (git-e28303bcf90b842d)"
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- 250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): -> AUTHENTICATE
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- 250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): -> GETINFO signal/names
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- (multi-line)
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250-signal/names=RELOAD HUP SHUTDOWN DUMP USR1 DEBUG USR2 HALT TERM INT NEWNYM CLEARDNSCACHE HEARTBEAT
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): -> GETINFO version
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- (multi-line)
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250-version=0.2.9.10 (git-e28303bcf90b842d)
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): -> GETINFO events/names
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- (multi-line)
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250-events/names=CIRC CIRC_MINOR STREAM ORCONN BW DEBUG INFO NOTICE WARN ERR NEWDESC ADDRMAP AUTHDIR_NEWDESCS DESCCHANGED NS STATUS_GENERAL STATUS_CLIENT STATUS_SERVER GUARD STREAM_BW CLIENTS_SEEN NEWCONSENSUS BUILDTIMEOUT_SET SIGNAL CONF_CHANGED CONN_BW CELL_STATS TB_EMPTY CIRC_BW TRANSPORT_LAUNCHED HS_DESC HS_DESC_CONTENT NETWORK_LIVENESS
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): -> USEFEATURE EXTENDED_EVENTS
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- 250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): -> SETEVENTS CONF_CHANGED
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): suppressed subscription to event 'CONF_CHANGED'
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- 250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): -> GETINFO config/names
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): rewrote response:
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250+config/names=
Mar 29 12:43:39 host tor-controlport-filter[7358]:     AccountingMax DataSize
Mar 29 12:43:39 host tor-controlport-filter[7358]:     AccountingRule String
Mar 29 12:43:39 host tor-controlport-filter[7358]:     AccountingStart String
Mar 29 12:43:39 host tor-controlport-filter[7358]:     Address String
...
...
...
Mar 29 12:43:39 host tor-controlport-filter[7358]:     TestingDirAuthVoteHSDir RouterList
Mar 29 12:43:39 host tor-controlport-filter[7358]:     TestingDirAuthVoteHSDirIsStrict Boolean
Mar 29 12:43:39 host tor-controlport-filter[7358]:     .
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: to:
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250+config/names=
Mar 29 12:43:39 host tor-controlport-filter[7358]:     SocksPort LineList
Mar 29 12:43:39 host tor-controlport-filter[7358]:     .
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250 OK
Mar 29 12:43:39 host tor-controlport-filter[7358]: 10.137.11.80:44624 (filter: 30_autogenerated): <- (multi-line)
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250+config/names=
Mar 29 12:43:39 host tor-controlport-filter[7358]:     SocksPort LineList
Mar 29 12:43:39 host tor-controlport-filter[7358]:     .
Mar 29 12:43:39 host tor-controlport-filter[7358]:     250 OK

However, either txtorcon gets confused by that or unMessage does not use add_onion due to some bug?

[meejah]

I just tried the above config (using 'carml') after a couple hacks to the filter code, but: ADD_ONION isn't in the yaml above.

Adding the below config allows this command to succeed: carml --connect oniongraterport cmd ADD_ONION NEW:BEST 'Port=80,127.0.0.1:80'. I haven't yet tried unmessage itself, nor the higher-level txtorcon APIs.

    ADD_ONION:
    - '.*'

[meejah]

Hmm, it looks like "something is hanging" when doing GETINFO config/names. Just with carml --connect oniongraterport cmd GETINFO config/names the filter never gives an answer.

I think it's actually simple to fix: control-port lines must end in \r\n not just \n so ~line 428 in onion-gater needs '\r\n'.join(response). @fred-a-kemp

[meejah]

You also need to allow the HS_DESC event.

[meejah]

Okay, with those changes (including HS_DESC) the release-1.x branch's web_onion_service.py example (which uses ADD_ONION) works successfully.

[felipedau]

Following @meejah's suggestions allowed me to run unMessage on Whonix. It was possible to send and receive connections. I just pushed a change (felipedau/unmessage@c51b9815f3e8c8771e9ea5548bedcac1689d0baa) that properly maps the hs to the correct (external) IP address. The tests we ran previously worked because we only tested making the connections from the Whonix VM - not receiving!

Thanks guys!

[adrelanos]

@meejah

I think it's actually simple to fix: control-port lines must end in \r\n not just \n so ~line 428 in onion-gater needs '\r\n'.join(response).

Does https://github.com/Whonix/onion-grater/commit/30c1de54f9feaa26464842241e217be6edf3b464 look alright?

If it is not too much to ask, while you are at it, could you please have a brief look if the filter otherwise looks alright wrt \n vs \r\n as well as other things?

[adrelanos]

create an unMessage onion-grater profile https://phabricator.whonix.org/T654

[meejah]

@adrelanos yes that change looks good for onion-grater

[meejah]

p.s. related to this original issue, PR #220 changes this API to "get_config". I'm still a little conflicted over calling this "create_config" or "get_config". Reasons:

get_config:

• you get the same one back on subsequent calls

create_config:

• possibly "more consistent" with other things (create_state) but the semantics are different (create_state always returns a brand new TorState instance).

Of course, I could make it create_config and always return a new TorConfig object. Although multiple config objects can keep in sync find when talking to one Tor, there are some "problems" with the attacher-stuff in tor (basically: it doesn't work when multiple controllers try to become stream-attachers, but also doesn't fail nicely). So, I'd like to keep "TorConfig" and "Tor" instances 1:1 since that seems safest (unless you "do something special" to break that, which is perfectly fine, but shouldn't be easy to do 'accidentally').

[felipedau]

I agree with the name you chose to use and IMO you shouldn't rename it. As a user, I would expect create_ to always return something different and get_ to always return the same thing even if on the first call it creates that. It kind of looks like how a singleton is used, I guess.

I am not sure if people would need to create new configs, so I would not bother about changing the behavior as well. Can you think of an example that would be used?

Maybe create_config could be provided as well so that it removes the current config and calls get_config to return the new one to kind of reset it, but I'm unsure of its usefulness. Is there a reason for someone recreating a TorConfig?

Thanks!

[meejah]

@felipedau Thanks for the feedback :) No, I can't think of any reason someone would need more than 1 TorConfig per tor instance they're talking to, especially in one process.

[meejah]

The code changing the Tor API to get_config is now on master. Also, a Tor object doesn't use/have a TorConfig unless you created one explicitly so there shouldn't be a surprising number of GETCONFs any more.

[felipedau]

Thanks @meejah, with these changes I was also able to use unMessage on both Whonix 14 (Debian 9) and non-Whonix (Debian 8) systems and successfully make and receive connections.

[meejah]

Great, thanks for testing @felipedau !