MQTTS client with SOCKS5 proxy support (IDFGH-9303)

[sramrajkar]

Is your feature request related to a problem?

ESP32 based product is going to be deployed in an corporate network with MQTTS client communicating with AWS broker using SSL on port 8883. The client has indicated that they will want us to use SOCKS5 proxy server they have onsite to reach our remote broker and will not open a port for us in the firewall. Can you please look into adding SOCK5 support to the MQTT(S) stack.

Describe the solution you'd like.

Ideally adding a .proxy element to the MQTT config structure just like .uri for the broker should handle SOCKS5 proxy for the MQTTS connection in the IDF component

Describe alternatives you've considered.

Not many options exist in C to port this over. One option is to use an HTTP CONNECT to use HTTP proxy then provide this to the MQTT client over WSS. Just an idea and we have not tried this. Also not sure if customer will allow HTTP proxy.

Additional context.

No response

[euripedesrocha]

Hi @sramrajkar we have this feature planned to be released later this year. Could you provide some information regarding your use case, in particular the authentication method for the proxy?

[sramrajkar]

This is great news. As this is a configurable gateway product we do not know exact use case per site. But we can start with None and User/Password for now.

[huming2207]

Hey @euripedesrocha

I also have similar requests for adding proxy supports for WebSocket. I've made a simple HTTP proxy transport here: https://github.com/huming2207/esp-tcp-transport-http-proxy

I'm working on an intermediate solution (aka. dirty hack) for now for WebSocket and MQTT client. I will add a proxy configuration field in the MQTT config struct. If the config exists, it will then connect to the proxy and use it. This will be out of the ESP's official source tree as it's a dirty hack for now

Later I will submit a few pull requests that are:

1. Add my HTTP proxy transport into ESP-IDF's tcp_transport, this PR may be submitted in this week or so;
2. Somehow implement a more generic way to configure the proxy (maybe a factory pattern thing to set that up) for all TCP transport handles - this may require quite a few mods for esp_mqtt_client_create_transport() and esp_websocket_client_create_transport() in those two libraries. If you have some better solutions, please point it out.

Regards, Jackson

[euripedesrocha]

@huming2207 Thanks for the heads-up! And we appreciate the contribution.

We are planning to have a way for user to inject custom transport with a new API or in the config struct. The easiest way is to add a transport handler to the config and use it and avoid the creation of the transports if present. I need to review the code, but I think a similar solution is possible on websocket client.

[huming2207]

Hi @euripedesrocha

Here's my quick dirty hack and I haven't tested that yet: https://github.com/huming2207/esp-mqtt/tree/feature/naive-proxy-support

The easiest way is to add a transport handler to the config and use it and avoid the creation of the transports if present. I need to review the code, but I think a similar solution is possible on websocket client.

Yep, so far I think the highest priority is to allow transport_ssl to support the parent transport handle. That will at least bring up support for something like WSS (WS over TLS) over HTTPS (HTTP over another TLS) proxy, or anything over TLS over a SOCKS proxy. This is useful for ESP32 to run in some cooperate networks with a security audit proxy, or let someone set up a MITM proxy for debugging protocols. However this only resolves the tcp_transport side and we need something extra to fit into WebSocket client, HTTP client and MQTT client. I will have a think tonight and see what I can do.

Jackson

[euripedesrocha]

I will draft something internally today, be aware that our internal review system take sometime, for the mqtt client providing a way to inject custom transport.

I would like to avoid adding new default transports to mqtt client, my reasoning here is to avoid having more configuration fields for the client, that struct is already big enough.

[huming2207]

Hi @euripedesrocha

We've implemented some external hacks for tcp_transport to enable HTTP/HTTPS proxy support recently. Here's the stuff we've done so far: https://github.com/huming2207/esp-tcp-transport-http-proxy/tree/5846ad189d21ad0e6c5aa860f8a5c9c95444d38f

We haven't got a chance to test it yet and we are going to do that today.

Here are some of my suggestions we probably need to change on mainline ESP-IDF:

1. Allow WebSocket/MQTT/HTTP client to take external tcp_transport handles. Basically:
   • We create a proxy-enabled tcp_transport handle outside by using the proxy library's init function, for example, mine is this one: https://github.com/huming2207/esp-tcp-transport-http-proxy/blob/5846ad189d21ad0e6c5aa860f8a5c9c95444d38f/esp_transport_http_proxy.h#L33
   • We add a field in the client's configuration struct, for example, in mqtt_config_storage_t for MQTT client, we add in a field esp_transport_handle_t ext_transport. If this ext_transport is not null, the client uses this external proxy handle instead of creating its own plain TCP or TLS handle.
2. Expose some of the useful fields in tcp_transport in the esp_transport_internal.h header, so that allows user to create their own tcp_transport layers. Currently, I'm doing a dirty hack by copying out that header in my own project, see here: https://github.com/huming2207/esp-tcp-transport-http-proxy/blob/5846ad189d21ad0e6c5aa860f8a5c9c95444d38f/dirty_hack/esp_transport_internal.h

If you are happy about my suggestions, I will write up a PR later.

Regards, Jackson

[huming2207]

Another off-topic reason why I strongly suggest we should expose some private fields in tcp_transport is, we then can extend these transport layers stuff to something else, not only just TCP, TLS and/or proxies, but also things like KCP, some custom UART/CAN/I2C-based networking protocols etc. and with the help of these transport layers we can reuse the code as much as possible.

In the future, I'd also suggest we should move these transport layer stuff to C++ to some extent so that we can use the object-oriented features to clean up the code.

[huming2207]

@euripedesrocha here's a quick follow-up, we've implemented our HTTP/HTTPS proxy and we've tested it working fine with our internal HTTPS proxy server (an Apache + Traefik instance with Let's Encrypt stuff) and it works fine.

Please see here for the HTTP proxy component: https://github.com/huming2207/esp-tcp-transport-http-proxy

Right now I'm going to write up some PRs to allow MQTT and WebSocket clients to use external tcp_transport handles. Please review them soon. Thanks.