Make it possible for `azure-eventhub` to work through an HTTP tunnel

[dikuchan]

Is your feature request related to a problem? Please describe. I'm trying to set up an EventHub consumer that uses an HTTP proxy. Unfortunately, the proprietary proxy server I'm using doesn't support WebSocket protocol, and I suppose that azure-eventhub uses this protocol internally. However, this proxy server supports HTTP tunneling. But it seems that azure-eventhub doesn't support the latter.

For testing, I tried to set up a simple HTTP tunnel using the http-tunnel project:

$ http-tunnel --bind 127.0.0.1:8080 http

Then I passed the address to the EventHubConsumer:

# main.py

from azure.eventhub import EventData, EventHubConsumerClient, PartitionContext
from pydantic import SecretStr
from pydantic_settings import BaseSettings
from structlog import get_logger

LOGGER = get_logger()

class Config(BaseSettings):
    connection_string: SecretStr
    eventhub_name: str
    consumer_group: str

def run(cfg: Config) -> None:
    with EventHubConsumerClient.from_connection_string(
        cfg.connection_string.get_secret_value(),
        consumer_group=cfg.consumer_group,
        eventhub_name=cfg.eventhub_name,
        logging_enable=True,
        http_proxy={
            "proxy_hostname": "127.0.0.1",
            "proxy_port": 8080,
        },
    ) as consumer:
        consumer.receive(
            on_event=on_event,
            starting_position="-1",
        )

def on_event(ctx: PartitionContext, event_data: EventData | None) -> None:
    if event_data is None:
        LOGGER.warning("no event data")
        return
    event = EventData.body_as_str(event_data)
    LOGGER.info("received an event", event=event)

if __name__ == "__main__":
    run(Config())

When running the code above, I received the following error message:

$ python main.py 
EventProcessor instance '34387061-8b9b-4272-8e6a-31bf79eb50d6' of eventhub '***' consumer group '***'. An error occurred while load-balancing and claiming ownership. The exception is AuthenticationError('CBS Token authentication failed.\nStatus code: None\nError: client-error\nCBS Token authentication failed.\nStatus code: None'). Retrying after 33.691810080799996 seconds

I tried to use uamqp:

@@ -22,6 +22,7 @@
             "proxy_hostname": "127.0.0.1",
             "proxy_port": 8080,
         },
+        uamqp_transport=True,
     ) as consumer:
         consumer.receive(
             on_event=on_event,

And interestingly enough, the error has changed:

$ python main.py
Authentication Put-Token failed. Retries exhausted.
Authentication Put-Token failed. Retries exhausted.
Authentication Put-Token failed. Retries exhausted.

Describe the solution you'd like I'd like to use the HTTP tunnel seamlessly. For example, the httpx library allows you to use tunnels. This method has the same interface as a regular forward proxy:

# httpx_client.py

import httpx

if __name__ == "__main__":
    with httpx.Client(proxy="http://127.0.0.1:8080") as client:
        response = client.get("https://httpbin.org/get")
        response.raise_for_status()
        print("OK", response.status_code)

$ python httpx_client.py
OK 200

[github-actions[bot]]

Thank you for your feedback. Tagging and routing to the team member best able to assist.

[kristapratico]

@dikuchan thanks for your issue, the team will take a look and get back to you as soon as possible.

[l0lawrence]

Hi @dikuchan, from the snippets you listed above it looks like you are hitting an authentication error from the service. This seems independent of the connection, would you be able to confirm that you can set up a consumer and receive without using a proxy?

[github-actions[bot]]

Hi @dikuchan. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[dikuchan]

Hi @l0lawrence! You're right, the problem was with authentication. My bad. I accidentally changed the eventhub_name when I tried to use the HTTP tunnel. When I changed it to the correct one, everything worked fine.

Sorry to bother you with such a stupid problem. I'm closing the issue now.

[l0lawrence]

Not stupid at all, I'm glad you figured it out. thanks!