How can I access virtual host using rhea with TLS?

[sossnowski]

I am able to connect now but I am not able to send something to rabbitmq (client is only authenticated but access to virtual host is not granted). Rabbitmq log: 2023-01-12 14:17:13.577765+00:00 [info] <0.5611.0> accepting AMQP connection <0.5611.0> (172.18.0.1:46954 -> 172.18.0.3:5671) 2023-01-12 14:17:13.583163+00:00 [info] <0.5611.0> AMQP 1.0 connection <0.5611.0>: user '63ec2047-6689-45c0-981d-f9b127a6bb7f' authenticated

I am using the same server with python pika and it working correctly: 2023-01-12 14:18:29.938260+00:00 [info] <0.5640.0> accepting AMQP connection <0.5640.0> (172.18.0.1:54738 -> 172.18.0.3:5671) 2023-01-12 14:18:29.943712+00:00 [info] <0.5640.0> connection <0.5640.0> (172.18.0.1:54738 -> 172.18.0.3:5671): user '63ec2047-6689-45c0-981d-f9b127a6bb7f' authenticated and granted access to vhost '/'

How can I manage? I can't see params to select virtual host in docs.

[grs]

I suspect pika is using an older version of AMQP. AMQP 1.0 has no equivalent of the old 'virtual host'. According to the rabbit mq docs you should use a prefixed value for the hostname in connection open: https://github.com/rabbitmq/rabbitmq-server/tree/main/deps/rabbitmq_amqp1_0#virtual-hosts

[sossnowski]

Thank you for your reply. It is as you aid Pika is using version 0.9. But I have another problem then. I am using rabbitmq and keycloak with rabbitmq rabbit_auth_backend_oauth2 plugin to authenticate users. In version 0.9 of protocol AMQP it was done by scope field in token (in version 0.9 I have "scope": "rabbitmq.read:*/* rabbitmq.write:*/* rabbitmq.configure:*/*",) and this was enough to grant access to virtual host. So now in version 1.0 what is equivalent of this key? How can I grant permission for user to read and write from my rabbitmq?

Logs in rabbitmq now: 2023-01-13 10:08:30.817765+00:00 [info] <0.1010.0> accepting AMQP connection <0.1010.0> (172.18.0.1:37144 -> 172.18.0.3:5672) 2023-01-13 10:08:30.824108+00:00 [debug] <0.1010.0> Computing username from client's JWT token, client ID: 'undefined', sub: '63ec2047-6689-45c0-981d-f9b127a6bb7f' 2023-01-13 10:08:30.824277+00:00 [debug] <0.1010.0> User '63ec2047-6689-45c0-981d-f9b127a6bb7f' authenticated successfully by backend rabbit_auth_backend_oauth2 2023-01-13 10:08:30.824367+00:00 [info] <0.1010.0> AMQP 1.0 connection <0.1010.0>: user '63ec2047-6689-45c0-981d-f9b127a6bb7f' authenticated 2023-01-13 10:08:30.826201+00:00 [debug] <0.1010.0> AMQP 1.0 connection.open frame: hostname = vhost:keycloak, extracted vhost = keycloak, idle_timeout = 60000 2023-01-13 10:08:30.831812+00:00 [debug] <0.1020.0> Authentication using an OAuth 2/JWT token failed: provided token is invalid 2023-01-13 10:08:30.831965+00:00 [debug] <0.1020.0> User '63ec2047-6689-45c0-981d-f9b127a6bb7f' failed authenticatation by backend rabbit_auth_backend_oauth2

It looks like user is known and authenticated and connection is open but when I call open_sender() method, second authentication is doing and its failed. I tried many different hostname in rhea and virtual hosts in rabbitmq. When i not provide hostname as rhea connection param rabbitmq is geting default virtual host '/'? Any advice?

[grs]

I'm sorry, I think the rabbitmq team would probably be better placed to help you there. From the log, the virtual host is clearly being extracted correctly, i.e. you are passing it from the rhea client correctly.

[mattiaskjellsson]

I've got a similar problem, don't know if it's related or not.

    const container = rhea.create_container();
    container.on('message', function (context) {
        console.log(context.message.body);
        context.connection.close();
    });

    container.once('sendable', function (context) {
        context.sender.send({body:'Hello World!'});
    });

    const keyFile = '< key file >';
    const certFile = '< cert file >';
    const caFile = '< ca file >';
    const ca = fs.readFileSync(caFile);
    const cert = fs.readFileSync(certFile);
    const key = fs.readFileSync(keyFile);
    const opts: rhea.ConnectionOptions = {
      host: host,
      port: port,
      transport: 'tls',
      key: key,
      cert: cert,
      ca: [ca],
      hostname: `${host}`,
      properties: {
        target: target,
      }
    };

    const connection = container.connect(opts);

    connection.open_receiver(target);
    connection.open_sender(target);
    connection.send({ body: 'Hello world' });

And I get the following error

code: 'ERR_UNHANDLED_ERROR',
  context: ConnectionError {
    message: "Permission PERFORM_ACTION(connect) is denied for : VirtualHost '< host name >' on VirtualHostNode 'default'",
    name: 'ConnectionError',
    condition: 'amqp:not-allowed',
    description: "Permission PERFORM_ACTION(connect) is denied for : VirtualHost '< host name >' on VirtualHostNode 'default'"
  }

I've tried setting properties.target, using the queue name as parameter to the open_sender, and open_receiver, I've tried to use the vhost://, vhost:, vhost:/ followed by the host name to connect to as hostname parameter, but still get

    {
      error: ConnectionError {
        message: "Permission PERFORM_ACTION(connect) is denied for : VirtualHost '<hostname>' on VirtualHostNode 'default'",
        name: 'ConnectionError',
        condition: 'amqp:not-allowed',
        description: "Permission PERFORM_ACTION(connect) is denied for : VirtualHost '<hostname>' on VirtualHostNode 'default'"
      },
...
...

Looks to me like the user in my certificates are not valid, but... How and where do I set the VirtualHostNode? Should that not match the queue name?

[grs]

I believe the VirtualHostNode is specified by target. It looks to me like the target you are using is 'default'. Is that what you expect the queue to be called?

[mattiaskjellsson]

@grs Thank you for your prompt reply. The string I pass to open_receiver/open_sender, try to set in the connection object is definitely not 'default'.

Where to set the target other than the places mentioned above?

[grs]

Is this also rabbitmq broker? I'd advise asking them on their list what the error means. I may be misunderstanding what they call VirtualHostNode is.

[mattiaskjellsson]

I've asked the server people to clarify what they mean by their messages. They are generating AMQP endpoints to connect to by calling a REST API, that returns the following relevant information

"host": "< AMQP host >", "port": "< AMQP port >", "target": "< AMQP target destination >"

I assume that the 'target' is the queue name, which I've asked them to confirm. I assume that the target is the queue name, meaning I should put it as a parameter to open_send, right?

I also see in the documentation, talk about vhost: and vhost:/ both of which I've tried (also vhost://)

In the meantime I also checked connection.js, lines 438 to 443 reads

Connection.prototype.send = function(msg) {
    if (this.default_sender === undefined) {
        this.default_sender = this.open_sender({target:{}});
    }
    return this.default_sender.send(msg);
};

The open_sender call looks a tad strange to me, I tried modifying it to

Connection.prototype.send = function(msg, target) {
    if (this.default_sender === undefined) {
        this.default_sender = this.open_sender({target: target});
    }
    return this.default_sender.send(msg);
};

But the error is still the same with "default" what I think is what should be a queue name in the "VirtualHostNode" :|

[grs]

The send on Connection is deliberately not using a specified target. It is a so called 'anonymous' sender, in which each message sent would have a a 'to' property that would indicate where it should go.

The normal use is to create a sender, specifying a specific target, and then to invoke send on that sender object.

[grs]

The term VirtualHostNode is not one that comes from AMQP. In AMQP a broker queue could certainly be referred to as a node. There is no field explicitly named virtual-host, though there is a hostname.

What might hekp is to get a protocol trace from one side or the other, either through logging (e.g. DEBUG=rhea* for rhea) or with wireshark or similar. That way the actual interaction over the wire is clear.

[mattiaskjellsson]

Thank you for clarifying

The server is a Apache QPID, and they call Node name is: default Virtual host name: < host name from error message > Regarding the "target" that I mention earlier as "< AMQP target destination >", they now call it "the exchange name"

I set DEBUG=rhea in my .env file and don't get any logs. I try to do $export DEBUG=rhea but to no avail. Any suggestions where to set this value if I'm working from a jest test?

In any case, I managed to get a wireshark capture of traffic during a test run, please find the file attached. Just remove the png- extension on the file and open in wireshark 🙏

[grs]

Sorry, because the traffic is encrypted using TLS, the wireshark trace isn't helpful. You will need logging.

Exporting the env var in the terminal you are running a test from should work fine. If you are running it through some IDE, that may require alternative configuration.

You can also set it in code, but you have to make sure you do so before loading the rhea module, e.g. process.env.DEBUG = 'rhea:frames'

[mattiaskjellsson]

Alright, thank you for looking into this.

The debug trace is attached below. Something happens when the connection is opening up.

Digging some more and seeing this https://github.com/amqp/rhea/issues/324

Made me set enable_sasl_external:true on the connection object. I think that solved the issue.

rhea:io [connection-1] connected 192.168.18.10:52963 -> 13.49.84.232:5671 +4s rhea:frames [connection-1] -> { protocol_id: 0, major: 1, minor: 0, revision: 0 } +4s rhea:frames [connection-1]:0 -> open#10 {"container_id":"99cd18e2-71da-0c42-8892-054a3cfd7a80","hostname":"< HOST NAME >"} +0ms rhea:raw [connection-1] SENT: 85 0000005502000000005310d00000004500000002a12439396364313865322d373164612d306334322d383839322d303534613363666437613830a1196e77332d696e7465726368616e67652d612e746c65782e7365 +4s rhea:frames [connection-1]:0 -> begin#11 {"incoming_window":2048,"outgoing_window":4294967295} +0ms rhea:raw [connection-1] SENT: 32 0000002002000000005311d000000010000000044043700000080070ffffffff +0ms rhea:frames [connection-1]:0 -> attach#12 {"name":"b2e48da1-2d99-e44f-a3e2-6a3423aec61c","source":[],"target":["< TARGET >"]} +0ms rhea:raw [connection-1] SENT: 190 000000be02000000005312d0000000ae0000000aa12462326534386461312d326439392d653434662d613365322d3661333432336165633631634342404000532845005329d00000007100000001a16b4c442d70696c6f74696e7465726368616e67652e65752e746c65782e73652e64746e77692e6469676974616c7476696c6c696e672e73652e736539303030392d3537702d384e686e7073704446356c54476a79506d3330756f567249776848346c41717547683569697a4d404043 +1ms rhea:io [connection-1] read 8 bytes +224ms rhea:frames [connection-1] <- { protocol_id: 0, major: 1, minor: 0, revision: 0 } +224ms rhea:io [connection-1] read 470 bytes +214ms rhea:io [connection-1] got frame of size 311 +1ms rhea:raw [connection-1] RECV: 311 0000013702000000005310d0000001270000000aa12436393533616638642d366432372d343162622d616630312d3734386438653239393034384043600000434040e03c03a30f414e4f4e594d4f55532d52454c41590b5348415245442d535542531d736f6c652d636f6e6e656374696f6e2d666f722d636f6e7461696e657240c1b40aa30770726f64756374a11941706163686520517069642042726f6b65722d4a20436f7265a30776657273696f6ea105382e302e36a30a717069642e6275696c64a12831626431633136623261376632373433613934376565346134616364346436326632616236356261a312717069642e696e7374616e63655f6e616d65a10642726f6b6572a325717069642e7669727475616c686f73745f70726f706572746965735f737570706f72746564a10474727565 +438ms rhea:frames [connection-1]:0 <- open#10 {"container_id":"6953af8d-6d27-41bb-af01-748d8e299048","offered_capabilities":["ANONYMOUS-RELAY","SHARED-SUBS","sole-connection-for-container"],"properties":{"product":"Apache Qpid Broker-J Core","version":"8.0.6","qpid.build":"1bd1c16b2a7f2743a947ee4a4acd4d62f2ab65ba","qpid.instance_name":"Broker","qpid.virtualhost_properties_supported":"true"}} +215ms rhea:events [connection-1] Connection got event: connection_open +1s rhea:events [99cd18e2-71da-0c42-8892-054a3cfd7a80] Container got event: connection_open +0ms rhea:io [connection-1] got frame of size 159 +0ms rhea:raw [connection-1] RECV: 159 0000009f02000000005318c0920100531dc08c02a310616d71703a6e6f742d616c6c6f776564a1775065726d697373696f6e20504552464f524d5f414354494f4e28636f6e6e656374292069732064656e69656420666f72203a205669727475616c486f737420276e77332d696e7465726368616e67652d612e746c65782e736527206f6e205669727475616c486f73744e6f6465202764656661756c7427 +0ms rhea:frames [connection-1]:0 <- close#18 {"error":{"condition":"amqp:not-allowed","description":"Permission PERFORM_ACTION(connect) is denied for : VirtualHost '< HOST NAME >' on VirtualHostNode 'default'"}} +0ms rhea:events [connection-1] Connection got event: connection_error +0ms rhea:events [99cd18e2-71da-0c42-8892-054a3cfd7a80] Container got event: connection_error +0ms rhea:events [connection-1] Connection got event: connection_close +0ms rhea:events [99cd18e2-71da-0c42-8892-054a3cfd7a80] Container got event: connection_close +0ms rhea:events [connection-1] Connection got event: error +0ms rhea:events [99cd18e2-71da-0c42-8892-054a3cfd7a80] Container got event: error +0ms rhea:events [connection-1] Connection got event: error +0ms rhea:events [99cd18e2-71da-0c42-8892-054a3cfd7a80] Container got event: error +0ms

[natallia-ivanchuk]

Hi @mattiaskjellsson, did you end up with solution? I'm facing the same problem and couldn't find anything but this thread

[mattiaskjellsson]

Hi @mattiaskjellsson, did you end up with solution? I'm facing the same problem and couldn't find anything but this thread

Hi @natallia-ivanchuk yes, I found a solution. Not sure exactly what it was though, but the comment above mentions enable_sasl_external, looking on my old code, I see one mention of that, and it's indeed on the connection object,

Something like this worked in the end.

this.connection = new Connection(
      this.connectionOptions(host, port, target, source),
    );
    this.target = target;
    this.addConnectionHandlers(this.connection);

    this.connection.container.options['enable_sasl_external'] = true;
...
...
...

private connectionOptions(
    host: string,
    port: number,
    target: string,
    source: string,
  ) {
    const connectionOptions: ConnectionOptions = {
      transport: 'tls',
      host: host,
      hostname: host,
      port: port,
      reconnect: true,
      key: this.key,
      cert: this.cert,
      ca: [this.ca],
      sender_options: {
        target: {
          address: target,
        },
      },
      receiver_options: {
        source: {
          address: source,
          filter: this.messageFilter,
        },
      },
    };
    return connectionOptions;
  }