search

rabbitmq / rabbitmq-server

Open source RabbitMQ: core server and tier 1 (built-in) plugins
https://www.rabbitmq.com/
Other
11.9k stars 3.9k forks source link

AMQP 1.0 connection authentication stopped working against the HTTP authN backend starting with 3.11.10 #9043

Closed lukebakken closed 11 months ago

lukebakken commented 11 months ago

Discussed in https://github.com/rabbitmq/rabbitmq-server/discussions/9031

Originally posted by **motmot80** August 8, 2023 ### Describe the bug Since RabbitMQ 3.11.10 it isnt possible to authenticate amqp-v1.0 clients with the **http-auth**-plugin. - RMQ 3.11.9 with Erlang 25.3.2.3: working - RMQ 3.10.25​ with Erlang 25.3.2.3: working - RMQ 3.11.10 with Erlang 25.3.2.3: **not** working => auth_failure,"Refused" It seems that the lastest RMQ versions may have broken the plugin combination - rabbitmq_auth_backend_http - rabbitmq_amqp1_0 ### Reproduction steps 1. Install RMQ **> 3.11.10** 2. Install REST-Mock as http auth plugin counter part replying with success 3. Activate and configure **rabbitmq_auth_backend_http** 4. Activate **rabbitmq_amqp1_0** 5. Connect AMQPv1.0 client (QPID-JMS, amqpnetlite, ...) with username & password for http-auth Minimal java client: ``` org.apache.qpid.jms.JmsConnectionFactory jmsConnectionFactory = new org.apache.qpid.jms.JmsConnectionFactory("amqps://host:5671"); jmsConnectionFactory.setUsername("BOB"); jmsConnectionFactory.setPassword("PASSWORD"); jmsConnectionFactory.createConnection().createSession(); ``` ### Expected behavior Same behavior as RMQ **3.11.9** or **3.10.25** (< 3.11.10): Authentication is working ### Additional context Maybe the Oauth 2 refactorings have broken the interface between the amqpv1 and the http-auth plugin. Thanks in advance and best regards Thomas
RabbitMQ stack trace ``` 2023-08-09 09:32:15.505849-07:00 [info] <0.822.0> accepting AMQP connection <0.822.0> (127.0.0.1:39236 -> 127.0.0.1:5672) 2023-08-09 09:32:15.513829-07:00 [debug] <0.822.0> User 'admin' failed authentication by backend rabbit_auth_backend_internal 2023-08-09 09:32:15.513994-07:00 [debug] <0.822.0> auth_backend_http: GET http://localhost:8000/auth/user?username=admin&password=foobar 2023-08-09 09:32:15.514031-07:00 [debug] <0.822.0> auth_backend_http: request timeout: 15000, connection timeout: 15000 2023-08-09 09:32:15.516584-07:00 [debug] <0.822.0> auth_backend_http: response code is 200, body: "allow administrator" 2023-08-09 09:32:15.516672-07:00 [debug] <0.822.0> User 'admin' authenticated successfully by backend rabbit_auth_backend_http 2023-08-09 09:32:15.516725-07:00 [info] <0.822.0> AMQP 1.0 connection <0.822.0>: user 'admin' authenticated 2023-08-09 09:32:15.525831-07:00 [debug] <0.822.0> AMQP 1.0 connection.open frame: hostname = localhost, extracted vhost = /, idle_timeout = 60000 2023-08-09 09:32:15.531141-07:00 [warning] <0.830.0> AMQP 0-9-1 client call timeout was 70000 ms, is updated to a safe effective value of 130000 ms 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> User 'admin' authentication failed with exit:{unknown_auth_props,<<"admin">>, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> [{rabbit_auth_backend_http, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> #Fun}]}: 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> [{rabbit_auth_backend_internal,user_login_authentication,2, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> [{file,"rabbit_auth_backend_internal.erl"}, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> {line,99}]}, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> {rabbit_access_control,try_authenticate,3, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> [{file,"rabbit_access_control.erl"},{line,86}]}, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> {rabbit_access_control,'-check_user_login/2-fun-0-',4, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> [{file,"rabbit_access_control.erl"},{line,51}]}, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> {lists,foldl,3,[{file,"lists.erl"},{line,1350}]}, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> {rabbit_access_control,check_user_login,2, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> [{file,"rabbit_access_control.erl"},{line,36}]}, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> {rabbit_direct,connect,5,[{file,"rabbit_direct.erl"},{line,103}]}, 2023-08-09 09:32:15.537525-07:00 [debug] <0.834.0> {erpc,execute_call,4,[{file,"erpc.erl"},{line,589}]}] 2023-08-09 09:32:15.537889-07:00 [warning] <0.830.0> Closing session for connection <0.822.0>: 2023-08-09 09:32:15.537889-07:00 [warning] <0.830.0> {auth_failure,"Refused"} 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> crasher: 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> initial call: rabbit_amqp1_0_session_process:init/1 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> pid: <0.830.0> 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> registered_name: [] 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> exception exit: {auth_failure,"Refused"} 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> in function gen_server2:init_it/6 (gen_server2.erl, line 600) 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> ancestors: [<0.828.0>,<0.824.0>,<0.821.0>,<0.820.0>,<0.737.0>,<0.736.0>, 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> <0.735.0>,<0.733.0>,<0.732.0>,rabbit_sup,<0.239.0>] 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> message_queue_len: 0 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> messages: [] 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> links: [<0.828.0>] 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> dictionary: [{gen_server_call_timeout,130000}] 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> trap_exit: true 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> status: running 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> heap_size: 1598 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> stack_size: 28 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> reductions: 7518 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> neighbours: 2023-08-09 09:32:15.538062-07:00 [error] <0.830.0> 2023-08-09 09:32:15.538336-07:00 [error] <0.822.0> Error on AMQP 1.0 connection <0.822.0> (running), channel 0: 2023-08-09 09:32:15.538336-07:00 [error] <0.822.0> <<"Reader error: {error,\n {{auth_failure,\"Refused\"},\n {child,undefined,channel,\n {rabbit_amqp1_0_session_process,start_link,\n [{0,<0.822.0>,<0.829.0>,\n {user,<<\"admin\">>,\n [administrator],\n [{rabbit_auth_backend_http,\n #Fun}]},\n <<\"/\">>,1048568,\n {amqp_adapter_info,\n {0,0,0,0,0,65535,32512,1},\n 5672,\n {0,0,0,0,0,65535,32512,1},\n 39236,<<\"127.0.0.1:39236 -> 127.0.0.1:5672\">>,\n {'AMQP',\"1.0\"},\n [{ssl,false},\n {authz_backends,\n [{rabbit_auth_backend_http,\n #Fun}]}]},\n <0.825.0>}]},\n transient,true,300000,worker,\n [rabbit_amqp1_0_session_process]}}}\n[{rabbit_amqp1_0_reader,send_to_new_1_0_session,3,\n [{file,\"rabbit_amqp1_0_reader.erl\"},{line,736}]},\n {rabbit_amqp1_0_reader,handle_1_0_frame,4,\n [{file,\"rabbit_amqp1_0_reader.erl\"},{line,317}]},\n {rabbit_amqp1_0_reader,recvloop,2,\n [{file,\"rabbit_amqp1_0_reader.erl\"},{line,120}]},\n {rabbit_reader,run,1,[{file,\"rabbit_reader.erl\"},{line,453}]},\n {rabbit_reader,start_connection,5,[{file,\"rabbit_reader.erl\"},{line,352}]},\n {proc_lib,init_p_do_apply,3,[{file,\"proc_lib.erl\"},{line,240}]}]">> ```
lukebakken commented 11 months ago

Hm, even with RabbitMQ 3.11.9 I can see that the password is not passed to the auth backend for an AMQP 1.0 connection:

INFO <QueryDict: {'username': ['admin']}>
INFO "GET /auth/user?username=admin HTTP/1.1" 200 4

This is what I see when I run rabbitmqctl authenticate_user admin foobar:

INFO <QueryDict: {'username': ['admin'], 'password': ['foobar']}>
INFO "GET /auth/user?username=admin&password=foobar HTTP/1.1" 200 19

...and this is what I see with a regular AMQP 0.9.1 connection via this test program:

INFO <QueryDict: {'username': ['admin'], 'password': ['foobar']}>
INFO "GET /auth/user?username=admin&password=foobar HTTP/1.1" 200 19
INFO <QueryDict: {'username': ['admin'], 'vhost': ['/'], 'ip': ['::ffff:127.0.0.1'], 'tags': ['administrator']}>
INFO "GET /auth/vhost?username=admin&vhost=%2F&ip=%3A%3Affff%3A127.0.0.1&tags=administrator HTTP/1.1" 200 5

Continuing to investigate...

cc @michaelklishin

lukebakken commented 11 months ago

@motmot80 well, testing with the RabbitMQ 3.10.x branch still fails. It appears that the AMQP 1.0 plugin hasn't sent the supplied password to the HTTP auth backend for quite a while now:

INFO <QueryDict: {'username': ['admin']}>
INFO "GET /auth/user?username=admin HTTP/1.1" 200 4

In your environment, does your HTTP auth server bother to look at the password supplied or is it just the username?

I'm moving on to just fixing this issue rather than trying to understand when it broke, or if it ever worked 😹

lukebakken commented 11 months ago

It looks like PR #6931 is what broke the combination of the AMQP 1.0 plugin and multiple auth backends. cc @MarcialRosales @michaelklishin

michaelklishin commented 11 months ago

Addressed in #9045.

@motmot80 would you be able to test #9045 with an OCI (Docker) image or do you need a specific package (e.g. Debian or Windows)?

michaelklishin commented 11 months ago

@MarcialRosales @lukebakken is this not applicable to main and v3.12.x?

MarcialRosales commented 11 months ago

@michaelklishin yes, it should be applied to main and v3.12.x. Let me know if there are any conflicts. If there is any conflict around the seleniun tests, just do not merge those changes. In any case, I am putting together a PR with main as base with the fix plus some other fixes around the test scripts. I am tagging it with backport to v3.12.x. I will make sure there are no conflicts.

lukebakken commented 11 months ago

Re-opening since #9045 is not the correct fix.

9052 should do it.

motmot80 commented 11 months ago

@motmot80 would you be able to test #9045 with an OCI (Docker) image or do you need a specific package (e.g. Debian or Windows)?

@michaelklishin We retested https://github.com/rabbitmq/rabbitmq-server-binaries-dev/releases/tag/v3.11.21-alpha.16 fix in one of our dev reference environments (RHEL 8, Erlang 25.3.2.3).

grafik

Works like a charm!

Thanks for the quick support.

Best regards Thomas

motmot80 commented 2 months ago

@lukebakken After upgraded to 3.12.13-debian-12-r2 the problem occurs again.

lukebakken commented 2 months ago

Hi @motmot80 - could you please do the following:

Thanks

MarcialRosales commented 2 months ago

Hi @motmot80 , I am not able to reproduce this issue in 3.12.13 (docker image). Could you please provide rabbitmq logs with debug level enabled and also the token you are using? Thanks