devinbost
commented
2 years ago @jerrypeng FYI
Open devinbost opened 2 years ago
devinbost
commented
2 years ago @jerrypeng FYI
eolivelli
commented
2 years ago I believe that the problem is that you are missing "-L", that tells curl to follow the redirections.
I have a test cluster with two workers, on port 8080 and 6751 The worker on port 8080 is the "leader"
If I issue this command:
curl -v -L -X PUT http://localhost:6751/admin/v2/worker/rebalance
This is the output:
* Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 6751 failed: Connection refused
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 6751 (#0)
> PUT /admin/v2/worker/rebalance HTTP/1.1
> Host: localhost:6751
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 307 Temporary Redirect
< Date: Wed, 01 Dec 2021 11:08:54 GMT
< Location: http://localhost:8080/admin/v2/worker/rebalance
< Content-Length: 0
< Server: Jetty(9.4.43.v20210629)
<
* Connection #0 to host localhost left intact
* Issue another request to this URL: 'http://localhost:8080/admin/v2/worker/rebalance'
* Found bundle for host localhost: 0x7fb020e05590 [can pipeline]
* Could pipeline, but not asked to!
* Trying ::1...
* TCP_NODELAY set
* Connection failed
* connect to ::1 port 8080 failed: Connection refused
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to localhost (127.0.0.1) port 8080 (#1)
> PUT /admin/v2/worker/rebalance HTTP/1.1
> Host: localhost:8080
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 204 No Content
< Date: Wed, 01 Dec 2021 11:08:54 GMT
< broker-address: localhost
< Server: Jetty(9.4.43.v20210629)
<
* Connection #1 to host localhost left intact
* Closing connection 0
* Closing connection 1Then I see correctly the logs on the "leader" that report that the "rebalance" is working
github-actions[bot]
commented
2 years ago The issue had no activity for 30 days, mark with Stale label.
Describe the bug Hitting the endpoint to rebalance functions does not appear to work consistently in Pulsar 2.7.2.
To Reproduce Steps to reproduce the behavior:
First, we look at the function assignments:
Next, we trigger functions to rebalance:
$ curl fab08.example.domain.com:8080/admin/v2/worker/rebalance -X PUT -H "Authorization: Bearer eyJ...mNog"Checking function assignments again after a few minutes shows no changes, as demonstrated below:
After some experimentation, I discovered that I was able to trigger rebalancing to occur if I targeted the function worker leader, but it's not clear if this happens consistently or not.
Expected behavior Triggering function rebalancing should work consistently when triggered on any broker. Also, if there is a failure, it should be reported in the logs. In the current implementation, when sending the rebalance request, no logs appeared in the targeted broker except when it succeeded. More logging should indicate if there's a problem on the broker that receives the signal to rebalance functions.