Broker list doesn't refresh

[hamishforbes]

The background metadata refresh doesn't seem to update the list of brokers correctly.

I've vendor'd in this lib to my app and tweaked the logging a little bit to dump out the list of brokers being used for the metadata request and the list of brokers being set back

    local broker_txt = {}
    for k,v in pairs(broker_list) do
        table.insert(broker_txt, {broker_id = k, host = v.host, port = v.port})
    end
    ngx_log(ngx.INFO, "[Kafka](",ngx.worker.id(),") Fetch Metadata broker list ", require("cjson").encode(broker_txt))

This gives me log output like:

11:06:03.821 info *291 [lua] client.lua:162: _fetch_metadata(): [Kafka](1) New broker list [{"host":"10.2.14.10","port":30597,"broker_id":0},{"host":"10.2.12.174","port":31353,"broker_id":2},{"host":"10.2.13.223","port":31341,"broker_id":1}], context: ngx.timer
11:06:03.816 info *291 [lua] client.lua:147: _fetch_metadata(): [Kafka](1) Fetching metadata from kafka-bootstrap:9092, context: ngx.timer
11:06:03.815 info *291 [lua] client.lua:143: _fetch_metadata(): [Kafka](1) Fetch Metadata broker list [{"host":"kafka-bootstrap","port":9092,"broker_id":1}], context: ngx.timer

and then ~30s later (my configured refresh interval)

11:06:32.797 info *539 [lua] client.lua:143: _fetch_metadata(): [Kafka](1) Fetch Metadata broker list [{"host":"kafka-bootstrap","port":9092,"broker_id":1}], context: ngx.timer

As you can see the broker list is still the original static bootstrap values.

This is a problem because if all the client is created with a static list of IPs and then all the brokers rotate out everything just fails.

[doujiang24]

@hamishforbes I can not understand your log, where does the New broker list come from?

[doujiang24]

This is a problem because if all the client is created with a static list of IPs and then all the brokers rotate out everything just fails.

there should have error messages in this case.

[hamishforbes]

@hamishforbes I can not understand your log, where does the New broker list come from?

Sorry maybe I didn't explain very well, i've just pushed my modified version: https://github.com/doujiang24/lua-resty-kafka/compare/master...hamishforbes:debug_logging

The new broker list comes from the response to the background metadata refresh request