hop-client issue when subscribing to newly-created topics: topic `does not exist on the broker, so can't subscribe`

[bfc5288]

Description

There appears to be a bug when attempting to use hop subscribe to read from a topic without any messages. The hop-client fails due to ValueError: topic <topic.name> does not exist on the broker, so can't subscribe, even when the topic has been synced and is recognized by kafka.

Example:

After waiting for the topic snews.alert-test to be synced (updateScram: creating topic: snews.alert-test), the topic is recognized by Kafka (via kafka-topics.sh --list), but the hop-client cannot subscribe to the topic:

# this occurs with or without the -s or -p options:
$ hop -d subscribe kafka://kafka.scimma.org/snews.alert-test -p -s EARLIEST    
Traceback (most recent call last):
  File "/home/bfc5288/git.hub/scimma/hop-SNalert-app/snews-workshop/snews-venv/lib/python3.6/site-packages/adc/consumer.py", line 36, in subscribe
    topic_meta = self.describe_topic(topic, timeout)
  File "/home/bfc5288/git.hub/scimma/hop-SNalert-app/snews-workshop/snews-venv/lib/python3.6/site-packages/adc/consumer.py", line 61, in describe_topic
    return cluster_meta.topics[topic]
KeyError: 'snews.alert-test'During handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/home/bfc5288/git.hub/scimma/hop-SNalert-app/snews-workshop/snews-venv/lib/python3.6/site-packages/hop/__main__.py", line 93, in main
    args.func(args)
  File "/home/bfc5288/git.hub/scimma/hop-SNalert-app/snews-workshop/snews-venv/lib/python3.6/site-packages/hop/subscribe.py", line 61, in _main
    with stream.open(args.url, "r", group_id=args.group_id) as s:
  File "/home/bfc5288/git.hub/scimma/hop-SNalert-app/snews-workshop/snews-venv/lib/python3.6/site-packages/hop/io.py", line 122, in open
    read_forever=self.persist,
  File "/home/bfc5288/git.hub/scimma/hop-SNalert-app/snews-workshop/snews-venv/lib/python3.6/site-packages/hop/io.py", line 289, in __init__
    self._consumer.subscribe(t)
  File "/home/bfc5288/git.hub/scimma/hop-SNalert-app/snews-workshop/snews-venv/lib/python3.6/site-packages/adc/consumer.py", line 38, in subscribe
    raise ValueError(f"topic {topic} does not exist on the broker, so can't subscribe")
ValueError: topic snews.alert-test does not exist on the broker, so can't subscribe

Note that using hop publish to this topic succeeds, after which the above hop subscribe command works as expected.

[cnweaver]

I am able to reproduce this, although not locally. After creating an empty topic sys.new-test:

$ hop subscribe -s EARLIEST kafka://weave132-969b4a83@kafka.scimma.org/sys.new-test
hop: topic sys.new-test does not exist on the broker, so can't subscribe

kafkacat gives a slightly different error:

$ kafkacat -F kafkacat-prod.conf -L -t sys.new-test -b kafka.scimma.org
% Reading configuration from file kafkacat-prod.conf                                                                                                            Metadata for sys.new-test (from broker -1: sasl_ssl://kafka.scimma.org:9092/bootstrap):
 3 brokers:
  broker 0 at kb-0.prod.hop.scimma.org:9092
  broker 2 at kb-2.prod.hop.scimma.org:9092
  broker 1 at kb-1.prod.hop.scimma.org:9092 (controller)
 1 topics:
  topic "sys.new-test" with 0 partitions: Broker: Topic authorization failed

This appears to acknowledge that the topic exists, but suggests that there is a permissions issue.

There is a READ permission in place:

$ kafka-acls.sh --command-config prod-admin.properties --bootstrap-server kafka.scimma.org:9092 --list --topic sys.new-test
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=sys.new-test, patternType=LITERAL)`: 
    (principal=User:*, host=*, operation=READ, permissionType=ALLOW)

Manually creating an ACL allowing any user to Describe the topic seems to fix it:

$ kafka-acls.sh --command-config prod-admin.properties --bootstrap-server kafka.scimma.org:9092 --add --topic sys.new-test --allow-principal 'User:*' --resource-pattern-type LITERAL --operation Describe
Adding ACLs for resource `ResourcePattern(resourceType=TOPIC, name=sys.new-test, patternType=LITERAL)`: 
    (principal=User:*, host=*, operation=DESCRIBE, permissionType=ALLOW) 
Current ACLs for resource `ResourcePattern(resourceType=TOPIC, name=sys.new-test, patternType=LITERAL)`: 
    (principal=User:*, host=*, operation=READ, permissionType=ALLOW)
    (principal=User:*, host=*, operation=DESCRIBE, permissionType=ALLOW) 
$ kafkacat -F kafkacat-prod.conf -C -t sys.new-test -b kafka.scimma.org
% Reading configuration from file kafkacat-prod.conf
% Reached end of topic sys.new-test [2] at offset 0
% Reached end of topic sys.new-test [1] at offset 0
% Reached end of topic sys.new-test [0] at offset 0
% Reached end of topic sys.new-test [5] at offset 0
% Reached end of topic sys.new-test [8] at offset 0
% Reached end of topic sys.new-test [11] at offset 0
% Reached end of topic sys.new-test [14] at offset 0
% Reached end of topic sys.new-test [4] at offset 0
% Reached end of topic sys.new-test [7] at offset 0
% Reached end of topic sys.new-test [10] at offset 0
% Reached end of topic sys.new-test [13] at offset 0
% Reached end of topic sys.new-test [3] at offset 0
% Reached end of topic sys.new-test [6] at offset 0
% Reached end of topic sys.new-test [9] at offset 0
% Reached end of topic sys.new-test [12] at offset 0
% Reached end of topic sys.new-test [15] at offset 0
^C
$ hop subscribe -s EARLIEST kafka://weave132-969b4a83@kafka.scimma.org/sys.new-test
$ echo $?
0

It is unclear why publishing a message into the topic works around this, but it looks like at a minimum hop-creds-sync should be creating Describe ACLs, at least for public topics, and possibly to match each non-All ACL for individual users/non-public topics.

EDIT: The snews-alert-test topic was not public, so this is not only an issue for public topics and Describe permissions should be added for per-user permissions on non-public topics.