I am trying to move my code to prod and the group has not yet been instantiated. When I run the python code I can't get it to start consuming any messages. Always get consumed 0 messages
When I tested in dev I first used the CLI to consume some messages and never had any issue until after 7 days of no testing then the issue appeared in dev. When I set reset to smallest in dev is seemed to start working but in prod it is not working and I have no access to CLI in prod. I have tried various version of the auto.reset parameter.
logs_pm_employee_kafka_consumer_redacted.log
How to reproduce
every run
Checklist
Please provide the following information:
Running as a docker container from https://packages.confluent.io/deb/4.1/archive.key
confluent-kafka==0.11.4
authentication using kerberos
broker is confluent 5.5.1
conf = {
'bootstrap.servers': f'{config_object.bootstrap_servers}',
'group.id': f'{config_object.group_id}',
'default.topic.config': {'auto.offset.reset': 'smallest'},
'security.protocol': 'SASL_SSL',
'sasl.mechanisms': 'GSSAPI',
'debug': 'ALL',
'sasl.kerberos.principal': f'{config_object.sasl_kerberos_principal}',
'sasl.kerberos.keytab': f'{config_object.sasl_kerberos_keytab}',
'sasl.kerberos.kinit.cmd': 'kinit -k -t "%{sasl.kerberos.keytab}" %{sasl.kerberos.principal}',
'ssl.ca.location': f'{config_object.ssl_ca_location}',
'error_cb': error_cb
}
group id is gcp-warehouse.privacy-mgr-compliance
topic is production.business-events.cpp.compliance-events
sstevens303
commented
3 years ago
Description
I am trying to move my code to prod and the group has not yet been instantiated. When I run the python code I can't get it to start consuming any messages. Always get consumed 0 messages
When I tested in dev I first used the CLI to consume some messages and never had any issue until after 7 days of no testing then the issue appeared in dev. When I set reset to smallest in dev is seemed to start working but in prod it is not working and I have no access to CLI in prod. I have tried various version of the auto.reset parameter. logs_pm_employee_kafka_consumer_redacted.log
How to reproduce
every run
Checklist
Please provide the following information:
Running as a docker container from https://packages.confluent.io/deb/4.1/archive.key confluent-kafka==0.11.4 authentication using kerberos broker is confluent 5.5.1 conf = { 'bootstrap.servers': f'{config_object.bootstrap_servers}', 'group.id': f'{config_object.group_id}', 'default.topic.config': {'auto.offset.reset': 'smallest'}, 'security.protocol': 'SASL_SSL', 'sasl.mechanisms': 'GSSAPI', 'debug': 'ALL', 'sasl.kerberos.principal': f'{config_object.sasl_kerberos_principal}', 'sasl.kerberos.keytab': f'{config_object.sasl_kerberos_keytab}', 'sasl.kerberos.kinit.cmd': 'kinit -k -t "%{sasl.kerberos.keytab}" %{sasl.kerberos.principal}', 'ssl.ca.location': f'{config_object.ssl_ca_location}', 'error_cb': error_cb }
group id is gcp-warehouse.privacy-mgr-compliance topic is production.business-events.cpp.compliance-events