search

arnaud-lb / php-rdkafka

Production-ready, stable Kafka client for PHP
MIT License
2.08k stars 263 forks source link

Compatibility with 1.0.0 #220

Closed Steveb-p closed 4 years ago

Steveb-p commented 5 years ago

I've run tests to prepare myself to give a try to add topic API to phprdkafka and I've noticed that integration tests fail on librdkafka 1.0.0, while they work just fine against 0.11.6. This affects integration tests only, hence probably why they went undetected in regard to CI, since travis seems to run non-integration tests only.

Here is output of my 1.0.0 run:

[steveb@y700 php-rdkafka]$ make test

Build complete.
Don't forget to run 'make test'.

=====================================================================
PHP         : /usr/bin/php 
PHP_SAPI    : cli
PHP_VERSION : 7.3.4
ZEND_VERSION: 3.3.4
PHP_OS      : Linux - Linux y700 4.19.36-1-MANJARO #1 SMP PREEMPT Sat Apr 20 22:05:46 UTC 2019 x86_64
INI actual  : /home/steveb/CLionProjects/php-rdkafka/tmp-php.ini
More .INIs  :   
CWD         : /home/steveb/CLionProjects/php-rdkafka
Extra dirs  : 
VALGRIND    : Not used
=====================================================================
TIME START 2019-05-09 01:12:56
=====================================================================
FAIL Allow null payload [tests/allow_null_payload.phpt] 
PASS Allow null payload [tests/allow_null_payload_and_key.phpt] 
FAIL Bug 115 [tests/bug115.phpt] 
PASS Bug 74 [tests/bug74.phpt] 
PASS Bug 88 [tests/bug88.phpt] 
PASS RdKafka\Conf [tests/bugConfSetArgument.phpt] 
PASS RdKafka\Conf [tests/conf.phpt] 
PASS RdKafka\Conf [tests/conf_callbacks.phpt] 
FAIL RdKafka\Conf [tests/conf_callbacks_integration.phpt] 
PASS RdKafka\Conf::setDefaultTopicConf() [tests/conf_setDefaultTopicConf.phpt] 
PASS constants [tests/constants.phpt] 
FAIL Message headers [tests/message_headers.phpt] 
FAIL Produce, consume [tests/produce_consume.phpt] 
FAIL Produce, consume queue [tests/produce_consume_queue.phpt] 
PASS rd_kafka_get_err_descs() [tests/rd_kafka_get_err_descs.phpt] 
PASS test0 [tests/test0.phpt] 
PASS RdKafka\TopicConf [tests/topic_conf.phpt] 
PASS RdKafka\TopicPartition [tests/topic_partition.phpt] 
=====================================================================
TIME END 2019-05-09 01:17:06

=====================================================================
TEST RESULT SUMMARY
---------------------------------------------------------------------
Exts skipped    :    0
Exts tested     :   29
---------------------------------------------------------------------

Number of tests :   18                18
Tests skipped   :    0 (  0.0%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    6 ( 33.3%) ( 33.3%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :   12 ( 66.7%) ( 66.7%)
---------------------------------------------------------------------
Time taken      :  250 seconds
=====================================================================

=====================================================================
FAILED TEST SUMMARY
---------------------------------------------------------------------
Allow null payload [tests/allow_null_payload.phpt]
Bug 115 [tests/bug115.phpt]
RdKafka\Conf [tests/conf_callbacks_integration.phpt]
Message headers [tests/message_headers.phpt]
Produce, consume [tests/produce_consume.phpt]
Produce, consume queue [tests/produce_consume_queue.phpt]
=====================================================================

I've double checked that tests run correct librdkafka version (output via phpinfo() in .phpt file:

rdkafka

rdkafka support => enabled
version => 3.1.0-dev
build date => May  9 2019 03:12:16
librdkafka version (runtime) => 1.0.0
librdkafka version (build) => 1.0.0.0

When looking into test output, I have for example in bug115.out:

** ERROR: process timed out **

I can see in my Kafka instance log that some communication occurs and it roughly resembles what I've seen for 0.11.6 librdkafka.

For reference that my Kafka is configured properly here's my results for 0.11.6:

[steveb@y700 php-rdkafka]$ make test

Build complete.
Don't forget to run 'make test'.

=====================================================================
PHP         : /usr/bin/php 
PHP_SAPI    : cli
PHP_VERSION : 7.3.4
ZEND_VERSION: 3.3.4
PHP_OS      : Linux - Linux y700 4.19.36-1-MANJARO #1 SMP PREEMPT Sat Apr 20 22:05:46 UTC 2019 x86_64
INI actual  : /home/steveb/CLionProjects/php-rdkafka/tmp-php.ini
More .INIs  :   
CWD         : /home/steveb/CLionProjects/php-rdkafka
Extra dirs  : 
VALGRIND    : Not used
=====================================================================
TIME START 2019-05-09 01:30:43
=====================================================================
PASS Allow null payload [tests/allow_null_payload.phpt] 
PASS Allow null payload [tests/allow_null_payload_and_key.phpt] 
PASS Bug 115 [tests/bug115.phpt] 
PASS Bug 74 [tests/bug74.phpt] 
PASS Bug 88 [tests/bug88.phpt] 
PASS RdKafka\Conf [tests/bugConfSetArgument.phpt] 
PASS RdKafka\Conf [tests/conf.phpt] 
PASS RdKafka\Conf [tests/conf_callbacks.phpt] 
PASS RdKafka\Conf [tests/conf_callbacks_integration.phpt] 
PASS RdKafka\Conf::setDefaultTopicConf() [tests/conf_setDefaultTopicConf.phpt] 
PASS constants [tests/constants.phpt] 
PASS Message headers [tests/message_headers.phpt] 
PASS Produce, consume [tests/produce_consume.phpt] 
PASS Produce, consume queue [tests/produce_consume_queue.phpt] 
PASS rd_kafka_get_err_descs() [tests/rd_kafka_get_err_descs.phpt] 
PASS test0 [tests/test0.phpt] 
PASS RdKafka\TopicConf [tests/topic_conf.phpt] 
PASS RdKafka\TopicPartition [tests/topic_partition.phpt] 
=====================================================================
TIME END 2019-05-09 01:31:02

=====================================================================
TEST RESULT SUMMARY
---------------------------------------------------------------------
Exts skipped    :    0
Exts tested     :   29
---------------------------------------------------------------------

Number of tests :   18                18
Tests skipped   :    0 (  0.0%) --------
Tests warned    :    0 (  0.0%) (  0.0%)
Tests failed    :    0 (  0.0%) (  0.0%)
Expected fail   :    0 (  0.0%) (  0.0%)
Tests passed    :   18 (100.0%) (100.0%)
---------------------------------------------------------------------
Time taken      :   19 seconds
=====================================================================

And exerpt from phpinfo():

rdkafka

rdkafka support => enabled
version => 3.1.0-dev
build date => May  9 2019 03:29:44
librdkafka version (runtime) => 0.11.6
librdkafka version (build) => 0.11.6.0
Steveb-p commented 5 years ago

I've tried to pinpoint issues that prevented integration tests from passing.

Apart from the linked issue with a separate PR regarding segmentation fault, it seems that in most cases consumption mostly no longer returns Kafka's message object, especially on EOF (RD_KAFKA_RESP_ERR__PARTITION_EOF on $msg->err). This in turn causes test scripts to time out.

From what I understand, this boils down to consumption methods to block until new message is available, instead of returning early when current end of topic is reached.

I've created a draft PR with changes that attempt to adjust tests and comments for things that are worth noting.

nick-zh commented 5 years ago

@Steveb-p RD_KAFKA_RESP_ERR__PARTITION_EOF is no longer returned per default, you can turn that on again with the setting enable.partition.eof but i think it is great, if both behaviours are supported anyway probably :smile:

Steveb-p commented 5 years ago

@Steveb-p RD_KAFKA_RESP_ERR__PARTITION_EOF is no longer returned per default, you can turn that on again with the setting enable.partition.eof but i think it is great, if both behaviours are supported anyway probably

@nick-zh I've tried tinkering with that particular setting in tests expecting it to do that exactly, but it didn't look like it had any effect. But it was late at night and maybe I missed something.

Anyway it looks like it's just a test issue, not incompatibility. Some documentation might need updating as well, which I'll give a try in PR if that's the case.

nick-zh commented 5 years ago

@Steveb-p what is the current state on this? Do you need any assistance with #223, i think it would be really cool if we could fix this :)

Steveb-p commented 4 years ago

@nick-zh I'll look into it again this weekend, once I'm done with enqueue.