simplesteph / medium-blog-kafka-udemy

Supporting repository for the blog post at https://medium.com/@stephane.maarek/how-to-use-apache-kafka-to-transform-a-batch-pipeline-into-a-real-time-one-831b48a6ad85
240 stars 199 forks source link

Reviews producer doesn't fetch the reviews #7

Closed sumedhyadav closed 3 years ago

sumedhyadav commented 3 years ago

The application starts, as can be seen the output,

[2020-09-16 16:43:00,181] INFO [Producer clientId=producer-1] Instantiated an idempotent producer. (org.apache.kafka.clients.producer.KafkaProducer) [2020-09-16 16:43:00,237] INFO Kafka version : 2.1.0 (org.apache.kafka.common.utils.AppInfoParser) [2020-09-16 16:43:00,237] INFO Kafka commitId : eec43959745f444f (org.apache.kafka.common.utils.AppInfoParser) [2020-09-16 16:43:00,238] INFO Application started! (ReviewsProducerMain) [2020-09-16 16:43:00,239] INFO Stuff submit (ReviewsProducerMain) [2020-09-16 16:43:00,239] INFO Latch await (ReviewsProducerMain) [2020-09-16 16:43:00,311] INFO [Producer clientId=producer-1] ProducerId set to 2 with epoch 0 (org.apache.kafka.clients.producer.internals.TransactionManager) [2020-09-16 16:43:00,986] WARN Invalid cookie header: "Set-Cookie: ud_cache_version=1; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,986] WARN Invalid cookie header: "Set-Cookie: ud_cache_release=0028d88b2f47b2a4928d; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: ud_user_jwt=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: ud_cache_device=None; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: ud_cache_modern_browser=0; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: exaff=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: evi="Sg4="; expires=Fri, 16 Oct 2020 11:13:00 GMT; Max-Age=2592000; Path=/". Invalid 'expires' attribute: Fri, 16 Oct 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: ud_firstvisit=2020-09-16T11:13:00.726900+00:00:1kIVMy:zGeZbMtC6immFfVnjmEYugKGQxM; expires=Thu, 16 Sep 2021 11:13:00 GMT; Max-Age=31536000; Path=/". Invalid 'expires' attribute: Thu, 16 Sep 2021 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,987] WARN Invalid cookie header: "Set-Cookie: udmy_2_v57r=b44a0e4903304111b679e14745e84692; Domain=.udemy.com; expires=Thu, 16 Sep 2021 11:13:00 GMT; Max-Age=31536000; Path=/; Secure". Invalid 'expires' attribute: Thu, 16 Sep 2021 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: eva=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: ud_cache_brand=INen_US; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: inaff=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: ud_cache_language=en; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: ud_cache_marketplace_country=IN; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,988] WARN Invalid cookie header: "Set-Cookie: udmy_evid=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: ud_rule_vars=eJyFjcsOwiAURH-lYas191L6gG8hIRRvK7GRCLSbpv8uiZq4czmTOXN2lm2cKdPVbD75HKIahbBAQkLTgEDEsesloehFS4PoJFcuhLsnpiq2azb5mPKbNVebSZdeMw4capA1dhWiwkYBXHqUvGlPACVodi6rxRY0h9XdTI52mrwzKazRkdls9HZcPm8hzvbh3Q8U6blS-mccoEXBv8aDHS_n2EbJ:1kIVMy:wjpX1EOv-_GEyfgJq2yDuM_CJxs; expires=Fri, 16 Sep 2022 11:13:00 GMT; Max-Age=63072000; Path=/". Invalid 'expires' attribute: Fri, 16 Sep 2022 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: ud_cache_user=""; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: udmy_branded_partner=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: ud_cache_logged_in=0; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:00,989] WARN Invalid cookie header: "Set-Cookie: ud_cache_price_country=IN; expires=Thu, 17 Sep 2020 11:13:00 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_brand=INen_US; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_price_country=IN; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: evi="Sg4="; expires=Fri, 16 Oct 2020 11:13:01 GMT; Max-Age=2592000; Path=/". Invalid 'expires' attribute: Fri, 16 Oct 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_user=""; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_language=en; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_firstvisit=2020-09-16T11:13:01.294599+00:00:1kIVMz:EtF4-PS6q0wHvXiqOkKVtpHj6Xs; expires=Thu, 16 Sep 2021 11:13:01 GMT; Max-Age=31536000; Path=/". Invalid 'expires' attribute: Thu, 16 Sep 2021 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,478] WARN Invalid cookie header: "Set-Cookie: ud_cache_release=0028d88b2f47b2a4928d; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: exaff=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: udmy_branded_partner=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: ud_cache_logged_in=0; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: ud_cache_version=1; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: eva=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: ud_user_jwt=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: ud_cache_marketplace_country=IN; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: inaff=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: __udmy_evid=""; expires=Thu, 01 Jan 1970 00:00:00 GMT; Max-Age=0; Path=/". Invalid 'expires' attribute: Thu, 01 Jan 1970 00:00:00 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,479] WARN Invalid cookie header: "Set-Cookie: __udmy_2_v57r=9f5a4908f5fc47d89dd17de039798b23; Domain=.udemy.com; expires=Thu, 16 Sep 2021 11:13:01 GMT; Max-Age=31536000; Path=/; Secure". Invalid 'expires' attribute: Thu, 16 Sep 2021 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,480] WARN Invalid cookie header: "Set-Cookie: ud_rule_vars="eJyFzUsOwiAUheGtNEy15t5SLLAWEkJ5VKKRCLSTpnu3qTp2eJL_y1lJNXny1Tu9xBJrylIEZnoBPLBg-8Fx4RwOzgMVg-BjR6VN6R49kQ1ZFXmYUnVNs73pmk0I0eqS5my9XkyOZnx4tZeKpDyZZ7SKnPcRYt7Vcaidqd-kgw5aEC1eG0SJVAJeOs4ZFScACfCxx2H2r9mXf5hSZMh-eCPbGyOgR-U=:1kIVMz:GhW8HauXT_CnL8xoXhyTV6A8g4w"; expires=Fri, 16 Sep 2022 11:13:01 GMT; Max-Age=63072000; Path=/". Invalid 'expires' attribute: Fri, 16 Sep 2022 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,480] WARN Invalid cookie header: "Set-Cookie: ud_cache_device=None; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,480] WARN Invalid cookie header: "Set-Cookie: ud_cache_modern_browser=0; expires=Thu, 17 Sep 2020 11:13:01 GMT; Max-Age=86400; Path=/; Secure". Invalid 'expires' attribute: Thu, 17 Sep 2020 11:13:01 GMT (org.apache.http.client.protocol.ResponseProcessCookies) [2020-09-16 16:43:01,480] INFO Fetched 0 reviews (ReviewsFetcherThread) [2020-09-16 16:43:01,530] INFO Closing (ReviewsFetcherThread) [2020-09-16 16:43:01,531] INFO Closed (ReviewsFetcherThread) [2020-09-16 16:43:01,641] INFO Closing Producer (ReviewsAvroProducerThread) [2020-09-16 16:43:01,642] INFO [Producer clientId=producer-1] Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms. (org.apache.kafka.clients.producer.KafkaProducer)

, but due to response cookie header mismatch, the reviews are not fetched.

GiuseppeMP commented 3 years ago

@sumedhyadav Fixed, PR #11.