Open qbadx opened 7 years ago
I have add this issue into TODO LIST.
nginx common var is very usefull
@lifehacking yep, I cannot agree more. PR is welcomed :)
If anyone here is looking for a workaround in the interim, I am successfully injecting nginx variables into the $request_body using the lua nginx module.
@dkinon Yes, using lua_nginx_module injecting ngx variables into $request_body is easy to work, and I am an openresty user. But I still want to do some works to pass variables to kafka topic without lua module, cause not everyone use lua_nginx_module. :)
what about http headers - would it be possible to add them to the message body?
@adridi http headers are actually the ngx variables (ngx.req.get_headers()) I'm injecting into the $request_body. Since they are just another ngx variable, I don't see why they couldn't technically be included. The bigger issue with including ngx variables into a kafka payload in some standard way is that kafka messages have no standard structure. It's just a text payload.
I prefer JSON so I make sure the incoming $request_body is JSON, I inject the ngx variables into the appropriate JSON key in $request_body, pass the resulting text payload of $request_body to the appropriate kafka topic using this module, and then parse that JSON structure on the other end with my kafka consumer. This works great for me but what if someone doesn't want to, or can't, use JSON... now the complexity of implementing this support in this module explodes.
@dkinon Totally Agree with your statement! I was referring to the existing module, I believe it should have set kafka message as = request header + request body. Both are equally important.
@adridi I was also referring to the existing module, I merely used my lua_ngx module pseudocode example to illustrate one of the many formatted ways this could be done within the confines of the module. The larger point of my previous statement was discussing what format the headers in $request_header + $request_body would look like. Whatever the format, we wouldn't want this to be the default as it would definitely break downstream consumers as they exist today. I'm curious if anyone has suggestions on what those headers would look like?
When I was testing,crul localhost/ kafka-d "this is a test data",kafka consumers could not accept the data
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据 kafka and nginx No errors were reported
Could you please give me a whole test case, including the nginx configuration.
在 2019年6月25日,下午4:31,1349236893 notifications@github.com 写道:
When I was testing,crul localhost/ kafka-d "this is a test data",kafka consumers could not accept the data
— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/brg-liuwei/ngx_kafka_module/issues/13?email_source=notifications&email_token=ABNEOHZGLSVPBRT5X5WLWF3P4HJXZA5CNFSM4C4WKI5KYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODYPPCYI#issuecomment-505344353, or mute the thread https://github.com/notifications/unsubscribe-auth/ABNEOHZ6ILDPI6JC5BA4CT3P4HJXZANCNFSM4C4WKI5A.
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据 kafka and nginx No errors were reported
哦,麻烦贴一下你的nginx配置文件
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据 kafka和nginx没有报错
哦,麻烦贴一下你的nginx的配置文件
稍等
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据 kafka和nginx没有报错
哦,麻烦贴一下你的nginx的配置文件
user nobody;
worker_processes 1;
events { worker_connections 1024; }
http { include mime.types; default_type application/octet-stream;
#log_format main '$remote_addr - $remote_user [$time_local] "$request" '
# '$status $body_bytes_sent "$http_referer" '
# '"$http_user_agent" "$http_x_forwarded_for"';
#access_log logs/access.log main;
sendfile on;
#tcp_nopush on;
#keepalive_timeout 0;
keepalive_timeout 65;
#gzip on;
kafka;
kafka_broker_list jaydon-hadoop2:9092jaydon-hadoop3:9092jaydon-hadoop4:9092;
server {
listen 80;
server_name node-6.xiaoniu.com;
#charset koi8-r;
#access_log logs/host.access.log main;
location = /kafka/bike {
kafka_topic bike;
}
location = /kafka/user {
kafka_topic user;
}
#error_page 404 /404.html;
# redirect server error pages to the static page /50x.html
#
error_page 500 502 503 504 /50x.html;
location = /50x.html {
root html;
}
}
}
broker_list每个节点之间要以逗号分隔:
kafka_broker_list jaydon-hadoop2:9092,jaydon-hadoop3:9092,jaydon-hadoop4:9092;
请确保主机名jaydon-hadoopX
在你的nginx上可以被访问,如果你不确定的话,可以换成IPv4地址试试。
另外以后有问题请新建一个issue进行提问,本issue的主题和你提的问题无关,在这个issue下沟通会打扰到别人,谢谢。
------------------ 原始邮件 ------------------ 发件人: "Wei Liu"notifications@github.com; 发送时间: 2019年6月25日(星期二) 下午4:36 收件人: "brg-liuwei/ngx_kafka_module"ngx_kafka_module@noreply.github.com; 抄送: "..."1349236893@qq.com;"Comment"comment@noreply.github.com; 主题: Re: [brg-liuwei/ngx_kafka_module] NGINX common variables not passedto kafka topic (#13)
当我测试时,crul localhost / kafka-d“这是一个测试数据”,kafka消费者无法接受数据 kafka and nginx No errors were reported
哦,麻烦贴一下你的nginx配置文件
— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.
broker_list每个节点之间要以逗号分隔:
kafka_broker_list jaydon-hadoop2:9092,jaydon-hadoop3:9092,jaydon-hadoop4:9092;
请中主机确保名
jaydon-hadoopX
在你的nginx的上可以被访问,如果你不确定的话,可以换成IPv4的地址试试。另外以后有问题请新建一个问题进行提问,本课题的主题和你提的问题无关,在这个问题下沟通会打扰到别人,谢谢。
Okay, I'm sorry
Hi,
Would it be possible to pass some extra NGINX common variables to the data sent to the Kafka topic?
Currently, only the data of theNGINX variable $request_body is passed but , for instance, data from variables like $remote_addr would be great .
The goal is to have the IP of the client that is sending the data.