Open kunalv89 opened 2 years ago
@kunalv89 Hi, I'm sorry for my late reply. Was the problem solved?
@kunalv89 Hi, I'm sorry for my late reply. Was the problem solved?
@mia-0032 , Thanks for the respond, Yes problem has been resolved, i have migrated my work from td-agent to fluentd. but again i'm facing issue while creating dynamic topic name and here is my configuration parameters. am not sure, is i'am using right placeholder.
`
`
Note : %{toolManagerName}, is coming in the data as a key as below data o/p - {"processid":"9","toolmanagerName":"loft","paramid":"****"}
here is my gcloud_pubsub version - fluent-plugin-gcloud-pubsub-custom (1.6.0)
@kunalv89 Could you try below config? This configuration uses buffer placeholder feature of Fluentd. https://docs.fluentd.org/configuration/buffer-section#placeholders
<match tool_event>
@type gcloud_pubsub
project ********36
key /home/fluentd/pqr.json
topic ${toolManagerName}
autocreate_topic true
max_messages 1000
max_total_size 9800000
max_message_size 4000000
timeout 60
<buffer toolManagerName>
@type file path /var/log/fluentd/buffer
flush_interval 3s
</buffer>
<format>
@type json
</format>
</match>
@kunalv89 Could you try below config? This configuration uses buffer placeholder feature of Fluentd. https://docs.fluentd.org/configuration/buffer-section#placeholders
<match tool_event> @type gcloud_pubsub project ********36 key /home/fluentd/pqr.json topic ${toolManagerName} autocreate_topic true max_messages 1000 max_total_size 9800000 max_message_size 4000000 timeout 60 <buffer toolManagerName> @type file path /var/log/fluentd/buffer flush_interval 3s </buffer> <format> @type json </format> </match>
@mia-0032 , this is one is flawless. but just i got to know that, data is going on to PubSub instead of PubSub lite. For PubSub Lite, are we require any additional plugin to transmitting data.
Hello @mia-0032 , Can we do PubSub Lite integration with this plugin on fluentd, i have searched alot for pubsub lite plugin but there is no direct plugin available.
@kunalv89 This plugin uses google-cloud-pubsub gem. But the gem doesn't support Pub/Sub Lite. Please see https://github.com/googleapis/google-cloud-ruby/issues/8930 . If the gem will support Pub/Sub Lite, I will support Pub/Sub Lite on this plugin.
Hi Team, I'am using this plugin as a fluentd output. but am using pubsub lite instead of pubsub. I don't get any error and any response even though the topic has not been created using this. Please suggest.
`<match **> @type gcloud_pubsub project *****6 key /etc/td-agent/****.json topic topic_3 autocreate_topic true max_messages 1000 max_total_size 9800000 max_message_size 4000000s timeout 60
`