search

GreptimeTeam / demo-scene

👾Scripts and samples to support Greptime Demos and Talks.
https://greptime.com
Apache License 2.0
25 stars 8 forks source link

feat: add a new pipeline to parse nginx log #46

Closed paomian closed 2 weeks ago

paomian commented 3 weeks ago
sunng87 commented 3 weeks ago

Do we need to modify the data source?

paomian commented 2 weeks ago

Do we need to modify the data source?

Do you want to create a real nginx log source ？

sunng87 commented 2 weeks ago

Yes, can we change the python script to produce some randomized log input in nginx log format (and randomized metrics as well)? It would be great to keep this demo lean while diverse in data.

paomian commented 2 weeks ago

Yes, can we change the python script to produce some randomized log input in nginx log format (and randomized metrics as well)? It would be great to keep this demo lean while diverse in data.

Generating a real Nginx log with Nginx is not a problem. The main problem is writing the Nginx log to Kafka. This may require an additional component. Using an existing vector to read the file is not a problem, but it is a bit more complicated.