search

GreptimeTeam / demo-scene

👾Scripts and samples to support Greptime Demos and Talks.
https://greptime.com
Apache License 2.0
29 stars 8 forks source link

feat: add a new pipeline to parse nginx log #46

Closed paomian closed 1 month ago

paomian commented 1 month ago
sunng87 commented 1 month ago

Do we need to modify the data source?

paomian commented 1 month ago

Do we need to modify the data source?

Do you want to create a real nginx log source ？

sunng87 commented 1 month ago

Yes, can we change the python script to produce some randomized log input in nginx log format (and randomized metrics as well)? It would be great to keep this demo lean while diverse in data.

paomian commented 1 month ago

Yes, can we change the python script to produce some randomized log input in nginx log format (and randomized metrics as well)? It would be great to keep this demo lean while diverse in data.

Generating a real Nginx log with Nginx is not a problem. The main problem is writing the Nginx log to Kafka. This may require an additional component. Using an existing vector to read the file is not a problem, but it is a bit more complicated.