danthegoodman1 / icedb

An in-process Parquet merge engine for better data warehousing in S3 with MVCC
https://blog.danthegoodman.com/icedb-v3--third-times-the-charm
Other
131 stars 5 forks source link

HTTP log ingestion example #83

Open danthegoodman1 opened 1 year ago

danthegoodman1 commented 1 year ago

Take in a JSON log, expand to columns, perform queries.

We can probably pre-define a data set that is useful and then can run queries against it. Or we can download something from S3/github and show against real data. We can probably store this in a folder in this repo as an example dataset too since pip only pulls from what's in the icedb directory in this repo.

Then write a query that shows things like percentile latencies by endpoint, top 1 most hit endpoints, abnormally slow requests.