elastic / makelogs

109 stars 44 forks source link

Dockerize makelogs #55

Closed ycombinator closed 3 years ago

ycombinator commented 3 years ago

👋 , I am considering using makelogs for an Elastic-internal project. The project doesn't plan to run Node.js so it would be nice if makelogs was Dockerized. This PR takes a stab at that.

A follow up might be using GitHub Actions to build a new Docker image for makelogs upon every release and publish it as a GitHub Package to ghcr.io (or alternatively Docker Hub or the Elastic Docker Registry but I suspect we don't want to "pollute" those registries with something that's not maintained as stringently as other Elastic images in those registries?). Anyway, that's food for future thought - for now the scope of this PR is merely to introduce the ability to Dockerize makelogs.

ycombinator commented 3 years ago

but I think only QA uses it right now

Just curious, is there a new/better way now of loading test data into Kibana? Are developers just using one of the "getting started" sample data indices?

(For our internal project we need to simulate loading data from outside Elastic Cloud so loading "getting started" sample data won't work, but I'm still curious to know if there are alternatives to makelogs that are in vogue).

spalger commented 3 years ago

I don't know of any other comparable products at the moment, but the majority of this library is about generating data which looks decent in visualizations. Bulk indexing documents as quickly into ES as possible would be a pretty simple script to write and what I'd do if all I needed was to simulate some load.

ycombinator commented 3 years ago

Yeah, we're not looking to simulate indexing load (lots of indexing happening in a short period of time) as much as we're looking for something that'll load a large corpus of timeseries data spread out between datetime X and Y (so it ends up in multiple indices). I thought of makelogs because I think it can satisfy all those requirements via the --count, --days, and --indexInterval options.