search

cloud-bulldozer / ocm-api-load

A set of load tests for OCM's clusters-service, based on vegeta.
Apache License 2.0
0 stars 8 forks source link

Integrates shared library go-commons - ES indexing #118

Closed chentex closed 1 year ago

chentex commented 1 year ago

Description

Removes local implementation of ES indexer and favors go-commons library.

Fixes

Fix #115

vishnuchalla commented 1 year ago

Thanks for this integration. From here, It looks like we have results in a file and then we are getting it in memory and indexing multiple docs to ES. Instead can't we simply have all the data in memory and pass that list to the indexer to index all at once? just trying to see if we can get rid of that file if it is redundant. And also can you paste the content of the file here just for a rought idea, please!

rsevilla87 commented 1 year ago

Thanks for this integration. From here, It looks like we have results in a file and then we are getting it in memory and indexing multiple docs to ES. Instead can't we simply have all the data in memory and pass that list to the indexer to index all at once? just trying to see if we can get rid of that file if it is redundant. And also can you paste the content of the file here just for a rought idea, please!

Apart from this, the library also includes a local file indexer in case you're interested on dumping the metrics in files. https://github.com/cloud-bulldozer/go-commons/blob/main/indexers/local.go

chentex commented 1 year ago

As for the moment the file is not redundant, since is the file vegeta library pours all it's data to, up to 100,000 items in case of some tests scenarios, after that we load to file to apply some data to identify it and then push it to ES.

I can try having only a writer and see if it will hold everything in memory and the app doesn't crash.