Initial draft for go-commons with indexer

[vishnuchalla]

Description

Adding indexers to go commons.

Testing

Tested integrating with kube-burner for elastic indexer.

time="2023-04-18 08:49:14" level=info msg="📁 Creating indexer: elastic" file="factory.go:31"
time="2023-04-18 08:49:18" level=info msg="👽 Initializing prometheus client with URL: https://prometheus-k8s-openshift-monitoring.apps.vchalla-quay-test-2.perfscale.devcluster.openshift.com" file="prometheus.go:55"
time="2023-04-18 08:49:21" level=info msg="Scraping for the prometheus entry with params - Endpoint: https://prometheus-k8s-openshift-monitoring.apps.vchalla-quay-test-2.perfscale.devcluster.openshift.com, Profile: /home/vchalla/quay-metrics.yaml, Start: 2023-04-18T07:49:14-04:00, End: 2023-04-18T08:49:14-04:00" file="utils.go:67"
time="2023-04-18 08:49:21" level=info msg="Indexing metrics with UUID test-uuid" file="utils.go:72"
time="2023-04-18 08:49:21" level=info msg="🔍 Scraping prometheus metrics for benchmark from 2023-04-18T07:49:14-04:00 to 2023-04-18T08:49:14-04:00" file="prometheus.go:115"
time="2023-04-18 08:49:22" level=info msg="Indexing metric QuayRequestDuration" file="elastic.go:86"
time="2023-04-18 08:49:29" level=info msg="Indexing metric QuaySecScanIndexDuration" file="elastic.go:86"
time="2023-04-18 08:49:35" level=info msg="Indexing metric QuayModelCacheTotalRate" file="elastic.go:86"
time="2023-04-18 08:49:35" level=info msg="Indexing metric QuayModelCacheTotalSum" file="elastic.go:86"
time="2023-04-18 08:49:36" level=info msg="Indexing metric QuayDbConnectCallsRate" file="elastic.go:86"
time="2023-04-18 08:49:43" level=info msg="Indexing metric QuayDbConnectCallsSum" file="elastic.go:86"
time="2023-04-18 08:49:47" level=info msg="Indexing metric QuayDbCloseCallsRate" file="elastic.go:86"
time="2023-04-18 08:49:56" level=info msg="Indexing metric QuayDbCloseCallsSum" file="elastic.go:86"
time="2023-04-18 08:50:00" level=info msg="Indexing metric QuayDbPooledConnectionsAvailableRate" file="elastic.go:86"
time="2023-04-18 08:50:04" level=info msg="Indexing metric QuayDbPooledConnectionsAvailableSum" file="elastic.go:86"
time="2023-04-18 08:50:10" level=info msg="Indexing metric QuayDbPooledConnectionsInUseRate" file="elastic.go:86"
time="2023-04-18 08:50:14" level=info msg="Indexing metric QuayDbPooledConnectionsInUseSum" file="elastic.go:86"
time="2023-04-18 08:50:20" level=info msg="👋 Exiting kube-burner test-uuid" file="kube-burner.go:198"

Tested integrating with kube-buner for opensearch indexer as well.

time="2023-04-18 15:23:51" level=info msg="📁 Creating indexer: opensearch" file="factory.go:31"
time="2023-04-18 15:23:55" level=info msg="👽 Initializing prometheus client with URL: https://prometheus-k8s-openshift-monitoring.apps.vchalla-quay-test-2.perfscale.devcluster.openshift.com" file="prometheus.go:55"
time="2023-04-18 15:23:56" level=info msg="Scraping for the prometheus entry with params - Endpoint: https://prometheus-k8s-openshift-monitoring.apps.vchalla-quay-test-2.perfscale.devcluster.openshift.com, Profile: /home/vchalla/quay-metrics.yaml, Start: 2023-04-18T09:55:04-04:00, End: 2023-04-18T15:16:04-04:00" file="utils.go:67"
time="2023-04-18 15:23:56" level=info msg="Indexing metrics with UUID f0c5e8ca-51f7-41fe-ad70-7c16289a6cc5" file="utils.go:72"
time="2023-04-18 15:23:56" level=info msg="🔍 Scraping prometheus metrics for benchmark from 2023-04-18T09:55:04-04:00 to 2023-04-18T15:16:04-04:00" file="prometheus.go:115"
time="2023-04-18 15:23:58" level=info msg="Indexing metric QuayRequestDuration" file="opensearch.go:74"
time="2023-04-18 15:24:09" level=info msg="Indexing metric QuaySecScanIndexDuration" file="opensearch.go:74"
time="2023-04-18 15:24:15" level=info msg="Indexing metric QuayModelCacheTotalRate" file="opensearch.go:74"
time="2023-04-18 15:24:21" level=info msg="Indexing metric QuayModelCacheTotalSum" file="opensearch.go:74"
time="2023-04-18 15:24:25" level=info msg="Indexing metric QuayDbConnectCallsRate" file="opensearch.go:74"
time="2023-04-18 15:24:33" level=info msg="Indexing metric QuayDbConnectCallsSum" file="opensearch.go:74"
time="2023-04-18 15:24:40" level=info msg="Indexing metric QuayDbCloseCallsRate" file="opensearch.go:74"
time="2023-04-18 15:24:45" level=info msg="Indexing metric QuayDbCloseCallsSum" file="opensearch.go:74"
time="2023-04-18 15:24:49" level=info msg="Indexing metric QuayDbPooledConnectionsAvailableRate" file="opensearch.go:74"
time="2023-04-18 15:24:54" level=info msg="Indexing metric QuayDbPooledConnectionsAvailableSum" file="opensearch.go:74"
time="2023-04-18 15:25:00" level=info msg="Indexing metric QuayDbPooledConnectionsInUseRate" file="opensearch.go:74"
time="2023-04-18 15:25:04" level=info msg="Indexing metric QuayDbPooledConnectionsInUseSum" file="opensearch.go:74"
time="2023-04-18 15:25:09" level=info msg="👋 Exiting kube-burner f0c5e8ca-51f7-41fe-ad70-7c16289a6cc5" file="kube-burner.go:198"

[vishnuchalla]

@rook and @rsevilla87 - Both elasticsearch and opensearch APIs share exactly same code (See elastic.go and opensearch.go). We can have any one of them for indexing functionality or else both in case we want to have some backward compatability. My idea is to fully migrate to opensearch. Related blogs:

• https://dattell.com/data-architecture-blog/opensearch-vs-elasticsearch/
• https://logit.io/blog/post/aws-elasticsearch-vs-opensearch/

Please feel free to share your thoughts here. Thanks!

[rsevilla87]

@rook and @rsevilla87 - Both elasticsearch and opensearch APIs share exactly same code (See elastic.go and opensearch.go). We can have any one of them for indexing functionality or else both in case we want to have some backward compatability. My idea is to fully migrate to opensearch. Related blogs:

* https://dattell.com/data-architecture-blog/opensearch-vs-elasticsearch/

* https://logit.io/blog/post/aws-elasticsearch-vs-opensearch/

Please feel free to share your thoughts here. Thanks!

thanks for the research!, I don't have a strong opinion on this, but I think we can keep both for backwards compatibility in case any user is using a recent version of elasticsearch (OpenSearch is compatible up to the 7.10 version). I'll add some notes to the kube-burner docs about this.

[jtaleric]

@rook and @rsevilla87 - Both elasticsearch and opensearch APIs share exactly same code (See elastic.go and opensearch.go). We can have any one of them for indexing functionality or else both in case we want to have some backward compatability. My idea is to fully migrate to opensearch. Related blogs:

• https://dattell.com/data-architecture-blog/opensearch-vs-elasticsearch/
• https://logit.io/blog/post/aws-elasticsearch-vs-opensearch/

Please feel free to share your thoughts here. Thanks!

Yeah - opensearch works fine for me... k8s-netperf started with opensearch, so I would defer to @rsevilla87 on eventually moving to opensearch from es.

[rsevilla87]

Hey @vishnuchalla, I just sent a PR to your fork's branch containing the logger interface abstraction we discussed about in yesterday's meeting. https://github.com/vishnuchalla/go-commons/pull/1

[vishnuchalla]

Hey @vishnuchalla, I just sent a PR to your fork's branch containing the logger interface abstraction we discussed about in yesterday's meeting. vishnuchalla#1

@rsevilla87 - Thanks for the quick help. IMHO, I don't think we should be passing logger as an argument to functions in go-commons (because we do not do it for any of the go imports in our code). Instead, either

• We should completely ignore logging in go-commons and do logging in our own applications based on the response returned by the function calls in go-commons (i.e if success log something or if error log something, because none of our upstream packages like context, os, path/filepath, time etc worry about logging). (OR)
  • We need to have a common interface implemented for logger something like [this](http://%20copyright%202020%20the%20kube-burner%20authors.%20//%20//%20Licensed%20under%20the%20Apache%20License,%20Version%202.0%20(the%20%22License%22);%20//%20you%20may%20not%20use%20this%20file%20except%20in%20compliance%20with%20the%20License.%20//%20You%20may%20obtain%20a%20copy%20of%20the%20License%20at%20//%20//%20%20%20%20%20%20http://www.apache.org/licenses/LICENSE-2.0%20//%20//%20Unless%20required%20by%20applicable%20law%20or%20agreed%20to%20in%20writing,%20software%20//%20distributed%20under%20the%20License%20is%20distributed%20on%20an%20%22AS%20IS%22%20BASIS,%20//%20WITHOUT%20WARRANTIES%20OR%20CONDITIONS%20OF%20ANY%20KIND,%20either%20express%20or%20implied.%20//%20See%20the%20License%20for%20the%20specific%20language%20governing%20permissions%20and%20//%20limitations%20under%20the%20License.%20%20package%20log%20%20import%20(%20%22os%22%20%20%22github.com/sirupsen/logrus%22%20)%20%20//%20Logger%20defines%20a%20set%20of%20methods%20for%20writing%20application%20logs.%20Derived%20from%20and%20//%20inspired%20by%20logrus.Entry.%20type%20Logger%20interface%20%7B%20Trace(args%20...interface%7B%7D)%20Tracef(format%20string,%20args%20...interface%7B%7D)%20Traceln(args%20...interface%7B%7D)%20Debug(args%20...interface%7B%7D)%20Debugf(format%20string,%20args%20...interface%7B%7D)%20Debugln(args%20...interface%7B%7D)%20Error(args%20...interface%7B%7D)%20Errorf(format%20string,%20args%20...interface%7B%7D)%20Errorln(args%20...interface%7B%7D)%20Fatal(args%20...interface%7B%7D)%20Fatalf(format%20string,%20args%20...interface%7B%7D)%20Fatalln(args%20...interface%7B%7D)%20Info(args%20...interface%7B%7D)%20Infof(format%20string,%20args%20...interface%7B%7D)%20Infoln(args%20...interface%7B%7D)%20Panic(args%20...interface%7B%7D)%20Panicf(format%20string,%20args%20...interface%7B%7D)%20Panicln(args%20...interface%7B%7D)%20Print(args%20...interface%7B%7D)%20Printf(format%20string,%20args%20...interface%7B%7D)%20Println(args%20...interface%7B%7D)%20Warn(args%20...interface%7B%7D)%20Warnf(format%20string,%20args%20...interface%7B%7D)%20Warning(args%20...interface%7B%7D)%20Warningf(format%20string,%20args%20...interface%7B%7D)%20Warningln(args%20...interface%7B%7D)%20Warnln(args%20...interface%7B%7D)%20%7D%20%20var%20defaultLogger%20*logrus.Logger%20%20func%20init()%20%7B%20defaultLogger%20=%20newLogrusLogger()%20%7D%20%20//%20SetLogLevel%20sets%20log%20level%20func%20SetLogLevel(logLevel%20string)%20%7B%20switch%20logLevel%20%7B%20case%20%22debug%22:%20defaultLogger.Level%20=%20logrus.DebugLevel%20case%20%22warn%22:%20defaultLogger.Level%20=%20logrus.WarnLevel%20case%20%22info%22:%20defaultLogger.Level%20=%20logrus.InfoLevel%20case%20%22error%22:%20defaultLogger.Level%20=%20logrus.ErrorLevel%20case%20%22fatal%22:%20defaultLogger.Level%20=%20logrus.FatalLevel%20default:%20Fatalf(%22Unknown%20log%20level%20%s%22,%20logLevel)%20%7D%20%7D%20%20func%20newLogrusLogger()%20*logrus.Logger%20%7B%20l%20:=%20logrus.New()%20l.Out%20=%20os.Stderr%20customFormatter%20:=%20new(logrus.TextFormatter)%20customFormatter.TimestampFormat%20=%20%222006-01-02%2015:04:05%22%20customFormatter.FullTimestamp%20=%20true%20l.SetFormatter(customFormatter)%20return%20l%20%7D%20%20//%20Fields%20is%20a%20map%20string%20interface%20to%20define%20fields%20in%20the%20structured%20log%20type%20Fields%20map[string]interface%7B%7D%20%20//%20With%20allow%20us%20to%20define%20fields%20in%20out%20structured%20logs%20func%20(f%20Fields)%20With(k%20string,%20v%20interface%7B%7D)%20Fields%20%7B%20f[k]%20=%20v%20return%20f%20%7D%20%20//%20WithFields%20allow%20us%20to%20define%20fields%20in%20out%20structured%20logs%20func%20(f%20Fields)%20WithFields(f2%20Fields)%20Fields%20%7B%20for%20k,%20v%20:=%20range%20f2%20%7B%20f[k]%20=%20v%20%7D%20return%20f%20%7D%20%20//%20WithFields%20allow%20us%20to%20define%20fields%20in%20out%20structured%20logs%20func%20WithFields(fields%20Fields)%20Logger%20%7B%20return%20defaultLogger.WithFields(logrus.Fields(fields))%20%7D%20%20//%20Trace%20package-level%20convenience%20method.%20func%20Trace(args%20...interface%7B%7D)%20%7B%20defaultLogger.Trace(args...)%20%7D%20%20//%20Tracef%20package-level%20convenience%20method.%20func%20Tracef(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Tracef(format,%20args...)%20%7D%20%20//%20Traceln%20package-level%20convenience%20method.%20func%20Traceln(args%20...interface%7B%7D)%20%7B%20defaultLogger.Traceln(args...)%20%7D%20%20//%20Debug%20package-level%20convenience%20method.%20func%20Debug(args%20...interface%7B%7D)%20%7B%20defaultLogger.Debug(args...)%20%7D%20%20//%20Debugf%20package-level%20convenience%20method.%20func%20Debugf(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Debugf(format,%20args...)%20%7D%20%20//%20Debugln%20package-level%20convenience%20method.%20func%20Debugln(args%20...interface%7B%7D)%20%7B%20defaultLogger.Debugln(args...)%20%7D%20%20//%20Error%20package-level%20convenience%20method.%20func%20Error(args%20...interface%7B%7D)%20%7B%20defaultLogger.Error(args...)%20%7D%20%20//%20Errorf%20package-level%20convenience%20method.%20func%20Errorf(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Errorf(format,%20args...)%20%7D%20%20//%20Errorln%20package-level%20convenience%20method.%20func%20Errorln(args%20...interface%7B%7D)%20%7B%20defaultLogger.Errorln(args...)%20%7D%20%20//%20Fatal%20package-level%20convenience%20method.%20func%20Fatal(args%20...interface%7B%7D)%20%7B%20defaultLogger.Fatal(args...)%20%7D%20%20//%20Fatalf%20package-level%20convenience%20method.%20func%20Fatalf(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Fatalf(format,%20args...)%20%7D%20%20//%20Fatalln%20package-level%20convenience%20method.%20func%20Fatalln(args%20...interface%7B%7D)%20%7B%20defaultLogger.Fatalln(args...)%20%7D%20%20//%20Info%20package-level%20convenience%20method.%20func%20Info(args%20...interface%7B%7D)%20%7B%20defaultLogger.Info(args...)%20%7D%20%20//%20Infof%20package-level%20convenience%20method.%20func%20Infof(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Infof(format,%20args...)%20%7D%20%20//%20Infoln%20package-level%20convenience%20method.%20func%20Infoln(args%20...interface%7B%7D)%20%7B%20defaultLogger.Infoln(args...)%20%7D%20%20//%20Panic%20package-level%20convenience%20method.%20func%20Panic(args%20...interface%7B%7D)%20%7B%20defaultLogger.Panic(args...)%20%7D%20%20//%20Panicf%20package-level%20convenience%20method.%20func%20Panicf(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Panicf(format,%20args...)%20%7D%20%20//%20Panicln%20package-level%20convenience%20method.%20func%20Panicln(args%20...interface%7B%7D)%20%7B%20defaultLogger.Panicln(args...)%20%7D%20%20//%20Print%20package-level%20convenience%20method.%20func%20Print(args%20...interface%7B%7D)%20%7B%20defaultLogger.Print(args...)%20%7D%20%20//%20Printf%20package-level%20convenience%20method.%20func%20Printf(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Printf(format,%20args...)%20%7D%20%20//%20Println%20package-level%20convenience%20method.%20func%20Println(args%20...interface%7B%7D)%20%7B%20defaultLogger.Println(args...)%20%7D%20%20//%20Warn%20package-level%20convenience%20method.%20func%20Warn(args%20...interface%7B%7D)%20%7B%20defaultLogger.Warn(args...)%20%7D%20%20//%20Warnf%20package-level%20convenience%20method.%20func%20Warnf(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Warnf(format,%20args...)%20%7D%20%20//%20Warning%20package-level%20convenience%20method.%20func%20Warning(args%20...interface%7B%7D)%20%7B%20defaultLogger.Warning(args...)%20%7D%20%20//%20Warningf%20package-level%20convenience%20method.%20func%20Warningf(format%20string,%20args%20...interface%7B%7D)%20%7B%20defaultLogger.Warningf(format,%20args...)%20%7D%20%20//%20Warningln%20package-level%20convenience%20method.%20func%20Warningln(args%20...interface%7B%7D)%20%7B%20defaultLogger.Warningln(args...)%20%7D%20%20//%20Warnln%20package-level%20convenience%20method.%20func%20Warnln(args%20...interface%7B%7D)%20%7B%20defaultLogger.Warnln(args...)%20%7D), with logrus or something, and make our individual applications use this go-commons logger instead of their own ones.

Open for other thoughts too. Please let me know your thoughts here. Thank you!

cc: @jtaleric