search

LucaCanali / sparkMeasure

This is the development repository for sparkMeasure, a tool and library designed for efficient analysis and troubleshooting of Apache Spark jobs. It focuses on easing the collection and examination of Spark metrics, making it a practical choice for both developers and data engineers.
Apache License 2.0
706 stars 145 forks source link

Question: InfluxDB 2.x Support #55

Open MasonLegere opened 8 months ago

MasonLegere commented 8 months ago

Is there support for InfluxDB 2.0 using Flight Recorder mode? The current credentials seem to only support username/password authentication versus token/org credentials needed for InfluxDB 2.0

 val url = Utils.parseInfluxDBURL(conf, logger)
  val (username, password) = Utils.parseInfluxDBCredentials(conf, logger)

  // Tries to connect to InfluxDB, using the given URL and credentials
  val influxDB =  username match {
    case username if username.isEmpty =>
      // no username and password, InfluxDB must be running with auth-enabled=false
      InfluxDBFactory.connect(url)
    case _ => InfluxDBFactory.connect(url, username, password)
  }
LucaCanali commented 8 months ago

The current version (sparkMeasure 0.24) supports only InfluxDB 1.x line protocol, see also: https://github.com/LucaCanali/sparkMeasure/blob/master/docs/Flight_recorder_mode_InfluxDBSink.md Support for InfluxDB 2.x may be added in a future version.

MasonLegere commented 8 months ago

Thanks for the fast response @LucaCanali! If you're open to contributions I can take a look at adding InfluxDB 2.x support in next couple of weeks but can't promise much for a timeline :)

LucaCanali commented 8 months ago

Thank you for the offer @MasonLegere . I'd like also to understand what's the need and how many people would be interested in this. Every integration brings in dependencies, so it would be nice to know that there is interested for real-world use cases.