search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
183 stars 93 forks source link

ConfigurationConverter.getProperties is unable to hundle Integer type values from Yaml #610

Closed cerveada closed 1 year ago

cerveada commented 1 year ago

For example:

spline.lineageDispatcher.kafka.producer.max.in.flight.requests.per.connection:1

This is fixed in commons-configuration:1.7 https://issues.apache.org/jira/browse/CONFIGURATION-432

Unfortunately, Spark 2.4 depends on version 1.6

There already were other issues with YAML as version 1.2 being interpreted as float and there may be more. Other properties sources just treat everything as string and do the conversion once it is asked for. Maybe we could use Yaml the same way.

cerveada commented 1 year ago

Reported in #609

cerveada commented 1 year ago

Should we keep this open for future complete fix? Since #611 is only fixing the default, not removing the issue completely.

wajda commented 1 year ago

I'd like to propose a proper fix. Give me a sec.

wajda commented 1 year ago

Since the above PR is already merged, I'll move the rest part to a new issue.