search

AbsaOSS / spline-spark-agent

Spline agent for Apache Spark
https://absaoss.github.io/spline/
Apache License 2.0
185 stars 95 forks source link

Customized Listener #799

Closed lantse closed 7 months ago

lantse commented 7 months ago

Discussed in https://github.com/AbsaOSS/spline-spark-agent/discussions/798

Originally posted by **lantse** March 28, 2024 We are using mappartition function in Spark to transform data in batch. The data read()/write() for each record are in a user defined adapter class. Is there a way to let the spline spark agent to `listen` to the write() function so that this specific transformation would not be missing in the data lineage record? In another word, how do we supplement or add to the data lineage record for the area that the agent cannot collect by default? Thanks.
wajda commented 7 months ago

please do not duplicate messages in discussions and issues.