It would be nice to allow the user to define arbitrary checks which will also be included into the reports. This can be achieved by allowing arbitrary code and wrapping it into a try catch internally. It should return the Try so you can work with it later. If the Try contains an exception the check will be marked as failed.
Usage Sketch
val tryRdd = Check.custom("file exists", {
spark.textFile("myfile.txt")
})
tryRdd.map(_.count) // do something with the read RDD
Questions
When should these checks be executed? On creation or when the run method is triggered?
Should it be possible to chain the checks? So use it to call another try which takes the previous try as input before you finally run them and receive the final output as a try?
Description
It would be nice to allow the user to define arbitrary checks which will also be included into the reports. This can be achieved by allowing arbitrary code and wrapping it into a try catch internally. It should return the
Tryso you can work with it later. If theTrycontains an exception the check will be marked as failed.Usage Sketch
Questions