AbsaOSS / enceladus

Dynamic Conformance Engine
Apache License 2.0
29 stars 14 forks source link

Update spark-job tests with Hermes #2168

Open miroslavpojer opened 1 year ago

miroslavpojer commented 1 year ago

Background

Spark-jobs test - Hermes tooling are not up to date.

Feature

Spark-job tests are up to date to current existing logic and rules.

Proposed Solution

Solution Ideas:

  1. Improve test data - remove duplicity (Aa >> ab, ac, …)
  2. Update reference data to be compatible with Spark 3
  3. Add missing rules
  4. Add missing ignores to get tests passing
  5. Add std & conf scenarios
  6. Try to change usage of property "enceladus-job-jar" to use current version instead of defined one.