AbsaOSS / atum-service

Apache License 2.0
5 stars 1 forks source link

Feature/create capture dispatcher #175

Closed filiphornak closed 5 months ago

filiphornak commented 5 months ago

Simplify writing tests for atum agent

There is an option to configure it:

atum.dispatcher.type = "capture"
atum.dispatcher.capture = {
  checkpoints-limit = 100
}

But most likely here is how it would be used

// za.co.absa.atum.agent
// imports ...
class TestAgent(disp: Dispatcher) extends AtumAgent {
  override val dispatcher: Dispatcher = disp
}

// in pramen sink
class AtumSink(agent: AtumAgent, ...) extends Sink {...}

new AtumSink(AtumAgent, ...)

// in tests
val cd = new CapturingDispatcher(0)
val agent = new TestAgent(cd)
val underTest = new AtumSink(agent, ...)

underTest.send(...)

cd.getPartition("key1=value1/key2=value2/key3=value3").checkpoints.size shouldBe 3

Closes #168