search

TresAmigosSD / SMV

Spark Modularized View
Apache License 2.0
42 stars 22 forks source link

Can we git rid of *.valid? #1259

Closed laneb closed 5 years ago

laneb commented 6 years ago

@AliTajeldin has suggested that the .valid files (which record DQM validation reports) are not used and can be discarded.

ninjapapa commented 6 years ago

There used to be a logic that if a DF validated (DQM action ran) it should not be validated again. Not sure whether that logic still needed.

ninjapapa commented 5 years ago

Since dqm results are in metadata already, and we are supporting quick run option, so no need to use .valid as a semaphore. Delete

ninjapapa commented 5 years ago

Will be solved in #1249.