Lock files should be in a separate dir and turn on/off through config

TresAmigosSD / SMV

Spark Modularized View

Apache License 2.0

42 stars 22 forks source link

Lock files should be in a separate dir and turn on/off through config #1473

Closed ninjapapa closed 5 years ago

ninjapapa commented 5 years ago

Create $dataDir/lock dir, and put all lock files there. Also introduce smv.lockDir and smv.lock options.

ninjapapa commented 5 years ago

When lock file is in use, it should lock the persisting + post_action together to fully prevent data-metadata out of synch, so the lock can't apply in SmvIoStrategy, instead need to be in the computeData method of SmvGenericModule.