Current SmvParquetOnHdfsIoStrategy's implementation on isPersisted is just to check whether the *.parquet file exist. It is risky to have some leftover half-written parquet data files ruin the whole result. Need to introduce a semaphore file which is created after the parquet file finished written.
Current SmvParquetOnHdfsIoStrategy's implementation on
isPersistedis just to check whether the*.parquetfile exist. It is risky to have some leftover half-written parquet data files ruin the whole result. Need to introduce a semaphore file which is created after the parquet file finished written.