data quality management

phillipcheng / log.analysis

1 stars 15 forks source link

data quality management #120

Open phillipcheng opened 8 years ago

phillipcheng commented 8 years ago

when some files are invalid, should output to a certain place for further processing.

hanzac commented 8 years ago

How about we provide another configuration for the ETL commands - failed.file.dir For map deduce, it could gather all these invalid csv records to one file For java process, it could be the failed input files

phillipcheng commented 7 years ago

for example, when loadDataCmd face network issue, the data should be backed up and load later.