phillipcheng / log.analysis

1 stars 15 forks source link

loadDBCmd should load according to the files, not according to the schema #143

Closed phillipcheng closed 7 years ago

hanzac commented 7 years ago

This issue is fixed.

Support load DB according to the files found from the input path Configuration is the same, but inherit the existing file table mapping config e.g. file.table.map=filename.substring(0\,filename.indexOf('.csv'))

The process logic is: first in mapping process, group all files by the tablename, if no table name specified, group all files to the key In reduce process, load all files table by table.

Total loaded rows will be saved to the output file (part-r-????)of reduce process.

The original java single process is still supported.