Open seadog007 opened 4 years ago
[x] Question Mark Filter (question-mark.s1.js) If there is a question mark in the line, which means something missing in the line, remove that line.
[ ] CSV Comma Remover Parsing CSV for helping stage converter.
[ ] TSV Tab Remover Parsing TSV for helping stage converter.
[ ] Space Remover Remove all kind of wired space, highly change appear between name where which is only 2 chinese character.
[x] 10 Digit ID to 6 digit (10d26d.s2.js) Coverter 10 digit id to last 6 digit for hashing.
[x] A123456789 Filter (dummy-id.s2.js) Too many testing data using this....
[x] Name English Filter (name-english-filter.s2.js) If there is English character in the name, then filter it out.
[x] ID Validator (id-validate.s2.js) Check if ID is vaild.
[ ] Unifier Unify the data, be sure the data will not repeat.
[x] Stage 1 to Stage 2 Converter (s1-to-s2.js) Convert line to [name, id]
[x] Stage 2 to Final By hashing the name + id form stage 2
name + id
ID Validator Name English filter
Example of Trash TSV (Copy and paste from Excel)
Need some filter for this
Filter (Validator): Filter out a record
Processor (Remover): Doing something to the record
Stage 1 (Line Filter/Processor)
[x] Question Mark Filter (question-mark.s1.js) If there is a question mark in the line, which means something missing in the line, remove that line.
[ ] CSV Comma Remover Parsing CSV for helping stage converter.
[ ] TSV Tab Remover Parsing TSV for helping stage converter.
[ ] Space Remover Remove all kind of wired space, highly change appear between name where which is only 2 chinese character.
Stage 2 ([name, id] Filter/Processor)
[x] 10 Digit ID to 6 digit (10d26d.s2.js) Coverter 10 digit id to last 6 digit for hashing.
[x] A123456789 Filter (dummy-id.s2.js) Too many testing data using this....
[x] Name English Filter (name-english-filter.s2.js) If there is English character in the name, then filter it out.
[x] ID Validator (id-validate.s2.js) Check if ID is vaild.
[ ] Unifier Unify the data, be sure the data will not repeat.
Stage Converter
[x] Stage 1 to Stage 2 Converter (s1-to-s2.js) Convert line to [name, id]
[x] Stage 2 to Final By hashing the
name + id
form stage 2