PPS List

[seadog007]

Filter (Validator): Filter out a record

Processor (Remover): Doing something to the record

Stage 1 (Line Filter/Processor)

• [x] Question Mark Filter (question-mark.s1.js) If there is a question mark in the line, which means something missing in the line, remove that line.

• [ ] CSV Comma Remover Parsing CSV for helping stage converter.

• [ ] TSV Tab Remover Parsing TSV for helping stage converter.

• [ ] Space Remover Remove all kind of wired space, highly change appear between name where which is only 2 chinese character.

Stage 2 ([name, id] Filter/Processor)

• [x] 10 Digit ID to 6 digit (10d26d.s2.js) Coverter 10 digit id to last 6 digit for hashing.

• [x] A123456789 Filter (dummy-id.s2.js) Too many testing data using this....

• [x] Name English Filter (name-english-filter.s2.js) If there is English character in the name, then filter it out.

• [x] ID Validator (id-validate.s2.js) Check if ID is vaild.

• [ ] Unifier Unify the data, be sure the data will not repeat.

Stage Converter

• [x] Stage 1 to Stage 2 Converter (s1-to-s2.js) Convert line to [name, id]

• [x] Stage 2 to Final By hashing the name + id form stage 2

[seadog007]

76

ID Validator Name English filter

[seadog007]

Example of Trash TSV (Copy and paste from Excel)

Need some filter for this