kmubiin
commented
6 years ago CIDB data were deceptively clean... The "directors" field itself is missing for some entries.
Then, even the field exist, there are 5 subfields: "idenfity_card_no", "name", "nationality", "shares", "year_of_experience" of which may be also empty!
Bad data overview (see WARNING):
Get list of JSONL files
Found files: 5
Read CIDB data from JSONL files
Read "directors" from contractors201509220.jsonl
Missing "directors": 8 (WARNING)
Found "directors": 4993
of which empty: 2475 (WARNING)
of which non-empty: 2518
but empty subfield: 2890 (WARNING)
Total lines checked: 5001
Read "directors" from contractors201509221.jsonl
Missing "directors": 18 (WARNING)
Found "directors": 4983
of which empty: 2451 (WARNING)
of which non-empty: 2532
but empty subfield: 2841 (WARNING)
Total lines checked: 5001
Read "directors" from contractors201509222.jsonl
Missing "directors": 9 (WARNING)
Found "directors": 4992
of which empty: 2356 (WARNING)
of which non-empty: 2636
but empty subfield: 2854 (WARNING)
Total lines checked: 5001
Read "directors" from contractors201509224.jsonl
Missing "directors": 7 (WARNING)
Found "directors": 4994
of which empty: 2417 (WARNING)
of which non-empty: 2577
but empty subfield: 2824 (WARNING)
Total lines checked: 5001
Read "directors" from contractors201509226.jsonl
Missing "directors": 14 (WARNING)
Found "directors": 4987
of which empty: 2415 (WARNING)
of which non-empty: 2572
but empty subfield: 2980 (WARNING)
Total lines checked: 5001Based on above, there are three levels of checking required:
- Check if "directors" field exist or else
- Check if "directors" field is empty or else
- Check if any of known subfields is empty or else
Hence the conversion script must implement above checking to pass.
P.S.: CIDB data took longer time to identify bad data compared to JKR or MyProcurement data. I hope I didn't miss other bad data besides my checking above... Will start writing script soon.
P.S.S.: Pasted correct output. Now contains "but empty subfield" information.
See Joint Ownership example as often companies have 2-3 directors http://beneficial-ownership-data-standard.readthedocs.io/en/master/examples.html