search

ufvivotech / ufDataQualityImprovement

Project to improve UF vivo data quality and accuracy
2 stars 1 forks source link

Create a new person ingest based on HR concepts #109

Closed mconlon17 closed 10 years ago

mconlon17 commented 11 years ago

Create a new person ingest based on person concepts. See Ingest People

Hope to complete in May.

mconlon17 commented 11 years ago

Work has begun. A frame of a python program is written. Processes HR data and VIVO data and has three cases: 1) HR only (these people will be added); 2) HR & VIVO (these people will be updated and marked current); 3) VIVO only (these people will have current removed and contact info removed.

Will need to determine how to avoid processing people who belong in VIVO because they have consented.

mconlon17 commented 11 years ago

Added exception lists for departments (examples: PD and ACS), UFIDs (examples: test accounts), and position titles (examples Academic Lump Sum Payment). Maps position types to VIVO and excludes people who have positions that do not qualify for inclusion in VIVO. Improve JOBCODE_DESCRIPTIONS to make good-looking position_title. Counts of qualifying people from the HR data:

clinical-faculty 984 courtesy-faculty 4318 faculty 3066 housestaff 1149 non-academic 3870 postdoc 659 temp-faculty 748

Counts of the three cases (HR only => Add to VIVO, mark as current, add position and contact info; VIVO only => remove current and contact info, close position; HR and VIVO => update current, contact info and position): 1549 (mostly house staff), 20096 (retirees, temps, consultants, time card employees), 13245 (current faculty and professional staff)

mconlon17 commented 11 years ago

Software for people is essentially complete -- three cases are identified properly, handled properly -- people added when appropriate, attributes adding and being removed as expected. Adding an exception list for test UFIDs. Adding the privacy flag exceptions.

Next up -- modifying position data as appropriate -- when adding a person, add their position. When marking some as former, close their position. When updating someone, check the position and update or add as necessary.

nrejac commented 11 years ago

Properties from former people ingest:

UFID FirstName LastName MiddleName Prefix Suffix DisplayName (label) Gatorlink WorkingTitle (PreferredTitle) privacyFlag primaryEmail primaryPhoneNumber faxNumber homeDept

Classes: Deceased UFEntity

mconlon17 commented 11 years ago

Coding is complete. Unit testing complete. Integration testing underway.

mconlon17 commented 11 years ago

Testing in staging underway.

mconlon17 commented 11 years ago

Placed in production 9/15/13. One hand edit required after ingest to repair a dual employment profile.

Will complete documentation of process, repeat next week. Following second successful run in production, we should, after discussion and comment, close this issue.

mconlon17 commented 10 years ago

Has been in production for several weeks. Working as expected.