Sync TC candidates to SF

[samschlicht]

As a TBB staff member I would like to be able to access and analyse live-ish candidate data using Salesforce and Sopact reporting tools To better understand and present information regarding our candidates

This was primarily instigated by the Sopact-TBB team's request for access to this data for the impact dashboard. There have also been other requests, such as from the comms team to populate their candidate mailing list and more efficiently produce the Talent Catalog snapshot.

John met with Sopact and tentatively settled on the most efficient and secure way of achieving this being to extend our SF API connection to include all live candidate records.

The Sopact team and other stakeholders will need to decide what data they need to gather on SF and as part of this issue we'll adapt the contact records accordingly. Then we can set up a nightly sync.

[samschlicht]

Draft plan Will test this in TC Staging-SF Sandbox before making any changes to SF Prod.

• [x] SF sandbox steps below to prepare new candidate contact records
• [x] Amend existing candidate update/create method to populate the new fields*
• [x] Write a candidate sync (upsert) method and test from dev system admin API -> SF sandbox
• [x] Verify that existing candidate contact records are successfully updated and new ones created

Once that's all working:

• [x] Prepare SF prod by same steps as sandbox (except for removing redundant candidate-specific fields from the 'General' layout, which will have to wait till after release, so as to preserve current candidate contact records)
• [x] Refresh the SF sandbox and then test the sync method stub again — this will ensure that I've correctly mirrored the sandbox settings to SF prod (which means it'll work on TC prod, post-release); it'll also mirror the RecordTypeId, removing the need for an environment-dependent variable there

Once that's all set:

• [x] Refactor for DRYness
• [x] Find a way to schedule, avoiding existing issue of Spring @ Scheduled tasks firing twice in prod — (moving ahead as #450 and I'll revisit here when that's done)
• [x] Final SystemAdminApi check on finished code

*Irrespective of the nightly batch update, we still need live candidate-creating methods, e.g., when someone hits the 'update/create SF' button on a candidate, or when someone creates a new candidate opportunity.

[samschlicht]

SF STEPS Time for a bit of SF housekeeping. We are adding even more candidate-specific fields that don't need to be cluttering up other types of contact records.

So let's make a candidate contact page layout and candidate contact record type that uses it. (This replicates the setup we have for different types of opportunities and probably should've been done a while ago.)

SANDBOX

• [x] Create 'Candidate Layout' page layout
• [x] Add these new fields, and the ones that Caroline created for email consent, assigning them to Candidate Layout only; they should be read-only except for system admins (TC should be source of truth for candidates, so no editing needs to happen on SF - it'll only get overwritten anyway!)
• [x] In Setup > Object Manager > Contact > Record Types create: 'General' for unspecified record types, 'Candidate' for candidates (we can add more types as we need them)
• [x] Again in Record Types, set 'Page Layout Assignment' so that all contacts of type 'Candidate' get the Candidate Layout - the 'General' type should retain the master layout, which should remain the generic contact layout we were using before (only now it'll have the candidate-specific fields removed)
• [x] Hide existing candidate-specific fields and 'Job Opportunities' widget from General Contact Layout, so they're only included on Candidate Layout
• [x] Remove redundancies from Candidate Layout: Added to Global Mailing List, Contact Owner, Reports To, Department, all fields from Additional Information section (not the section itself), Opportunities (Migration Agent) widget, Phone, Mobile, Opportunities widget, Title — all can be re-added if we ever start to use them

PROD

• [x] Create 'Candidate Layout' page layout
• [x] Add these new fields, and the ones that Caroline created for email consent, assigning them to Candidate Layout only; they should be read-only except for system admins (TC should be source of truth for candidates, so no editing needs to happen on SF - it'll only get overwritten anyway!)
• [x] In Setup > Object Manager > Contact > Record Types create: 'General' for unspecified record types, 'Candidate' for candidates (we can add more types as we need them)
• [x] Again in Record Types, set 'Page Layout Assignment' so that all contacts of type 'Candidate' get the Candidate Layout - the 'General' type should retain the master layout, which should remain the generic contact layout we were using before (only now it'll have the candidate-specific fields removed)
• [x] Remove redundancies from Candidate Layout: Added to Global Mailing List, Contact Owner, Reports To, Department, all fields from Additional Information section (not the section itself), Opportunities (Migration Agent) widget, Phone, Mobile, Opportunities widget, Title — all can be re-added if we ever start to use them

👇❗️ At time of STAGING MERGE ❗️👇

• [x] Refresh SF sandbox, check that Record Type ID is the same on both and corresponds to the value in SalesforceServiceImpl for recordTypeId
• [x] Follow these steps to renew TC API access
• [x] Deploy staging from new JSON task definition including the environment variable

👇❗️ Immediately after TC release ❗️👇

• [ ] Hide existing candidate-specific fields and 'Job Opportunities' widget from General Contact Layout, so they're only included on Candidate Layout (this might be a good moment to do some general fine-tuning of contact records, creating some other layouts for our various types)

[samschlicht]

Here's how a Candidate Contact Record ends up looking to a regular SF user: Private Zenhub Image

[samschlicht]

John asked for a recap in the last meeting of what the current ways that the TC creates/updates candidates in SF.

There are 2:

• candidate profile > additional info > create/update Salesforce button Private Zenhub Image

• when a candidate is added to a list for an open job and they're not already on SF, the TC makes a call to create them; if they're already on there, the TC will do an update of any changed fields.

They both use the same create/update method.

[samschlicht]

After some consternation, just confirming that we have 168,000 API calls per 24-hour period to play with (100k for the org + 1k per licenced user), so no concerns about doing batches of 200 records so that we can use the regular SF REST API (which has a 200-record limit per call.

Private Zenhub Image Private Zenhub Image Private Zenhub Image

[samschlicht]

Q for the team: should I include 'incomplete' statuses or just 'active' and 'pending'? We seem to include them in total TC candidate count.

[samschlicht]

NB: because the TC stops syncing a candidate once they're employed, this means that TBB users can start to make lasting changes to the editable fields (name, account name, email, description, contact type) once someone becomes an alum — useful because the person will often become a contact of another sort, and also change contact details, job, etc.

[samschlicht]

Another useful tidbit: in SF you can make a record type (e.g. 'Candidate', which assigns the Candidate Page Layout) available only to certain users. All users will be able to see the record, but only those users you've selected will be able to create or clone a record of that type.

So, in this case, I've made the Candidate record type available only to system admins — because why should anyone be able to clone or create candidates when the TC is handling that? No biggie, but just another way to add assurance that the candidate data in SF is sound for reporting.

[samschlicht]

The only remaining thing to do here is to add the environment if variable so the sync only runs on prod - got to wait for my PR on that issue to be approved and merge from the new staging

[samschlicht]

Since this is still open, I'm working on adding Zeynep's request to include intake date (#481)!

Still to do:

• [x] fix candidate search table, which is affected by this change

• [x] test on large batch of candidates via SystemAdminApi

• [x] add to SF prod if all good

• [X] add environment variable for 'if' statement once it's available (as noted, we only want the sync firing from prod)

[samschlicht]

From Zeynep re 'incomplete' statuses (I suggest we keep them included but in future perhaps merge that status with 'pending': Honestly, I’ve never seen “incomplete” used in our work. On Tettra we have this description “in 2019 we employed an intern to go through the data and mark all candidates whose data was largely incomplete. Currently there are over 4,000 such candidates on the database. We were going to delete this status but that means throwing away some substantial work. We suggested retaining the status - but asked the field not to use it. Candidates with incomplete status are treated like “pending” candidates.” I’m also two minds about it. If the quality is not there why should we include it. At the same time, we have thousands of “pending” profiles that have almost no information that we count in our numbers, so I don’t know if we can really separate them. Bases on the last sentence of the description, shall we include them? I’m also curious to hear your thoughts on whether merging these status’ would be an option?

(moving this one ahead as #498)

[samschlicht]

• [x] Final action — need to update terraform.tfvars with the new environment variable, but this is not essential for merging to staging and is recorded in an Asana issue for my and Sadat's attention