search

OpenFn / lwala

2 stars 3 forks source link

Bulkify Salesforce job to upsert `Person Visit` records #91

Closed jayesse closed 1 year ago

jayesse commented 1 year ago

Background

Lwala's current CommCare-to-Salesforce workflows run on a real-time basis, with jobs that have message-filter triggers. To reduce its total number of OpenFn runs and improve system performance, it now wants to change its existing workflows to run on a cron, scheduled basis and to process records in bulk (rather than individual case updates).

Request

There is an existing job Upsert Person Visit that is configured to process only 1 case record. It has 2 key operations:

  1. Query Salesforce Location__c table to search for existing location Ids to use in the person visits mapping (for step 2)
  2. Upsert Person_visit__c

Please bulkify this job and the above three operations to (1) handle an input of an array of cases, and (2) to leverage the Salesforce bulk API.

Workflow diagram

See the workflow steps in the red box for the operations https://lucid.app/lucidchart/e3411bda-1f0e-492f-b35d-6baf2dd3972a/edit?invitationId=inv_218cd740-0c18-446b-b225-b327e581f1b8&page=jWqM8lMzZmt4#

FYI - This is a really old job originally written by the client, so no mapping spec is available.

expression

⚠️ Please start work on the bulk-testing branch, and feel free to commit changes to this branch. ⚠️

Existing job to bulkify: https://github.com/OpenFn/lwala/blob/bulk-testing/commcare-salesforce-jobs/test_jobs/TEST_upsert_person_and_person_visit.js

The field-level mappings should be the same as the existing job.

state

output

There are 2 items in the objects array in sample state.data. This should result in the same number of Person Visit records to upsert.

These Person Visit records will be child to the related Person records created via issue #90 .

toggl

Use the Lwala Bulk Workflows [2023] - 40 hrs toggl project.

aleksa-krolls commented 1 year ago

@mtuchi this one is also ready to be worked on, once the Person issue #90 is complete.
cc @jayesse

aleksa-krolls commented 1 year ago

Tested locally, fixed any comments left regarding mapping issues, and it's now working well. Next steps will be even greater bulk testing.