search

OpenFn / lwala

2 stars 3 forks source link

TypeError [Error]: Cannot read properties of undefined (reading 'filter') #101

Closed aleksa-krolls closed 1 year ago

aleksa-krolls commented 1 year ago

Background

See issue #91. This is related to the bulk workflow for Person Visits

Issue

See run failure for Job 2 with error TypeError [Error]: Cannot read properties of undefined (reading 'filter') https://www.openfn.org/projects/staging-lwala-chw-support/runs/0644f9a3-22d4-74b3-b487-02c0317ef54d

The workflow needs to handle scenarios where Job 1 gets NO records from Commcare.

expression

Please work on the bulk-testing branch!! https://github.com/OpenFn/lwala/blob/bulk-testing/bulk/getCases-PersonVisits.js

If this type of error handling needs to be added to ALL of the bulk upsert jobs, please make the changes on all related jobs: https://github.com/OpenFn/lwala/blob/bulk-testing/bulk/getCases-HH.js https://github.com/OpenFn/lwala/blob/bulk-testing/bulk/getCases-Person.js https://github.com/OpenFn/lwala/blob/bulk-testing/bulk/getCases-Referrals.js

state

Run Job 1 (https://github.com/OpenFn/lwala/blob/bulk-testing/bulk/getCases-PersonVisits.js) to generate state for Job 2...

Job 1: configuration: SEE LP OpenFn CommCareHQ (Lwala) - PROD data: {}

Job 2: configuration: see LP Lwala Salesforce Sandbox (Bulk 2023) data: use the final state from job 1

output

If Job 1 gets 0 records from CommCare, then no action should happen in Job 2.

aleksa-krolls commented 1 year ago

@mtuchi please work on this tomorrow AM, and @daissatou2 please test when he his done.

FYI this is to resolve the bug Hildah reported in row 7 here.

mtuchi commented 1 year ago

@aleksa-krolls on i have pushed a fix for this issue (See PR) but regarding this👇🏽 If Job 1 gets 0 records from CommCare, then no action should happen in Job 2.,

Taylor suggested that we throw an error saying We don't have the data we need. to stop the down stream from being executed. Is this pattern okay with you ?

The alternative is putting a lot of if statements in down stream jobs

cc @daissatou2

aleksa-krolls commented 1 year ago

@mtuchi I do NOT want to throw an error if 0 results are returned. We will run these jobs every 1 hour, so it's OK if no data hasn't been updated in the last hour... I do not want to alert the admin, because this is expected. (We only want to alert the admin if there are issues connecting with CommCare/Salesforce and/or upserting data in Salesforce.)

Therefore, please make changes to ensure the job succeeds, even if 0 records are returned from CommCare.

mtuchi commented 1 year ago

@mtuchi I do NOT want to throw an error if 0 results are returned. We will run these jobs every 1 hour, so it's OK if no data hasn't been updated in the last hour... I do not want to alert the admin, because this is expected. (We only want to alert the admin if there are issues connecting with CommCare/Salesforce and/or upserting data in Salesforce.)

Therefore, please make changes to ensure the job succeeds, even if 0 records are returned from CommCare.

Noted @aleksa-krolls i reverted the changes and updated commcare-salesforce-jobs/test_jobs/TEST_upsert_person_and_person_visit.js to execute operation only when we have data

Should i apply this pattern to the following jobs ?

commcare-salesforce-jobs/Referrals-Upsert-Service_V2.js
commcare-salesforce-jobs/upsert_household_and_household_visit.js
commcare-salesforce-jobs/Upsert-Person-Case-Update.js