Job run succeeds when Person record is not inserted - bug?

aleksa-krolls commented 3 years ago

Overview

The job MOH513-Enroll-Household-in-SF.js upserts Households and Persons in Salesforce when a CHW registers new households. However, the client has reported that sometimes Person records are missing from Salesforce even though they are present in CommCare.

Current Behavior

MOH513-Enroll-Household-in-SF.js job runs succeed and show green checks even when not all operations in the job are successfull. See run 6879421 as an example where OpenFn marks the run as successful, but the last Person (see L270 in run log) in the run log was not successfully upload to Salesforce (no result returned in the log).

Upserting Person__c with externalId CommCare_ID__c : { CommCare_ID__c: 'd215cbb0-531a-45a7-adf5-4361536e2207',
  CommCare_HH_Code__c: '8d05b920-cff4-4b6a-a155-aaa1c19614c5',
  RecordType: { Name: 'Youth' },
  Name: 'BXXXXXXXXXX',

Request

Please troubleshoot and make the appropriate changes so that:

OpenFn will not return a false "success" message when one of the job operations failed
Benny's Person__c record is inserted correctly

Resources

state.json

Test with Lwala Salesforce sandbox. See LP for creds. Use sample_data/state_personB.json to test the above scenario.

adaptor

Use language-salesforce and make use of this LP's FakeAdaptor for offline testing.

expression.js

Related to job MOH513-Enroll-Household-in-SF.js

Toggl

Log your time to Lwala Cleanup 2020

lakhassane commented 3 years ago

@aleksa-krolls There is an invalid value in field Disability_c.

I don't have access to the salesforce platform but my guess is there is an invalid value in a picklist for this field.

In the sample_data/state_personB.json Benny's Person__c is the only one with 2 values for that field.

aleksa-krolls commented 3 years ago

@lakhassane Nice work on the logs! To resolve this mapping issue, can you please add a transformation to the job that returns null if CommCare disability=='none'?

aleksa-krolls commented 3 years ago

@lakhassane I deployed the disability change, but what did you implement in your logs to troubleshoot this issue? Can you add enhanced logging to this job? I want to avoid a future scenario where this happens again (the run succeeds when actually one of the operations fails... if this happens, it should fail in OpenFn).

Ideally we should implement this enhanced logging across all jobs (which is why I was wondering if this should be a change to language-salesforce). But please share what you did and we can evaluate from there! cc @taylordowns2000

aleksa-krolls commented 3 years ago

@lakhassane if this will turn into another task re: the LP, feel free to close this issue and link to a new one.

lakhassane commented 3 years ago

@taylordowns2000 what's your call on this? Should we open one about refactoring language-salesforce so it catches error, with a callback error.

So far it only has callbacks success

taylordowns2000 commented 3 years ago

Please close this if it's delivered and open an issue on the salesforce adaptor to discuss over there! Will need to take a look, but is it that there's no "reject(...)", or that errors are somehow actually being swallowed? (Please copy this question over to the salesforce issue.)

On Thu, Sep 10, 2020 at 5:34 PM lakhassane notifications@github.com wrote:

@taylordowns2000 https://github.com/taylordowns2000 what's your call on this? Should we open one about refactoring language-salesforce so it catches error, with a callback error.

So far it only has callbacks success

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenFn/lwala/issues/23#issuecomment-690744517, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCUBLNKH7LJFCMS4GZIO2TSFFA75ANCNFSM4RCFPSKA .

-- Taylor Downs | Head of Product Open Function Group https://www.openfn.org/ | How Data Integration Makes ICT4D Work https://youtu.be/GVB9voGp6T4

aleksa-krolls commented 3 years ago

@lakhassane Can you please analyze these other jobs to see if this combine() --> alterState change needs to be applied as well?

MOH513-Enroll-Person-in-SF.js
MOH514-Update-Person-in-SF-Production.js

OpenFn / lwala