Update Donors workflow to accommodate V2 migration

[daissatou2]

Background, context, and business value

Migrate the donors workflow to V2.

The specific request, in as few words as possible

Donors workflow jobs on V1:

1. E0 - Get Donor CSVs from SFTP
2. upsertDonorsV2
3. E4 - Email Alert for Dupe Donors
4. E3 - Alert Files Not Found

Workflow on V2: https://app.openfn.org/projects/d84ec7c9-7b5a-4f01-a0d2-88f359deb0e2/w/7a2aacd9-0f47-4e1f-b89d-e3ceca028d04?s=834893d5-78d4-48b8-850d-400fa787229c

You'll notice that the 3 downstream jobs are no longer message filters but now flow jobs. Check out the suggested job changes below to make this work.

Suggested job changes:

1. In V1, the E0 - Get Donor CSVs from SFTP job posts messages to the inbox to trigger the other three jobs. Please update this job so that the output of the jobs contains what is in the JS path conditions on V2.
2. REMOVE CLIENT EMAILS FROM the E4 - Email Alert for Dupe Donors & E3 - Alert Files Not Found jobs before testing. You can test by adding your email to the jobs instead.

state.json

E0 Get Donor CSVs from SFTP

{
  "configuration": { "See LP WFWI SFTP" }
}

Email Alert Jobs

{
  "configuration": { "See LP WfWI Mailgun Prod" }
}

upsertDonorsV2

{
  "configuration": ["SEE LAST PASS: 'Admin user - WFW Salesforce Permissions Sandbox (Q2 2024)'"],
}

adaptor

SFTP Mailgun Salesforce

expression.js

https://github.com/OpenFn/women-for-women/blob/master/jobs/committed/1.fetchJSON_Donors.js https://github.com/OpenFn/women-for-women/blob/master/jobs/committed/upsertDonorsV2.js https://github.com/OpenFn/women-for-women/blob/master/jobs/committed/alertFilesNotFound.js https://github.com/OpenFn/women-for-women/blob/master/jobs/committed/emailErrorAlert.js

The links above are to the V1 files. DO NOT MAKE ANY CHANGES TO THOSE FILES. Create a new V2 directory in the same repo to hold the new V2 files.

Toggl

Lightning Migration

[daissatou2]

@mtuchi quicks flags:

1. REMOVE CLIENT EMAILS FROM the E4 - Email Alert for Dupe Donors & E3 - Alert Files Not Found jobs before testing. You can test by adding your email to the jobs instead.
2. Do not make any changes to the V1 files linked above. Create a new V2 directory in the same repo to hold the new V2 files.
3. Please feel free to provide feedback on the suggested workflow we've set up on V2 given your knowledge of how these jobs work.

[daissatou2]

@mtuchi this job fetches CSV files manually: https://app.openfn.org/projects/d84ec7c9-7b5a-4f01-a0d2-88f359deb0e2/w/d43f64d1-0a6f-457f-b765-5ec6006f19c6?s=3abc0d0b-f92b-4cbc-a275-fbb41ec36ea0&m=expand

[mtuchi]

Hello @daissatou2 just a quick update I have update the workflow in v2, here is the detailed breakdown

• Both mailgun steps are downstream jobs and will use the final state of E0 Get Donor CSVs from SFTp
• The upsertDonorsv2 is a webhook triggered workflow just like in v1. The reason for this is because the upsertDonorsv2 job is already looping through each row in a csv file and has 5 nested salesforce queries. Making this job a down stream job will mean we have to add another each function on top which will make the job code more complex and introduce performance issues. I have tried to look for ways to improve the job code for upsertDonorsv2 but it's not an easy fix it will require a-lot of decoupling the logic and making it work again. So it's best for this job to remain a webhook trigger for now until we have more budget for a complete refactor

[daissatou2]

@mtuchi feedback below:

1. I tested with a manual cursor date: April 25th and the "upsert donors" step is failing with the following typeerror. This date passed on V1.
2. I tested with a cursor date in the future and the "Alert files not found" step is failing with a TypeError.

[mtuchi]

@daissatou2 both issues are fixed, Please try testing again

[aleksa-krolls]

@mtuchi as discussed, this workflow is not upserting all donors. If you run this workflow for the file wfwi Donors 20240501.csv, then I'd expect the following donors to be upserted in Salesforce with these Committed_Giving_ID__c values:

1. 010520151134100743WomenforWomen (row 612 in source csv)
2. 010520151212160877WomenforWomen (row 644 in source csv) Right now I do not see these donors loaded as Contacts in Salesforce.

[mtuchi]

I have created the records for the Committed_Giving_ID__c mentioned above to unblock debit workflow testing. But still running Donors workflow for 20240501 is timing out. I have tested running the workflow locally with a 2hrs timeout option and the workflow still failed.

[R/T] ✘ Error: Timeout expired (7200000ms)
[CLI] ♦ Workflow failed in 120m 3.4520000000002256s.

[aleksa-krolls]

@mtuchi I'm re-running these WFW workflows as I test SF Oauth and now this Donors run is failing with this error - can you pls check it out?

{"message":"TypeError: Cannot read properties of undefined (reading 'length')","name":"RuntimeError"}

[mtuchi]

@aleksa-krolls i think, this is related to upgrading salesforce from 2.7.3 to 4.6.5. The job code needs to be updated to be able to work with the latest version

[aleksa-krolls]

@mtuchi Let's discuss in standup, bc maybe we need add to the next sprint. We will need to use these later versions of the SF adaptor in order to use Oauth.