Update debit workflow to accommodate V2 migration

[daissatou2]

Background, context, and business value

Migrate the debit card workflow to V2

The specific request, in as few words as possible

Debit workflow jobs on V1:

1. E1. Get CSVs from SFTP
2. upsert direct debits
3. upsertTransactionDDV2
4. upsertCustomDDV2

Workflow on V2: https://app.openfn.org/projects/d84ec7c9-7b5a-4f01-a0d2-88f359deb0e2/w/46758111-9ef9-483b-be63-386b51c11107?placeholder=true

You'll notice that the downstream jobs are not message filters but now flow jobs. Check out the suggested job changes below to make this work.

Suggested job changes:

1. In V1, the E1. Get CSVs from SFTP] job posts messages to the inbox to trigger the other three jobs. Please update this job so that the output of the jobs contains what is in the JS path conditions on V2. Also make sure this job only fetches the debit files ("wfwi Direct Debits", "wfwi Transactions - DD", and "wfwi Custom DD Fields").
2. We have added three new steps: "Chunk direct debit data", "Chunk transactions debit data" and "Chunk custom debit data". These jobs should chunk each file to account for data volume limits as done in V1.

state.json

E1. Get CSVs from SFTP

{
  "configuration": { "See LP WFWI SFTP" }
}

Email Alert Jobs

{
  "configuration": { "See LP WfWI Mailgun Prod" }
}

upsertDonorsV2

{
  "configuration": ["SEE LAST PASS: 'Admin user - WFW Salesforce Permissions Sandbox (Q2 2024)'"],
}

adaptor

SFTP Mailgun Salesforce

expression.js

https://github.com/OpenFn/women-for-women/blob/staging/jobs/committed/1.fetchJSON.js https://github.com/OpenFn/women-for-women/blob/master/jobs/committed/upsertDirectDebits.js https://github.com/OpenFn/women-for-women/blob/master/jobs/committed/upsertTransactionDDV2.js https://github.com/OpenFn/women-for-women/blob/master/jobs/committed/upsertCustomDDV2.js

The links above are to the V1 files. DO NOT MAKE ANY CHANGES TO THOSE FILES. Create a new V2 directory in the same repo to hold the new V2 files.

Toggl

Lightning Migration

[mtuchi]

@daissatou2 i have update both credit and debit workflow on app.openfn.org.

The credit workflow throwing an error at upsertTransactionCardV2 step and debit is throwing an error at upsert direct debits step. I will pair with @aleksa-krolls in the AM to resolve these issues

If you want to see file changes see PR #156

[mtuchi]

I have created the ability to test manual DDids. Simply include manualDDids in your initial input. You can create this from the first step of the workflow (Probably the easiest). But you can also create this in down stream steps, just make sure you also have the upstream final output in your new input.

See example of manualDDids From first step

{
    "manualDDids": [
        "711682",
        "711684",
        "700292"
    ]
}

From mapDirecrDebits step

...Final output of getCsvFiles,
{
    "manualDDids": [
        "711682",
        "711684",
        "700292"
    ]
}

cc @aleksa-krolls

[mtuchi]

The donors, debit & credit workflow in app.openfn.org have the latest changes in this PR #156

[aleksa-krolls]

@mtuchi we can move this debit workflow issue to Review, yeah? I think Aicha is unblocked to test... but want to make sure there is no work left for you to wrap up on this one? We were able to successfully sync all donors from the May 1 file 20240501 last week ... so maybe that's the last step you needed before we can test this workflow?

[mtuchi]

Facing couple of errors testing the debit workflow, See slack thread here and see error logs here cc @aleksa-krolls

[aleksa-krolls]

@mtuchi I fixed the errors with the debit workflow and then re-ran. I'm now encountering this error - see run. Can you pls check it out?

TypeError: TypeError: Cannot read properties of undefined (reading 'includes')

[mtuchi]

@aleksa-krolls i think, this is related to upgrading salesforce from 2.7.3 to 4.6.5. The job code needs to be updated to be able to work with the latest version

[aleksa-krolls]

See issue #159 for the spec to upgrade the jobs to salesforce v4.6.5. Please complete this first across WFW jobs and then resume with re-testing the debit workflows.