Allow admin to specify multiple filters when fetch Kobo forms

[aleksa-krolls]

Background

Currently this Wildmeat job (https://www.openfn.org/projects/1767/jobs/3604) runs on a scheduled basis to fetch all Kobo forms with Rural Consumption in the name. This is used to filter forms.

Request

Modify the job so that the admin can specify multiple filters...

1. Admins definitely want to fetch any forms with the name Rural Consumption (e.g., Rural consumption, rural consumption). Is there a way to make the filters case insensitive?
2. Later on, admins may want to add other keywords to search across Kobo form names (e.g., Market consumption).

Also allow the admin to set a default string value defaultUnit for the Kobo forms (e.g., defaultUnit: 'kilograms'). This defaultUnit should be added to state that is posted back to the OpenFn Inbox so that it can be mapped on L51 in this 2nd job rural-consumption-to-postgres.js that is triggered.

state

See LP for openfn_kobo. Run this job on prod to test: https://www.openfn.org/projects/1767/jobs/3604

We're connecting to the Wildmeat Transitional DB for testing.

adaptor

language-http

expression

https://github.com/OpenFn/wcs-wildmeat/blob/b5ff94a8214732b9c7a534b9eb2744b97730dcc8/jobs/fetch-rc-submissions.js#L10

output

This Rural consumption form should be returned: https://kf.kobotoolbox.org/#/forms/aXecHjmbATuF6iGFmvBLBX/summary

[aleksa-krolls]

@lakhassane @taylordowns2000 Please let @chaiwa-berian know when ready to test.

[aleksa-krolls]

When I remove the sample and ran this job to fetch all Kobo forms that matched the filter, the job timed out: https://openfn.org/projects/1767/runs/10591037

I added back in a sample so that we can resume testing. Can you try running locally to see why it's maybe taking so long to process? And do you have any recommendations for how to shorten the processing time? Let me know if easier to talk this one through after running this job locally: https://openfn.org/projects/1767/jobs/3604 @taylordowns2000

[taylordowns2000]

Line 67 is what does it. We've been throttling these jobs (1 submission per second) but could bring that number way down. If you have a rough sense of the total number of submissions, then you can determine the runtime. I might try 200ms as the throttle.

https://github.com/OpenFn/wcs-wildmeat/blob/master/jobs/fetch-rc-submissions.js#L67

[aleksa-krolls]

@taylordowns2000 Why did we choose to throttle in this job? Pretty sure we're not doing this in the other "Get Kobo Data" jobs... so wondering what's different about this use case?

[taylordowns2000]

I've taken a look at those other getKobo jobs and it seems you're right. The concept here was that the result set was going to be very big, and since it was happening asynchronously anyway we might as well spread it out a bit to reduce the spike in traffic to inboxes/the receipt processor. Honestly, it's not critical, more an optimization from the platform perspective.