Allow for multiple search strings to be used

[taylordowns2000]

@aleksa-krolls , multiple search strings can be provided (see below—filters and filters.some(...)) but it's unclear how we should then modify the payload from Kobo. Right now, we "tag" the form based on the filter:

get('https://kf.kobotoolbox.org/api/v2/assets/?format=json', {}, state => {
  console.log(`Previous cursor: ${state.lastEnd}`);
  // Set a manual cursor if you'd like to only fetch data after a certain date
  const manualCursor = '2020-05-25T14:32:43.325+01:00';
  const filters = ['Rural Consumption', 'Préférences consommation'];
  state.data.forms = state.data.results
    .filter(resource =>
      filters.some(f => resource.name.toLowerCase().includes(f.toLowerCase()))
    )
    .map(form => {
      console.log(form.name);
      const url = form.url.split('?').join('data/?');
      return {
        formId: form.uid,
        tag: filters, // how should we tag the form?
        url,
        query: `&query={"end":{"$gte":"${manualCursor}"}}`,
        //query: `&query={"end":{"$gte":"${state.lastEnd || manualCursor}"}}`,
      };
    });
  console.log(`Forms to fetch: ${JSON.stringify(state.data.forms, null, 2)}`);
  return { ...state, filters };
});

I'm not a fan of the solution because "search" implies that they want to find anything that matches X, Y or Z. What happens when it matches both X and Y? How should we then modify the data? In other words, you never see an interface where you Search and Replace with multiple search strings. You might find and replace, but that finding a specific string and replacing it with another.

Do you have a sense of how they want to proceed? I will push back hard if they say that they want to tag a form with the "first" or "last" string that matches from the filters. That level of unpredictability is just no good.

Instead, I might push for them to tag forms manually somehow when they BUILD them. They should be specifying, at the level of the form or in some other rigid way how we process these submissions. I don't see a way around that. If they wanted, they could keep a big list of all the forms and how they'd like OpenFn to process them in OpenFn. That would be fine too, but there's still this big manual piece of work which is effectively telling OpenFn how to process each form.

[aleksa-krolls]

@taylordowns2000 we're well aware of the tagging recommendation, but it's not easy to implement across forms where data collection has already started. We therefore need to have a manual tagging step.

Can you update this so that they can search by multiple filters (e.g., Rural consumption, Préférences consommation)... but that all forms will be assigned 1 default tag Rural Consumption to trigger job 02: https://www.openfn.org/projects/1767/jobs/3603

If they want to provide specific tags for each form, we'll borrow the 00 job from the automation solution where the admin user can specify the exact form Id, tag, etc.

[taylordowns2000]

Ah! Yes. That's super easy and perfectly fine from a data integrity perspective. I had imagined that different forms would need different tags. Implementing now.

On Tue, Nov 24, 2020 at 7:58 AM Aleksa Krolls notifications@github.com wrote:

@taylordowns2000 https://github.com/taylordowns2000 we're well aware of the tagging recommendation, but it's not easy to implement across forms where data collection has already started. We therefore need to have a manual tagging step. Can you update this so that they can search by multiple filters (e.g., Rural consumption, Préférences consommation)... but that all forms will be assigned 1 default tag Rural Consumption to trigger job 02: https://www.openfn.org/projects/1767/jobs/3603

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/OpenFn/wcs-wildmeat/issues/20#issuecomment-732723853, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACCUBLP3PE353AKLTIGCKE3SRNRTBANCNFSM4UAPTN2A .