pldn / LDWizard

🧙 LDWizard: A generic framework for simplifying the creation of linked data. Supported by the PLDN community.
European Union Public License 1.2
13 stars 7 forks source link

Allow for bulk processing of transformations #151

Closed philipperenzen closed 11 months ago

philipperenzen commented 11 months ago

Currently in LDWizard, during the transformation step each columnRefinement is executed on a cell per row basis. This issue proposes the introduction of bulk processing, allowing entire columns to be transformed into refined values at once (or in batches).

An example column refinement with bulk enabled would look as follows:

// ... rest of columnrefinement
// batchProcess: boolean | number (e.g. false by default, true => process all, 30 => process 30 rows in bulk at a time)
batchProcess: 30
transformation: async (...searchTerm: string[]) => {
        const results: string[] = []
        for (const q of searchTerm) {
        const result = await fetch(q)
          results.push(await result.text())
        }
        return results;
      },
// ...

Acceptance criteria:

philipperenzen commented 11 months ago

Feature is now available in version 3.0.4, see CONFIGURING.md for examples.