Currently in LDWizard, during the transformation step each columnRefinement is executed on a cell per row basis. This issue proposes the introduction of bulk processing, allowing entire columns to be transformed into refined values at once (or in batches).
An example column refinement with bulk enabled would look as follows:
// ... rest of columnrefinement
// batchProcess: boolean | number (e.g. false by default, true => process all, 30 => process 30 rows in bulk at a time)
batchProcess: 30
transformation: async (...searchTerm: string[]) => {
const results: string[] = []
for (const q of searchTerm) {
const result = await fetch(q)
results.push(await result.text())
}
return results;
},
// ...
Acceptance criteria:
previous single cell refinements are still supported
bulk processing can be enabled with an argument for a refinement, where the refinement takes in an array of the batch/entire column and returns an array of strings (or undefined)
Currently in LDWizard, during the transformation step each
columnRefinement
is executed on a cell per row basis. This issue proposes the introduction of bulk processing, allowing entire columns to be transformed into refined values at once (or in batches).An example column refinement with bulk enabled would look as follows:
Acceptance criteria: