search

jordanbrown0 / reg3

REG.PRG version 3
Other
2 stars 0 forks source link

External import corrections #164

Open jordanbrown0 opened 3 years ago

jordanbrown0 commented 3 years ago

The way that I have historically imported external data is to write a program to do the import. In addition to the obvious field selection and rearrangement, I have had it match specific problematic cases and fix them, both so that it's not necessary to update the upstream data and for cases (e.g. badge name split) where heuristics might get the wrong answer.

jordanbrown0 commented 3 years ago

Could write custom importers that read an input file and emit a JSON-DBMS file.

jordanbrown0 commented 3 years ago

Some, but not all, of this could perhaps be done by using the corrections infrastructure, correcting values on import. But: corrections are client-side, and import is server-side.

jordanbrown0 commented 3 years ago

An import request could include an expression that examines the record and corrects it. That expression would of course be dynamically created based on tabular data supplied by the user.

It is tempting to say that it would just use the corrections from the "corrections" table, but that's not really the same kind of corrections - that's more standardizing the form, while this sort of correction is more like fixing errors in the input.

If it's an expression, it could use arbitrarily complex logic to decide whether to apply a particular correction. Of course, presenting that flexibility in a UI would be tough, but maybe a simple "if this field has this value, set this other field to this other value" mechanism would be powerful enough and understandable.

jordanbrown0 commented 3 years ago

The InputObject and InputMultiple mechanisms might stack to scary levels...

jordanbrown0 commented 8 months ago

Can also fix specific problematic cases with an "update" import with a patch list.