search

mimiro-io / datahub

The MIMIRO data hub is a semantic, entity graph database combined with data integration capabilities and a jobs engine for data transformation.
Apache License 2.0
14 stars 5 forks source link

simplify marking entities as deleted #263

Open ashmanto opened 1 year ago

ashmanto commented 1 year ago

usually when filtering out entities in transforms we would want to mark it as deleted if it exists in the sink dataset.

currently doing: (typescript)

if (type !== "foo") {
            return;
        }

will just igjore the entitiy that comes in, and not mark it as deleted if type changes.

we would need to do this everywhere: (typescript)

if (type !== "foo") {
            let hasBeenEmitted = dh.FindById(dh.GetId(newEntity), ["<sink-dataset>"]);
            if (hasBeenEmitted) {
                dh.SetDeleted(<entity>, true);
            } else {
                return;
            }
        }

simplifying this with a function would be appreciated.

rompetroll commented 1 year ago

To make this specific beaviour a built in for typescript only, it could be added to the datahub typescript lib, similar to how SQuery here is done https://github.com/mimiro-io/datahub-tslib/blob/master/datahub.ts#L33.

In general, it could also be a good option to make it easy to import local typescript libs. So that you can have your local "common" file with typescript helper functions, which then can be imported into all local transform scripts.