8451 / labrea

A framework for declarative, functional dataset definitions.
MIT License
12 stars 0 forks source link

Enable Callbacks to change the result of dataset #15

Open richardhathaway-8451 opened 1 month ago

richardhathaway-8451 commented 1 month ago

Enable callbacks to change the result of a dataset.

This is particularly useful for adding extra logic to the result of ALL implementations of an abstractdataset, like in an interface.

@interface(dispatch=Option("MODE"))
class A:
  @abstractdataset(callbacks = my_extra_logic)
  def number():
    ...

@pipeline_step
def my_extra_logic(df):
  # do extra logic and return

Callbacks are different from Effects. Callbacks are fixed at the time the dataset is defined and cannot be altered, but they can change the result of the dataset. Effects are not fixed at the time the dataset is defined, so they can be altered later, but they cannot affect the result of the dataset and the chain of evaluation.

richardhathaway-8451 commented 1 month ago

Another issue that a callback would solve is when a user wants to by default tack on additional logic to the end of a dataset (but also have the option to turn off that additional logic).

Currently this is being achieved with overloads using .default, but it is a bit strange to have an overload where the default implementation is to always use the overloaded version.

@dataset(dispatch = Option("TEST", True))
def x():
  return 1

@x.overload(alias = True)
def x2(val = x.default):
  return val + 1