DAGWorks-Inc / hamilton

Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.
https://hamilton.dagworks.io/en/latest/
BSD 3-Clause Clear License
1.71k stars 105 forks source link

Can't stack @inject and @parameterize decorators #160

Open j7zAhU opened 1 year ago

j7zAhU commented 1 year ago

As per the issue title. This throws an error:

from hamilton.function_modifiers import inject, source, parameterize

@inject(params=source('my_func__params'))
@parameterize(
    my_func_a={'date_range': source('my_func_a_date_range')},
    my_func_b={'date_range': source('my_func_b_date_range')}
)
def my_func(date_range: tuple[pd.Timestamp, pd.Timestamp], params: int) -> int:
    return 1

temp_module = ad_hoc_utils.create_temporary_module(
    my_func, module_name="my_module"
)

config = {
    'my_func__params': 1,
    'my_func_a__date_range': [pd.Timestamp('2022-01-01'), pd.Timestamp('2022-01-10')],
    'my_func_b__date_range': [pd.Timestamp('2023-01-01'), pd.Timestamp('2023-01-10')],
}
dr = driver.Driver(config, temp_module)
df = dr.execute(final_vars=['my_func_a'])

Library & System Information

Hamilton: 1.22.5 , Python 3.11.1

elijahbenizzy commented 1 year ago

OK, so no inherent reason why this should work -- its an implementation detail that it doesn't. That said, the implementation is a little complex, and the more "hamiltonian" way is one of the following:

  1. (cleanest IMO) -- actually make the source the name in the function (or if its a value make it the default value)
  2. (slightly repetitive) -- integrate it with the @parameterize

details below

The reason this doesn't work is because @inject is implemented as another parameterization with one value. This is not the cleanest way -- instead it should be able to inject a node into the subdag/replace a dependency within the subdag on something specific. I think this could pretty easily be done, yet it requires a small refactor. Will dig in at some point soon, but for now I think there are a few unblocking strategies here.

As discussed on slack, using group might be helpful as well.

149189 commented 11 months ago

I would like to work on this Bug! Can i get the path of the file for this code please

skrawcz commented 1 month ago

I would like to work on this Bug! Can i get the path of the file for this code please

Apologies @149189. We never got back to you here.

@elijahbenizzy this feels similar to https://github.com/DAGWorks-Inc/hamilton/issues/748 right?