pyinvoke / invoke

Pythonic task management & command execution.
http://pyinvoke.org
BSD 2-Clause "Simplified" License
4.32k stars 365 forks source link

optional flags not being forwarded to dependent tasks #852

Closed gvwilson closed 2 years ago

gvwilson commented 2 years ago

I have two tasks: one cleans a raw data file to produce a clean data file, and the other produces several plots from the clean data file:

RAW_FILE = "raw.csv"
CLEAN_FILE = "clean.csv"
PLOT_FILE = "plot.svg"

@task(optional=["logging"])
def clean_data(c, logging=None):
    """Produce cleaned-up dataset."""
    print("CLEAN", logging)
    _configure_logging(logging)
    df = clean_raw_data(RAW_FILE)
    df.to_csv(CLEAN_FILE, index=False)

@task(pre=[clean_data], optional=["logging"])
def plot_data(c, logging=None):
    """Create plots of data."""
    print("PLOT", logging)
    _configure_logging(logging)
    make_plot(CLEAN_FILE, PLOT_FILE)

def _configure_logging(log_level):
    """Initialize logging."""
    if log_level is not None:
        print("SETTING LOGGING TO", log_level)
        CONFIG["LOGGING_LEVEL"] = log_level.upper()

If I run:

$ invoke clean-data --logging info

then logging is set to INFO and I get a message from inside clean_raw_data. However, if I run:

$ invoke plot-data --logging info

then:

  1. clean_data is invoked with logging=None, so no log message appears.
  2. plot_data is then invoked with logging="info", so its log message appears.

My expectation was that command-line flags would be passed down to dependent tasks. I tried doing this manually:

@task(pre=[call(clean_data, logging=logging)], optional=["logging"])
def plot_data(c, logging=None):
    ...as before...

but this produces an error message because logging isn't defined at the point the @task decorator is invoked.

Is there a way to chain optional arguments in the desired fashioned?

neozenith commented 2 years ago

This is not yet supported but it is on the roadmap. https://bitprophet.org/projects/#roadmap

If you'd like to track the progress of that specific feature, we are linking back to this issue I think is the best canonical issue to describe it: https://github.com/pyinvoke/invoke/issues/461

But the timeline to work on this will be after all of Fabric, Paramiko and Invoke all migrate to CircleCI and drop support for Python2. Then we stand a better chance at feature work with a smaller support matrix.

So there is a lot of catch up maintenance work, but this is a popular request we intend to implement.