"CRUDE" examples -- building up the Flat Navigation Dispatch tools

[thorwhalen]

The task is to write streamlit apps of growing complexity to build up (and later demo) the CRUDE GUI tools (aka "Flat Navigation Dispatch"). (CRUDE is for CRUD + Execution).

The following are (or are planned to be) in examples/crude/:

• take_01_salary.py: A simple app to demo the use of Mappings to handle complex type
• take_02_salary.py: Like take01, but using i2.wrapper to transform the function
• take_03_salary.py: Like take02, but where salary key entry is done via a selectbox
• take_04_model_run.py: An example of how to transform (manually) a function that runs some data through a model (both too complex to enter directly and explicitly in forms) into a streamlit dispatchable function that uses a store (a Mapping) to manage complex data.
• take_05_model_run.py: Same as take_04_model_run, but where the dispatch is not as manual.

name	func(s)	dispatch_prep	store_type	store_contents	key_entry		comments
take_01_salary	multiplication	manual	dict	static	freeform
take_02_salary	multiplication	semi-auto	dict	static	freeform
take_03_salary	multiplication	semi-auto	dict	static	selectbox
take_04_model_run	apply_model	manual	dict	static	freeform
take_05_model_run	apply_model	semi-auto+	dict	static	freeform

Flat Navigation Dispatch front issue

Copied from front issue#12.

The design pattern I'm proposing is that when you need to dispatch a function like

train(learner, data)

where learner and data are complex python objects, you do it by creating three components:

• CRUD for learners
• CRUD for data
• train component, which points to the learner and data CRUD components to get its inputs

The CRUD components are made of two parts

• For the creation of the object: A specific UI component
• For all other CRUD operations: A (py2)store that can handle the type of objects being made

Sometimes the creation will itself require some complex object inputs. For example, the case of model in: run(model, data) But a model is just the output of train(learner, data), so we fall back on the train component, which in turn relies on the learner and data components.

The GUI navigation that the use of this pattern enables is "flat" in the sense that the process structure is less of a tree or DAG, and more of a "choose your action, and if the inputs you require don't exist, go choose the action that will make these inputs available".

This flatness may or may not be desirable from a UX point of view, but is certainly desirable from an architectural simplicity point of view, and we can get a great bang for the buck as far as component reuse.

Any code that appears in this form:

data = get_data(data_key)
prepped_data_1 = prep1(data, another_param)
prepped_data_2 = prep2(data, yet_another, and_another)
result = do_something(prepped_data_1, prepped_data_2, configs)
...

can be dispatched to such a "flat GUI" dispatching get_data, prep1, prep2 and do_something functions, after wrapping these in a "dispatchable" form that uses stores to both solve the complex data entry problem and the action/navigation order problem. The store keys play the role of intermediate variables in the python code above.

Code example

A function such as:

from typing import Any

FVs = Any
FittedModel = Any

# to dispatch:
def apply_model(fvs: FVs, fitted_model: FittedModel, method='transform'):
    method_func = getattr(fitted_model, method)
    return method_func(list(fvs))

would have a dispatchable form (automatically created by wrappers) as such:

# This is what we want our "dispatchable" wrapper to look like

# There should be real physical stores for those types (FVs, FittedModel) that need them
fvs_store = dict(
    train_fvs_1=[[1], [2], [3], [5], [4], [2], [1], [4], [3]],
    train_fvs_2=[[1], [10], [5], [3], [4]],
    test_fvs=[[1], [5], [3], [10], [-5]],
)

fitted_model_store = dict(
    fitted_model_1=MinMaxScaler().fit(fvs_store['train_fvs_1']),
    fitted_model_2=MinMaxScaler().fit(fvs_store['train_fvs_2']),
)

# And a store for the result of the function if we want this result to be reused later
# (for example for further inputs)
apply_model_store = dict()

from typing import Any, Mapping, Tuple

FVsKey = str  # really, should be a str from a list of options, given by list(fvs_store)
FittedModelKey = str  # really, should be a str from a list of options, given by list(fitted_model_store)
Result = Any

# dispatchable function:
def apply_model_using_stores(
    fvs: FVsKey,
    fitted_model: FittedModelKey,
    method: str = 'transform',
    fvs_store: Mapping[FVsKey, FVs] = fvs_store,
    fitted_model_store: Mapping[FittedModelKey, FittedModel] = fitted_model_store,
    apply_model_store: Mapping[Tuple[FVsKey, FittedModelKey], Result] = apply_model_store
):
    # get the inputs
    fvs_value = fvs_store[fvs]
    fitted_model_value = fitted_model_store[fitted_model]
    # compute the function
    result = apply_model(fvs_value, fitted_model_value)
    # store the outputs
    apply_model_store[fvs, fitted_model] = result
    return result  # or not

[thorwhalen]

take_01_salary.py: https://github.com/i2mint/streamlitfront/blob/f09c310a3d27e6c255c661b77cfd74d33e927828/streamlitfront/examples/crude/take_01_salary.py#L6-L5

[thorwhalen]

https://github.com/i2mint/streamlitfront/blob/cea67c6d1a40839de36a48193497677424e30091/streamlitfront/examples/crude/take_04_model_run.py#L36

[thorwhalen]

take_06_model_run

https://github.com/i2mint/crude/blob/2827210c2ad05a5c371c041bc54338408bf07512/crude/extrude_crude/take_06_model_run.py#L93

We now have tools for persisting stores to be used in crude. See that in the demo, when you run a model (and specify a name), it will save the results in a persisting store (in dill files placed in a temp folder of the filesystem)