Building-ML-Pipelines / building-machine-learning-pipelines

Code repository for the O'Reilly publication "Building Machine Learning Pipelines" by Hannes Hapke & Catherine Nelson
MIT License
585 stars 250 forks source link

interactive_pipeline.ipynb error when running transform step. #18

Closed jimwill3 closed 4 years ago

jimwill3 commented 4 years ago

using interactive_pipeline notebook on a mac catalina - all steps prior to transform ran successfully: transform = Transform( examples=example_gen.outputs['examples'], schema=schema_gen.outputs['schema'], module_file=transform_file) context.run(transform)

Raises: ----zipcode = tf.strings.regex_replace(zipcode, r"X{0,5}", "0") TypeError: Input 'input' of 'StaticRegexReplace' Op has type int64 that does not match expected type of string.

versions: Tensorflow Version: 2.2.0 TFX Version: 0.21.4 TFDV Version: 0.21.5 TFMA Version: 0.21.6

thanks,

hanneshapke commented 4 years ago

Hi @jimwill3,

We recently updated the dataset for the notebooks and I have the suspicion that you are using an older version of the dataset.

I have made the notebook backward compatible in order to ingest both file versions. Can you please pull the main branch and let me know if the transformer step is now working for you?

Please reopen this issue if the error persists on your side.

Thank you, Hannes

hanneshapke commented 4 years ago

Here are the changes: https://github.com/Building-ML-Pipelines/building-machine-learning-pipelines/commit/615b084837e8735eff08f787ba35e158d1461fef

jimwill3 commented 4 years ago

Thanks - that got me through transform and train. When i get to run evaluator at the next step I receive this error: Unresolved input channel Artifact(type_name: Model, uri: , id: 0) for input 'model' was passed in interactive mode. When running in interactive mode, upstream components must first be run with interactive_context.run(component) before their outputs can be used in downstream components.

All of the prior steps were run with interactive context so not sure what is going on - seems this is not related to the data issue but don't want to assume anything.

Went back and tried the cell - after a couple of fails, it then worked. i assume a timing issue but impossible to tell - i waited for the prior cell to finish before moving the evaluator.

hanneshapke commented 4 years ago

@jimwill3 This error happens from time to time when all cells are executed altogether. I noticed that the kernel crashes sometimes after the training and therefore the input channel can't be resolved.

Can you try to rerun each cell and execute them manually, one-by-one?

Does the issue persist?

jimwill3 commented 4 years ago

Update: this cleared up. looks like there was an old copy of eval2.csv in data folder - removing that removed this error.

I have been running cell by cell so it may have been a one off fluke. Subsequent execution of that cell has been fine. I do now see errors in the evaluator step that, to me, don't seem related, but will mention here and, if needed, open a new issue. Here is the output/error:

ValueError: no value provided for label: model_name=, output_name=, sub_key=None, StandardMetricInputs=StandardMetricInputs(label=None, prediction=array([0.04309091], dtype=float32), example_weight=None, features=None)

This may be caused by a configuration error (i.e. label, and/or prediction keys were not specified) or an error in the pipeline. [while running 'ExtractEvaluateAndWriteResults/ExtractAndEvaluate/EvaluateMetricsAndPlots/ComputeMetricsAndPlots()/ComputePerSlice/ComputeUnsampledMetrics/CombinePerSliceKey/WindowIntoDiscarding']

thanks