search

zenml-io / zenml

ZenML 🙏: The bridge between ML and Ops. https://zenml.io.
https://zenml.io
Apache License 2.0
4.07k stars 437 forks source link

[BUG] ValueError: An error occured while trying to apply the transformation #92

Closed ucsky closed 3 years ago

ucsky commented 3 years ago

Expected behavior I try to run the basic example:

from zenml.datasources import CSVDatasource
from zenml.pipelines import TrainingPipeline
from zenml.steps.evaluator import TFMAEvaluator
from zenml.steps.split import RandomSplit
from zenml.steps.preprocesser import StandardPreprocesser
from zenml.steps.trainer import TFFeedForwardTrainer

training_pipeline = TrainingPipeline(name='Quickstart')

# Add a datasource. This will automatically track and version it.
ds = CSVDatasource(name='Pima Indians Diabetes Dataset',
                   path='gs://zenml_quickstart/diabetes.csv')
training_pipeline.add_datasource(ds)

# Add a random 70/30 train-eval split
training_pipeline.add_split(RandomSplit(split_map={'train': 0.7, 
                                                   'eval': 0.2,
                                                   'test': 0.1}))

# StandardPreprocesser() has sane defaults for normal preprocessing methods
training_pipeline.add_preprocesser(
    StandardPreprocesser(
        features=['times_pregnant', 'pgc', 'dbp', 'tst', 
                  'insulin', 'bmi', 'pedigree', 'age'],
        labels=['has_diabetes'],
        overwrite={'has_diabetes': {
            'transform': [{'method': 'no_transform', 'parameters': {}}]}}
    ))

# Add a trainer
training_pipeline.add_trainer(TFFeedForwardTrainer(
    loss='binary_crossentropy',
    last_activation='sigmoid',
    output_units=1,
    metrics=['accuracy'],
    epochs=20))

# Add an evaluator
training_pipeline.add_evaluator(
    TFMAEvaluator(slices=[['has_diabetes']],
                  metrics={'has_diabetes': ['binary_crossentropy',
                                            'binary_accuracy']}))

# Run the pipeline locally
training_pipeline.run()

But I got the following error.

Stack Trace

2021-07-12 12:19:13,646 — zenml.pipelines.training_pipeline — INFO — Datasource Pima Indians Diabetes Dataset has no commits. Creating the first one..
2021-07-12 12:19:13,648 — zenml.pipelines.base_pipeline — INFO — Pipeline 1626085153648 created.

2021-07-12 12:19:13,724 — apache_beam.options.pipeline_options — WARNING — Discarding unparseable args: ['-f', '/home/gs/.local/share/jupyter/runtime/kernel-20bd2f0f-6d09-43d9-9eec-d3553c030468.json']

2021-07-12 12:19:15,886 — zenml.datasources.csv_datasource — INFO — Matched 1: ['gs://zenml_quickstart/diabetes.csv']
2021-07-12 12:19:15,892 — zenml.datasources.csv_datasource — INFO — Using header from file: gs://zenml_quickstart/diabetes.csv.
2021-07-12 12:19:16,070 — zenml.datasources.csv_datasource — INFO — Header: ['times_pregnant', 'pgc', 'dbp', 'tst', 'insulin', 'bmi', 'pedigree', 'age', 'has_diabetes'].

2021-07-12 12:19:16,430 — apache_beam.runners.interactive.interactive_environment — WARNING — Dependencies required for Interactive Beam PCollection visualization are not available, please use: `pip install apache-beam[interactive]` to install necessary dependencies to enable all data visualization features.

2021-07-12 12:19:16,600 — apache_beam.options.pipeline_options — WARNING — Discarding unparseable args: ['-f', '/home/gs/.local/share/jupyter/runtime/kernel-20bd2f0f-6d09-43d9-9eec-d3553c030468.json']
2021-07-12 12:19:17,496 — apache_beam.io.tfrecordio — WARNING — Couldn't find python-snappy so the implementation of _TFRecordUtil._masked_crc32c is not as fast as it could be.
2021-07-12 12:19:18,762 — apache_beam.options.pipeline_options — WARNING — Discarding unparseable args: ['-f', '/home/gs/.local/share/jupyter/runtime/kernel-20bd2f0f-6d09-43d9-9eec-d3553c030468.json']
2021-07-12 12:19:19,850 — apache_beam.options.pipeline_options — WARNING — Discarding unparseable args: ['-f', '/home/gs/.local/share/jupyter/runtime/kernel-20bd2f0f-6d09-43d9-9eec-d3553c030468.json']
2021-07-12 12:19:20,900 — apache_beam.options.pipeline_options — WARNING — Discarding unparseable args: ['-f', '/home/gs/.local/share/jupyter/runtime/kernel-20bd2f0f-6d09-43d9-9eec-d3553c030468.json']
2021-07-12 12:19:24,609 — apache_beam.options.pipeline_options — WARNING — Discarding unparseable args: ['-f', '/home/gs/.local/share/jupyter/runtime/kernel-20bd2f0f-6d09-43d9-9eec-d3553c030468.json']
2021-07-12 12:19:26,134 — tensorflow — WARNING — From /home/gs/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow_transform/tf_utils.py:266: Tensor.experimental_ref (from tensorflow.python.framework.ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use ref() instead.
2021-07-12 12:19:26,600 — root — WARNING — This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Tuple[Dict[str, Union[NoneType, _Dataset]], Union[Dict[str, Dict[str, PCollection]], NoneType]] instead.
2021-07-12 12:19:27,134 — root — WARNING — This output type hint will be ignored and not used for type-checking purposes. Typically, output type hints for a PTransform are single (or nested) types wrapped by a PCollection, PDone, or None. Got: Tuple[Dict[str, Union[NoneType, _Dataset]], Union[Dict[str, Dict[str, PCollection]], NoneType]] instead.
2021-07-12 12:19:27,168 — tensorflow — WARNING — Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 
2021-07-12 12:19:29,328 — tensorflow — WARNING — Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 
2021-07-12 12:19:29,393 — tensorflow — WARNING — Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 
2021-07-12 12:19:29,458 — tensorflow — WARNING — Tensorflow version (2.4.1) found. Note that Tensorflow Transform support for TF 2.0 is currently in beta, and features such as tf.function may not work as intended. 
2021-07-12 12:19:29,495 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,495 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>
2021-07-12 12:19:29,496 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,496 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>
2021-07-12 12:19:29,497 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,497 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>
2021-07-12 12:19:29,539 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,540 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>
2021-07-12 12:19:29,540 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,541 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>
2021-07-12 12:19:29,541 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,542 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>
2021-07-12 12:19:29,584 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,585 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>
2021-07-12 12:19:29,585 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,586 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>
2021-07-12 12:19:29,587 — apache_beam.typehints.typehints — WARNING — Ignoring send_type hint: <class 'NoneType'>
2021-07-12 12:19:29,588 — apache_beam.typehints.typehints — WARNING — Ignoring return_type hint: <class 'NoneType'>

---------------------------------------------------------------------------
NotFoundError                             Traceback (most recent call last)
~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow_transform/beam/impl.py in _handle_batch(self, batch)
    380       else:
--> 381         result = self._graph_state.callable_get_outputs(feed_dict)
    382         assert len(self._graph_state.outputs_tensor_keys) == len(result)

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow_transform/saved/saved_transform_io_v2.py in apply_transform_model(self, logical_input_map)
    363     elif self._is_finalized:
--> 364       return self._apply_v2_transform_model_finalized(logical_input_map)
    365     else:

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow_transform/saved/saved_transform_io_v2.py in _apply_v2_transform_model_finalized(self, logical_input_map)
    288     modified_inputs = self._format_input_map_as_tensors(logical_input_map)
--> 289     return self._wrapped_function_finalized(modified_inputs)
    290 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py in __call__(self, *args, **kwargs)
   1668     """
-> 1669     return self._call_impl(args, kwargs)
   1670 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py in _call_impl(self, args, kwargs, cancellation_manager)
   1678           return self._call_with_structured_signature(args, kwargs,
-> 1679                                                       cancellation_manager)
   1680         except TypeError as structured_err:

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py in _call_with_structured_signature(self, args, kwargs, cancellation_manager)
   1761         captured_inputs=self.captured_inputs,
-> 1762         cancellation_manager=cancellation_manager)
   1763 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py in _call_flat(self, args, captured_inputs, cancellation_manager)
   1918       return self._build_call_outputs(self._inference_function.call(
-> 1919           ctx, args, cancellation_manager=cancellation_manager))
   1920     forward_backward = self._select_forward_and_backward_functions(

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow/python/eager/function.py in call(self, ctx, args, cancellation_manager)
    559               attrs=attrs,
--> 560               ctx=ctx)
    561         else:

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow/python/eager/execute.py in quick_execute(op_name, num_outputs, inputs, attrs, ctx, name)
     59     tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
---> 60                                         inputs, attrs, num_outputs)
     61   except core._NotOkStatusException as e:

NotFoundError:  No registered 'Min' OpKernel for 'GPU' devices compatible with node {{node StatefulPartitionedCall/max_6/min_and_max/Max_1}}
     (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, Tidx=DT_INT32, _XlaHasReferenceVars=false, keep_dims=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"
    .  Registered:  device='XLA_CPU_JIT'; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_INT64, DT_QINT8, DT_QUINT8, DT_QINT32, DT_BFLOAT16, DT_UINT16, DT_HALF, DT_UINT32, DT_UINT64]
  device='XLA_GPU_JIT'; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_INT64, DT_QINT8, DT_QUINT8, DT_QINT32, DT_BFLOAT16, DT_UINT16, DT_HALF, DT_UINT32, DT_UINT64]
  device='GPU'; T in [DT_INT32]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_INT32]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_DOUBLE]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_DOUBLE]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_FLOAT]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_FLOAT]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_HALF]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_HALF]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_DOUBLE]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_DOUBLE]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_FLOAT]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_BFLOAT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_HALF]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_HALF]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT32]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT32]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT8]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT8]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT8]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT8]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT32]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT32]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT64]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT64]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT64]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT64]; Tidx in [DT_INT32]

     [[StatefulPartitionedCall/max_6/min_and_max/Max_1]] [Op:__inference_wrapped_finalized_5475]

Function call stack:
wrapped_finalized

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow_transform/beam/impl.py in process(self, batch, saved_model_dir)
    440 
--> 441     yield self._handle_batch(batch)
    442 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow_transform/beam/impl.py in _handle_batch(self, batch)
    387           Fetching the values for the following Tensor keys: {}.""".format(
--> 388               str(e), batch, self._graph_state.outputs_tensor_keys))
    389 

ValueError: An error occured while trying to apply the transformation: " No registered 'Min' OpKernel for 'GPU' devices compatible with node {{node StatefulPartitionedCall/max_6/min_and_max/Max_1}}
     (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, Tidx=DT_INT32, _XlaHasReferenceVars=false, keep_dims=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"
    .  Registered:  device='XLA_CPU_JIT'; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_INT64, DT_QINT8, DT_QUINT8, DT_QINT32, DT_BFLOAT16, DT_UINT16, DT_HALF, DT_UINT32, DT_UINT64]
  device='XLA_GPU_JIT'; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_INT64, DT_QINT8, DT_QUINT8, DT_QINT32, DT_BFLOAT16, DT_UINT16, DT_HALF, DT_UINT32, DT_UINT64]
  device='GPU'; T in [DT_INT32]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_INT32]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_DOUBLE]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_DOUBLE]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_FLOAT]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_FLOAT]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_HALF]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_HALF]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_DOUBLE]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_DOUBLE]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_FLOAT]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_BFLOAT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_HALF]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_HALF]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT32]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT32]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT8]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT8]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT8]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT8]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT32]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT32]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT64]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT64]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT64]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT64]; Tidx in [DT_INT32]

     [[StatefulPartitionedCall/max_6/min_and_max/Max_1]] [Op:__inference_wrapped_finalized_5475]

Function call stack:
wrapped_finalized
".
          Batch instances: pyarrow.RecordBatch
age: large_list<item: int64>
  child 0, item: int64
bmi: large_list<item: float>
  child 0, item: float
dbp: large_list<item: int64>
  child 0, item: int64
has_diabetes: large_list<item: int64>
  child 0, item: int64
insulin: large_list<item: int64>
  child 0, item: int64
pedigree: large_list<item: float>
  child 0, item: float
pgc: large_list<item: int64>
  child 0, item: int64
times_pregnant: large_list<item: int64>
  child 0, item: int64
tst: large_list<item: int64>
  child 0, item: int64,
          Fetching the values for the following Tensor keys: {'max_2/min_and_max/Identity_1', 'max_6/min_and_max/Identity_1', 'max_4/min_and_max/Identity', 'max/min_and_max/Identity_1', 'max_6/min_and_max/Identity', 'max_4/min_and_max/Identity_1', 'max_1/min_and_max/Identity', 'max_7/min_and_max/Identity_1', 'max_8/min_and_max/Identity_1', 'max_3/min_and_max/Identity', 'max_1/min_and_max/Identity_1', 'max_3/min_and_max/Identity_1', 'max/min_and_max/Identity', 'max_5/min_and_max/Identity', 'max_2/min_and_max/Identity', 'max_7/min_and_max/Identity', 'max_5/min_and_max/Identity_1', 'max_8/min_and_max/Identity'}.

During handling of the above exception, another exception occurred:

ValueError                                Traceback (most recent call last)
<ipython-input-16-43278ec7954a> in <module>
      1 # Run the pipeline locally
----> 2 training_pipeline.run()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/zenml/utils/analytics_utils.py in inner_func(*args, **kwargs)
    175     def inner_func(*args, **kwargs):
    176         track_event(event, metadata=metadata)
--> 177         result = func(*args, **kwargs)
    178         return result
    179 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/zenml/pipelines/base_pipeline.py in run(self, backend, metadata_store, artifact_store)
    455             self.register_pipeline(config)
    456 
--> 457         self.run_config(config)
    458 
    459         # After running, pipeline is immutable

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/zenml/pipelines/base_pipeline.py in run_config(self, config)
    376         """
    377         assert issubclass(self.backend.__class__, OrchestratorBaseBackend)
--> 378         self.backend.run(config)
    379 
    380     @track(event=RUN_PIPELINE)

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/zenml/backends/orchestrator/base/orchestrator_base_backend.py in run(self, config)
    107         """
    108         tfx_pipeline = self.get_tfx_pipeline(config)
--> 109         ZenMLLocalDagRunner().run(tfx_pipeline)

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/zenml/backends/orchestrator/base/zenml_local_orchestrator.py in run(self, pipeline)
     95                     custom_driver_spec=custom_driver_spec)
     96                 logging.info('Component %s is running.', node_id)
---> 97                 component_launcher.launch()
     98                 logging.info('Component %s is finished.', node_id)

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tfx/orchestration/portable/launcher.py in launch(self)
    429     if is_execution_needed:
    430       try:
--> 431         executor_output = self._run_executor(execution_info)
    432       except Exception as e:  # pylint: disable=broad-except
    433         execution_output = (

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tfx/orchestration/portable/launcher.py in _run_executor(self, execution_info)
    323     outputs_utils.make_output_dirs(execution_info.output_dict)
    324     try:
--> 325       executor_output = self._executor_operator.run_executor(execution_info)
    326       code = executor_output.execution_result.code
    327       if code != 0:

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tfx/orchestration/portable/beam_executor_operator.py in run_executor(self, execution_info)
     84         stateful_working_dir=execution_info.stateful_working_dir)
     85     executor = self._executor_cls(context=context)
---> 86     return python_executor_operator.run_with_executor(execution_info, executor)

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tfx/orchestration/portable/python_executor_operator.py in run_with_executor(execution_info, executor)
     64   output_dict = copy.deepcopy(execution_info.output_dict)
     65   result = executor.Do(execution_info.input_dict, output_dict,
---> 66                        execution_info.exec_properties)
     67   if not result:
     68     # If result is not returned from the Do function, then try to

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tfx/components/transform/executor.py in Do(self, input_dict, output_dict, exec_properties)
    490       label_outputs[labels.CACHE_OUTPUT_PATH_LABEL] = cache_output
    491     status_file = 'status_file'  # Unused
--> 492     self.Transform(label_inputs, label_outputs, status_file)
    493     absl.logging.debug('Cleaning up temp path %s on executor success',
    494                        temp_path)

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tfx/components/transform/executor.py in Transform(***failed resolving arguments***)
   1025                       output_cache_dir, compute_statistics,
   1026                       per_set_stats_output_paths, materialization_format,
-> 1027                       len(analyze_data_paths))
   1028   # TODO(b/122478841): Writes status to status file.
   1029 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tfx/components/transform/executor.py in _RunBeamImpl(self, analyze_data_list, transform_data_list, preprocessing_fn, stats_options_updater_fn, force_tf_compat_v1, input_dataset_metadata, transform_output_path, raw_examples_data_format, temp_path, input_cache_dir, output_cache_dir, compute_statistics, per_set_stats_output_paths, materialization_format, analyze_paths_count)
   1338                      Executor._RecordBatchToExamples)
   1339                  | 'Materialize[{}]'.format(infix) >> self._WriteExamples(
-> 1340                      materialization_format, dataset.materialize_output_path))
   1341 
   1342     return _Status.OK()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/pipeline.py in __exit__(self, exc_type, exc_val, exc_tb)
    578     try:
    579       if not exc_type:
--> 580         self.result = self.run()
    581         self.result.wait_until_finish()
    582     finally:

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/pipeline.py in run(self, test_runner_api)
    557         finally:
    558           shutil.rmtree(tmpdir)
--> 559       return self.runner.run_pipeline(self, self._options)
    560     finally:
    561       shutil.rmtree(self.local_tempdir, ignore_errors=True)

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/direct/direct_runner.py in run_pipeline(self, pipeline, options)
    131       runner = BundleBasedDirectRunner()
    132 
--> 133     return runner.run_pipeline(pipeline, options)
    134 
    135 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_pipeline(self, pipeline, options)
    181 
    182     self._latest_run_result = self.run_via_runner_api(
--> 183         pipeline.to_runner_api(default_environment=self._default_environment))
    184     return self._latest_run_result
    185 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_via_runner_api(self, pipeline_proto)
    191     # TODO(pabloem, BEAM-7514): Create a watermark manager (that has access to
    192     #   the teststream (if any), and all the stages).
--> 193     return self.run_stages(stage_context, stages)
    194 
    195   @contextlib.contextmanager

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in run_stages(self, stage_context, stages)
    357           stage_results = self._run_stage(
    358               runner_execution_context,
--> 359               bundle_context_manager,
    360           )
    361           monitoring_infos_by_stage[stage.name] = (

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in _run_stage(self, runner_execution_context, bundle_context_manager)
    553               input_timers,
    554               expected_timer_output,
--> 555               bundle_manager)
    556 
    557       final_result = merge_results(last_result)

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in _run_bundle(self, runner_execution_context, bundle_context_manager, data_input, data_output, input_timers, expected_timer_output, bundle_manager)
    593 
    594     result, splits = bundle_manager.process_bundle(
--> 595         data_input, data_output, input_timers, expected_timer_output)
    596     # Now we collect all the deferred inputs remaining from bundle execution.
    597     # Deferred inputs can be:

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/fn_runner.py in process_bundle(self, inputs, expected_outputs, fired_timers, expected_output_timers, dry_run)
    894             process_bundle_descriptor.id,
    895             cache_tokens=[next(self._cache_token_generator)]))
--> 896     result_future = self._worker_handler.control_conn.push(process_bundle_req)
    897 
    898     split_results = []  # type: List[beam_fn_api_pb2.ProcessBundleSplitResponse]

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/portability/fn_api_runner/worker_handlers.py in push(self, request)
    378       self._uid_counter += 1
    379       request.instruction_id = 'control_%s' % self._uid_counter
--> 380     response = self.worker.do_instruction(request)
    381     return ControlFuture(request.instruction_id, response)
    382 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py in do_instruction(self, request)
    605       # E.g. if register is set, this will call self.register(request.register))
    606       return getattr(self, request_type)(
--> 607           getattr(request, request_type), request.instruction_id)
    608     else:
    609       raise NotImplementedError

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/sdk_worker.py in process_bundle(self, request, instruction_id)
    642         with self.maybe_profile(instruction_id):
    643           delayed_applications, requests_finalization = (
--> 644               bundle_processor.process_bundle(instruction_id))
    645           monitoring_infos = bundle_processor.monitoring_infos()
    646           monitoring_infos.extend(self.state_cache_metrics_fn())

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py in process_bundle(self, instruction_id)
    998           elif isinstance(element, beam_fn_api_pb2.Elements.Data):
    999             input_op_by_transform_id[element.transform_id].process_encoded(
-> 1000                 element.data)
   1001 
   1002       # Finish all operations.

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/bundle_processor.py in process_encoded(self, encoded_windowed_values)
    226       decoded_value = self.windowed_coder_impl.decode_from_stream(
    227           input_stream, True)
--> 228       self.output(decoded_value)
    229 
    230   def monitoring_infos(self, transform_id, tag_to_pcollection_id):

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SdfProcessSizedElements.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SdfProcessSizedElements.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process_with_sized_restriction()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.FlattenOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.Operation.output()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.ConsumerSet.receive()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.SimpleInvoker.invoke_process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.SingletonConsumerSet.receive()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/worker/operations.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.worker.operations.DoOperation.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner._reraise_augmented()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/future/utils/__init__.py in raise_with_traceback(exc, traceback)
    444         if traceback == Ellipsis:
    445             _, _, traceback = sys.exc_info()
--> 446         raise exc.with_traceback(traceback)
    447 
    448 else:

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.DoFnRunner.process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker.invoke_process()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/apache_beam/runners/common.cpython-36m-x86_64-linux-gnu.so in apache_beam.runners.common._OutputProcessor.process_outputs()

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow_transform/beam/impl.py in process(self, batch, saved_model_dir)
    439     assert self._graph_state.saved_model_dir == saved_model_dir
    440 
--> 441     yield self._handle_batch(batch)
    442 
    443 

~/miniconda3/envs/zenml-py36/lib/python3.6/site-packages/tensorflow_transform/beam/impl.py in _handle_batch(self, batch)
    386           Batch instances: {},
    387           Fetching the values for the following Tensor keys: {}.""".format(
--> 388               str(e), batch, self._graph_state.outputs_tensor_keys))
    389 
    390     result.update(self._get_passthrough_data_from_recordbatch(batch))

ValueError: An error occured while trying to apply the transformation: " No registered 'Min' OpKernel for 'GPU' devices compatible with node {{node StatefulPartitionedCall/max_6/min_and_max/Max_1}}
     (OpKernel was found, but attributes didn't match) Requested Attributes: T=DT_INT64, Tidx=DT_INT32, _XlaHasReferenceVars=false, keep_dims=false, _device="/job:localhost/replica:0/task:0/device:GPU:0"
    .  Registered:  device='XLA_CPU_JIT'; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_INT64, DT_QINT8, DT_QUINT8, DT_QINT32, DT_BFLOAT16, DT_UINT16, DT_HALF, DT_UINT32, DT_UINT64]
  device='XLA_GPU_JIT'; Tidx in [DT_INT32, DT_INT64]; T in [DT_FLOAT, DT_DOUBLE, DT_INT32, DT_UINT8, DT_INT16, DT_INT8, DT_INT64, DT_QINT8, DT_QUINT8, DT_QINT32, DT_BFLOAT16, DT_UINT16, DT_HALF, DT_UINT32, DT_UINT64]
  device='GPU'; T in [DT_INT32]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_INT32]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_DOUBLE]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_DOUBLE]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_FLOAT]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_FLOAT]; Tidx in [DT_INT32]
  device='GPU'; T in [DT_HALF]; Tidx in [DT_INT64]
  device='GPU'; T in [DT_HALF]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_DOUBLE]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_DOUBLE]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_FLOAT]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_FLOAT]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_BFLOAT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_BFLOAT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_HALF]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_HALF]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT32]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT32]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT8]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT8]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT8]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT8]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT16]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT16]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT32]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT32]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_INT64]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_INT64]; Tidx in [DT_INT32]
  device='CPU'; T in [DT_UINT64]; Tidx in [DT_INT64]
  device='CPU'; T in [DT_UINT64]; Tidx in [DT_INT32]

     [[StatefulPartitionedCall/max_6/min_and_max/Max_1]] [Op:__inference_wrapped_finalized_5475]

Function call stack:
wrapped_finalized
".
          Batch instances: pyarrow.RecordBatch
age: large_list<item: int64>
  child 0, item: int64
bmi: large_list<item: float>
  child 0, item: float
dbp: large_list<item: int64>
  child 0, item: int64
has_diabetes: large_list<item: int64>
  child 0, item: int64
insulin: large_list<item: int64>
  child 0, item: int64
pedigree: large_list<item: float>
  child 0, item: float
pgc: large_list<item: int64>
  child 0, item: int64
times_pregnant: large_list<item: int64>
  child 0, item: int64
tst: large_list<item: int64>
  child 0, item: int64,
          Fetching the values for the following Tensor keys: {'max_2/min_and_max/Identity_1', 'max_6/min_and_max/Identity_1', 'max_4/min_and_max/Identity', 'max/min_and_max/Identity_1', 'max_6/min_and_max/Identity', 'max_4/min_and_max/Identity_1', 'max_1/min_and_max/Identity', 'max_7/min_and_max/Identity_1', 'max_8/min_and_max/Identity_1', 'max_3/min_and_max/Identity', 'max_1/min_and_max/Identity_1', 'max_3/min_and_max/Identity_1', 'max/min_and_max/Identity', 'max_5/min_and_max/Identity', 'max_2/min_and_max/Identity', 'max_7/min_and_max/Identity', 'max_5/min_and_max/Identity_1', 'max_8/min_and_max/Identity'}. [while running 'Analyze/ApplySavedModel[Phase0][AnalysisIndex0]/ApplySavedModel']
htahir1 commented 3 years ago

Hmm @ucsky thanks for the bug report. It feels like there is an particular issue with your OS system and the GPU you have installed that does causes the underlying Tensorflow Transform step to fail. Could you try disabling the GPU and trying again? You can do this by setting the following env variable at the top of your runner script:

import os
os.environ["CUDA_VISIBLE_DEVICES"] = "-1"
htahir1 commented 3 years ago

@ucsky Did that fix it?

ucsky commented 3 years ago

Hello, Yes that fixed the problem, thanks.