I'm using tfx Transform component with tensorflow-text op in dataflow, and facing this issue. (Env: tensorflow-text==2.8.2, tfx==1.7.1, tensorflow-transform==1.7.0)
File "apache_beam/runners/common.py", line 1198, in apache_beam.runners.common.DoFnRunner.process
File "apache_beam/runners/common.py", line 718, in apache_beam.runners.common.PerWindowInvoker.invoke_process
File "apache_beam/runners/common.py", line 836, in apache_beam.runners.common.PerWindowInvoker._invoke_process_per_window
File "apache_beam/runners/common.py", line 1334, in apache_beam.runners.common._OutputProcessor.process_outputs
File "/usr/local/lib/python3.8/dist-packages/tensorflow_transform/beam/impl.py", line 402, in process self._graph_state = self._shared_graph_state_handle.acquire(
File "/usr/local/lib/python3.8/dist-packages/apache_beam/utils/shared.py", line 305, in acquire return _shared_map.acquire(self._key, constructor_fn, tag)
File "/usr/local/lib/python3.8/dist-packages/apache_beam/utils/shared.py", line 246, in acquire result = control_block.acquire(constructor_fn, tag)
File "/usr/local/lib/python3.8/dist-packages/apache_beam/utils/shared.py", line 139, in acquire result = constructor_fn()
File "/usr/local/lib/python3.8/dist-packages/tensorflow_transform/beam/impl.py", line 403, in <lambda> lambda: self._make_graph_state(saved_model_dir))
File "/usr/local/lib/python3.8/dist-packages/tensorflow_transform/beam/impl.py", line 369, in _make_graph_state result = self._GraphStateV2(saved_model_dir,
File "/usr/local/lib/python3.8/dist-packages/tensorflow_transform/beam/impl.py", line 219, in __init__ saved_model_loader = saved_transform_io_v2.SavedModelLoader(
File "/usr/local/lib/python3.8/dist-packages/tensorflow_transform/saved/saved_transform_io_v2.py", line 126, in __init__ imported = tf.compat.v2.saved_model.load(saved_model_dir)
File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 936, in load result = load_internal(export_dir, tags, options)["root"]
File "/usr/local/lib/python3.8/dist-packages/tensorflow/python/saved_model/load.py", line 977, in load_internal raise FileNotFoundError( FileNotFoundError: Op type not registered 'NormalizeUTF8' in binary running on user-demographic-model-ta-06130957-qubq-harness-cp40. Make sure the Op and Kernel are registered in the binary running in this process. Note that if you are loading a saved graph which used ops from tf.contrib, accessing (e.g.) `tf.contrib.resampler` should be done before importing the graph, as contrib ops are lazily registered when the module is first accessed. You may be trying to load on a different device from the computational device. Consider setting the `experimental_io_device` option in `tf.saved_model.LoadOptions` to the io_device such as '/job:localhost'.
I think the below code is the main reason for this issue.
As you can see from the code, tf-transform tries to register required ops in L154, but it also tries to load the model in L126 before the required ops are registered.
I'm using tfx Transform component with tensorflow-text op in dataflow, and facing this issue. (Env: tensorflow-text==2.8.2, tfx==1.7.1, tensorflow-transform==1.7.0)
I think the below code is the main reason for this issue.
https://github.com/tensorflow/transform/blob/74789ff3818f01e58df88bf67306a0e89bf11457/tensorflow_transform/saved/saved_transform_io_v2.py#L112-L154
As you can see from the code, tf-transform tries to register required ops in L154, but it also tries to load the model in L126 before the required ops are registered.