search

tensorflow / tfx

TFX is an end-to-end platform for deploying production ML pipelines
https://tensorflow.org/tfx
Apache License 2.0
2.11k stars 704 forks source link

tfjs_next_page_prediction example failed #6023

Open Bbean opened 1 year ago

Bbean commented 1 year ago

System information

Describe the current behavior

python tfjs_next_page_prediction_e2e_test.py get error: FAIL: testTFJSPagePredictionPipeline (main.TFJSNextPagePredictionPipelineEndToEndTest) TFJSNextPagePredictionPipelineEndToEndTest.testTFJSPagePredictionPipeline

Traceback (most recent call last): File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 813, in _convert_tf_saved_model resource_ids_maps = _get_resource_ids_maps(model, concrete_func, resource_initializer_concrete_func) File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 680, in _get_resource_ids_maps captured_input_index = resource_id_to_captured_input_index[_id] KeyError: 3207

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "/home/Documents/tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_e2e_test.py", line 94, in testTFJSPagePredictionPipeline LocalDagRunner().run(pipeline) File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/tfx_runner.py", line 124, in run return self.run_with_ir(pipeline_pb, run_options=run_options_pb, kwargs) File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/local/local_dag_runner.py", line 109, in run_with_ir component_launcher.launch() File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/launcher.py", line 573, in launch executor_output = self._run_executor(execution_info) File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/launcher.py", line 448, in _run_executor executor_output = self._executor_operator.run_executor(execution_info) File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/python_executor_operator.py", line 135, in run_executor return run_with_executor(execution_info, executor) File "/opt/conda/lib/python3.7/site-packages/tfx/orchestration/portable/python_executor_operator.py", line 59, in run_with_executor execution_info.exec_properties) File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/executor.py", line 178, in Do run_fn(fn_args) File "/home/Documents/tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_util.py", line 198, in run_fn rewriter.ModelType.TFJS_MODEL) File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/converters.py", line 131, in rewrite_saved_model dst_model_type) File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/converters.py", line 47, in _invoke_rewriter rewriter_inst.perform_rewrite(original_model, rewritten_model) File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/rewriter.py", line 106, in perform_rewrite self._rewrite(original_model, rewritten_model) File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/tfjs_rewriter.py", line 91, in _rewrite _ensure_str(rewritten_model.path)) File "/opt/conda/lib/python3.7/site-packages/tfx/components/trainer/rewriting/tfjs_rewriter.py", line 29, in _convert_tfjs_model saved_model_path, destination_path File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 818, in convert weight_shard_size_bytes, metadata_map) File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/converter.py", line 539, in _dispatch_converter metadata=metadata_map) File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 988, in convert_tf_saved_model frozen_graph_dir=frozen_graph_dir) File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 820, in _convert_tf_saved_model output_node_names) File "/opt/conda/lib/python3.7/site-packages/tensorflowjs/converters/tf_saved_model_conversion_v2.py", line 434, in _freeze_saved_model_v1 sess, meta_graph_def, output_node_names) File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 357, in new_func return func(*args, *kwargs) File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 322, in convert_variables_to_constants variable_names_denylist=variable_names_blacklist) File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py", line 1288, in convert_variables_to_constants_from_session_graph variable_names_denylist=variable_names_denylist)) File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/convert_to_constants.py", line 943, in init graph_def = graph_util.extract_sub_graph(graph_def, output_node_names) File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/util/deprecation.py", line 357, in new_func return func(args, kwargs) File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 251, in extract_sub_graph _assert_nodes_are_present(name_to_node, dest_nodes) File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/graph_util_impl.py", line 204, in _assert_nodes_are_present assert d in name_to_node, "%s is not in graph" % d AssertionError: Identity is not in graph

Describe the expected behavior

tfjs testcase pass

Standalone code to reproduce the issue

tfx/tfx/examples/tfjs_next_page_prediction/tfjs_next_page_prediction_e2e_test.py

Any reply is welcome, thanks

singhniraj08 commented 1 year ago

@Bbean,

This error looks like the pipeline fails at the tf model to tfjs model conversion step. I was able to replicate the issue. Both test case and pipeline fails with "AssertionError: Identity is not in graph" error.

Please find the attached gist. @briron, Could you please have a look why the tfjs_next_page_prediction example fails. Thanks