Windows path incompatibility

[azizderbel]

I encountered this issue when following the guide example for building TFX pipeline locally : build_local_pipeline. According some blogs TFX is not compatible yet with windows due to issue related to path interpretation, but there is not confirmation in the official documentation. Is there any solution to work around thaht in windows env ? INFO:absl:MetadataStore with DB connection initialized INFO:absl:select span and version = (0, None) INFO:absl:latest span and version = (0, None) INFO:absl:MetadataStore with DB connection initialized INFO:absl:Going to run a new execution 1 ERROR:absl:Failed to make stateful working dir: .\tfx_pipeline_output\myfirstpipeline\CsvExampleGen.system\stateful_working_dir\2022-10-08T12:38:16.377353 Traceback (most recent call last): File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\orchestration\portable\outputs_utils.py", line 226, in get_stateful_working_directory fileio.makedirs(stateful_working_dir) File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\dsl\io\fileio.py", line 80, in makedirs _get_filesystem(path).makedirs(path) File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\dsl\io\plugins\tensorflow_gfile.py", line 76, in makedirs tf.io.gfile.makedirs(path) File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 511, in recursive_create_dir_v2 _pywrap_file_io.RecursivelyCreateDir(compat.path_to_bytes(path)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Failed to create a directory: .\tfx_pipeline_output\myfirstpipeline\CsvExampleGen.system/stateful_working_dir/2022-10-08T12:38:16.377353; Invalid argument Traceback (most recent call last): File "local_runner.py", line 80, in run() File "local_runner.py", line 56, in run tfx.orchestration.LocalDagRunner().run( File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\orchestration\portable\tfx_runner.py", line 124, in run return self.run_with_ir(pipeline_pb, run_options=run_options_pb, **kwargs) File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\orchestration\local\local_dag_runner.py", line 109, in run_with_ir component_launcher.launch() File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\orchestration\portable\launcher.py", line 550, in launch execution_preparation_result = self._prepare_execution() File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\orchestration\portable\launcher.py", line 412, in _prepare_execution self._output_resolver.get_stateful_working_directory()), File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\orchestration\portable\outputs_utils.py", line 226, in get_stateful_working_directory fileio.makedirs(stateful_working_dir) File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\dsl\io\fileio.py", line 80, in makedirs _get_filesystem(path).makedirs(path) File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tfx\dsl\io\plugins\tensorflow_gfile.py", line 76, in makedirs tf.io.gfile.makedirs(path) File "c:\users\lenovo\appdata\local\programs\python\python38\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 511, in recursive_create_dir_v2 _pywrap_file_io.RecursivelyCreateDir(compat.path_to_bytes(path)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Failed to create a directory: .\tfx_pipeline_output\myfirstpipeline\CsvExampleGen.system/stateful_working_dir/2022-10-08T12:38:16.377353; Invalid argument Error while running "c:\users\lenovo\appdata\local\programs\python\python38\python.exe local_runner.py"

[singhniraj08]

@azizderbel,

As per similar issue, currently there is no plan to support TFX on Windows.

Although, the root for this issue looks like path difference between Linux and Windows OS. The fix is to use the platform-appropriate path separators while setting up directories via os.path.join rather than hard-coding forward slashes. Please refer here. Thank you!

[NarasimmanSaravana1994]

@azizderbel

Same issue I'm also facing,

INFO:absl:Using deployment config: executor_specs { key: "CsvExampleGen" value { beam_executable_spec { python_executor_spec { class_path: "tfx.components.example_gen.csv_example_gen.executor.Executor" } } } } custom_driver_specs { key: "CsvExampleGen" value { python_class_executable_spec { class_path: "tfx.components.example_gen.driver.FileBasedDriver" } } } metadata_connection_config { database_connection_config { sqlite { filename_uri: ".\tfx_metadata\ids_pipeline\metadata.db" connection_mode: READWRITE_OPENCREATE } } }

INFO:absl:Using connection config: sqlite { filename_uri: ".\tfx_metadata\ids_pipeline\metadata.db" connection_mode: READWRITE_OPENCREATE }

INFO:absl:Component CsvExampleGen is running. INFO:absl:Running launcher for node_info { type { name: "tfx.components.example_gen.csv_example_gen.component.CsvExampleGen" } id: "CsvExampleGen" } contexts { contexts { type { name: "pipeline" } name { field_value { string_value: "ids_pipeline" } } } contexts { type { name: "pipeline_run" } name { field_value { string_value: "2022-10-13T13:36:25.244089" } } } contexts { type { name: "node" } name { field_value { string_value: "ids_pipeline.CsvExampleGen" } } } } outputs { outputs { key: "examples" value { artifact_spec { type { name: "Examples" properties { key: "span" value: INT } properties { key: "split_names" value: STRING } properties { key: "version" value: INT } base_type: DATASET } } } } } parameters { parameters { key: "input_base" value { field_value { string_value: ".\data" } } } parameters { key: "input_config" value { field_value { string_value: "{\n \"splits\": [\n {\n \"name\": \"single_split\",\n \"pattern\": \"*\"\n }\n ]\n}" } } } parameters { key: "output_config" value { field_value { string_value: "{\n \"split_config\": {\n \"splits\": [\n {\n \"hash_buckets\": 2,\n \"name\": \"train\"\n },\n {\n \"hash_buckets\": 1,\n \"name\": \"eval\"\n }\n ]\n }\n}" } } } parameters { key: "output_data_format" value { field_value { int_value: 6 } } } parameters { key: "output_file_format" value { field_value { int_value: 5 } } } } execution_options { caching_options { enable_cache: true } }

INFO:absl:MetadataStore with DB connection initialized INFO:absl:select span and version = (0, None) INFO:absl:latest span and version = (0, None) INFO:absl:MetadataStore with DB connection initialized INFO:absl:Going to run a new execution 1 ERROR:absl:Failed to make stateful working dir: .\my_pipeline_output\CsvExampleGen.system\stateful_working_dir\2022-10-13T13:36:25.244089 Traceback (most recent call last): File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\orchestration\portable\outputs_utils.py", line 226, in get_stateful_working_directory fileio.makedirs(stateful_working_dir) File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\dsl\io\fileio.py", line 80, in makedirs _get_filesystem(path).makedirs(path) File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\dsl\io\plugins\tensorflow_gfile.py", line 76, in makedirs tf.io.gfile.makedirs(path) File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 511, in recursive_create_dir_v2 _pywrap_file_io.RecursivelyCreateDir(compat.path_to_bytes(path)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Failed to create a directory: .\my_pipeline_output\CsvExampleGen.system/stateful_working_dir/2022-10-13T13:36:25.244089; Invalid argument Traceback (most recent call last): File ".\ids_pipeline.py", line 9, in IDSTFXPIPELINE(pipeline_name=PIPELINE_NAME, File "C:\Users\XXX\Local\poc\tfx\pipeline\pipeline.py", line 45, in run tfx.orchestration.LocalDagRunner().run(self.__pipeline) File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\orchestration\portable\tfx_runner.py", line 124, in run return self.run_with_ir(pipeline_pb, run_options=run_options_pb, **kwargs) File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\orchestration\local\local_dag_runner.py", line 109, in run_with_ir component_launcher.launch() File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\orchestration\portable\launcher.py", line 550, in launch execution_preparation_result = self._prepare_execution() File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\orchestration\portable\launcher.py", line 412, in _prepare_execution self._output_resolver.get_stateful_working_directory()), File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\orchestration\portable\outputs_utils.py", line 226, in get_stateful_working_directory fileio.makedirs(stateful_working_dir) File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\dsl\io\fileio.py", line 80, in makedirs _get_filesystem(path).makedirs(path) File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tfx\dsl\io\plugins\tensorflow_gfile.py", line 76, in makedirs tf.io.gfile.makedirs(path) File "C:\Users\XXX\Local\environments\tfx\lib\site-packages\tensorflow\python\lib\io\file_io.py", line 511, in recursive_create_dir_v2 _pywrap_file_io.RecursivelyCreateDir(compat.path_to_bytes(path)) tensorflow.python.framework.errors_impl.InvalidArgumentError: Failed to create a directory: .\my_pipeline_output\CsvExampleGen.system/stateful_working_dir/2022-10-13T13:36:25.244089; Invalid argument

[singhniraj08]

@NarasimmanSaravana1994, @azizderbel

This happens because in windows the character : is not allowed to be used in file/folder names. I have found a similar issue where the workaround is suggested by adding self._pipeline_run_id.replace(':', '_') in the get_stateful_working_directory function in in tfx\orchestration\portable\outputs_utils.py file.

Kindly let me know if it helps. Thank you!

[singhniraj08]

Closing this due to inactivity. Please take a look into the answers provided above, feel free to reopen and post your comments(if you still have queries on this). Thank you!

[google-ml-butler[bot]]

Are you satisfied with the resolution of your issue? Yes No