While running aizynthtrain step to train and validate model, the program fails at model_validation step. can you give some idea of this error? I guess I miss the expansion policy configration for aizynthfinder package? the error is like below.
Many Thanks!
2023-07-28 16:31:08.698 [1690533000752048/model_validation/8 (pid 19599)] failed:
2023-07-28 16:31:09.160 [1690533000752048/model_validation/8 (pid 19599)] Internal error
2023-07-28 16:31:09.166 [1690533000752048/model_validation/8 (pid 19599)] Traceback (most recent call last):
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/cli.py", line 1171, in main
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] start(auto_envvar_prefix="METAFLOW", obj=state)
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 829, in call
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] return self.main(args, kwargs)
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 782, in main
2023-07-28 16:31:09.168 [1690533000752048/model_validation/8 (pid 19599)] rv = self.invoke(ctx)
2023-07-28 16:31:09.168 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
2023-07-28 16:31:09.168 [1690533000752048/model_validation/8 (pid 19599)] return _process_result(sub_ctx.command.invoke(sub_ctx))
2023-07-28 16:31:09.169 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
2023-07-28 16:31:09.169 [1690533000752048/model_validation/8 (pid 19599)] return ctx.invoke(self.callback, ctx.params)
2023-07-28 16:31:09.169 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
2023-07-28 16:31:09.170 [1690533000752048/model_validation/8 (pid 19599)] return callback(args, kwargs)
2023-07-28 16:31:09.170 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/decorators.py", line 21, in new_func
2023-07-28 16:31:09.170 [1690533000752048/model_validation/8 (pid 19599)] return f(get_current_context(), args, kwargs)
2023-07-28 16:31:09.171 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/cli.py", line 580, in step
2023-07-28 16:31:09.171 [1690533000752048/model_validation/8 (pid 19599)] task.run_step(
2023-07-28 16:31:09.171 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/task.py", line 587, in run_step
2023-07-28 16:31:09.172 [1690533000752048/model_validation/8 (pid 19599)] self._exec_step_function(step_func)
2023-07-28 16:31:09.172 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/task.py", line 61, in _exec_step_function
2023-07-28 16:31:09.172 [1690533000752048/model_validation/8 (pid 19599)] step_function()
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] File "/mnt/d/projects/ai/code/aizynthtrain-main/aizynthtrain/pipelines/expansion_model_pipeline.py", line 99, in model_validation
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] eval_one_step([self.config_path])
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] File "/mnt/d/projects/ai/code/aizynthtrain-main/aizynthtrain/modelling/expansion_policy/eval_one_step.py", line 194, in main
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] expander_output = _run_expander(ref_reactions_path, config_path, config)
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] File "/mnt/d/projects/ai/code/aizynthtrain-main/aizynthtrain/modelling/expansion_policy/eval_one_step.py", line 143, in _run_expander
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] expander = AiZynthExpander(configfile=config_path)
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/aizynthfinder.py", line 249, in init
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] self.config = Configuration.from_file(configfile)
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/context/config.py", line 116, in from_file
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] return Configuration.from_dict(_config)
2023-07-28 16:31:09.175 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/context/config.py", line 96, in from_dict
2023-07-28 16:31:09.175 [1690533000752048/model_validation/8 (pid 19599)] config_obj.expansion_policy.load_from_config(src_copy.get("policy", {}))
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/context/policy/policies.py", line 112, in load_from_config
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] strategy = TemplateBasedExpansionStrategy(
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/context/policy/expansion_strategies.py", line 94, in init
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] self.templates: pd.DataFrame = pd.read_hdf(templatefile, "table")
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/pandas/io/pytables.py", line 416, in read_hdf
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] store = HDFStore(path_or_buf, mode=mode, errors=errors, kwargs)
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/pandas/io/pytables.py", line 578, in init
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] self.open(mode=mode, kwargs)
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/pandas/io/pytables.py", line 737, in open
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] self._handle = tables.open_file(self._path, self._mode, kwargs)
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/tables/file.py", line 300, in open_file
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] return File(filename, mode, title, root_uep, filters, kwargs)
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/tables/file.py", line 750, in init
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] self._g_new(filename, mode, params)
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "tables/hdf5extension.pyx", line 486, in tables.hdf5extension.File._g_new
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] tables.exceptions.HDF5ExtError: HDF5 error back trace
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)]
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "H5F.c", line 620, in H5Fopen
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] unable to open file
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "H5VLcallback.c", line 3501, in H5VL_file_open
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] failed to iterate over available VOL connector plugins
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "H5PLpath.c", line 578, in H5PLpath_table_iterate
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] can't iterate over plugins in plugin path '(null)'
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5PLpath.c", line 620, in H5PLpath_table_iterate_process_path
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] can't open directory: /usr/local/hdf5/lib/plugin
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5VLcallback.c", line 3351, in H5VLfile_open
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] open failed
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5VLnative_file.c", line 97, in H5VLnative_file_open
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] unable to open file
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5Fint.c", line 1990, in H5F_open
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] unable to read superblock
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5Fsuper.c", line 405, in H5F__super_read
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] file signature not found
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)]
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] End of HDF5 error back trace
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)]
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] Unable to open/create file 'uspto_unique_templates.csv.gz'
2023-07-28 16:31:09.848 [1690533000752048/model_validation/8 (pid 19599)]
2023-07-28 16:31:09.862 [1690533000752048/model_validation/8 (pid 19599)] Task failed.
yangxfei
commented
1 year ago
While running aizynthtrain step to train and validate model, the program fails at model_validation step. can you give some idea of this error? I guess I miss the expansion policy configration for aizynthfinder package? the error is like below.
Many Thanks!
2023-07-28 16:31:08.698 [1690533000752048/model_validation/8 (pid 19599)] failed:
2023-07-28 16:31:09.160 [1690533000752048/model_validation/8 (pid 19599)] Internal error
2023-07-28 16:31:09.166 [1690533000752048/model_validation/8 (pid 19599)] Traceback (most recent call last):
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/cli.py", line 1171, in main
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] start(auto_envvar_prefix="METAFLOW", obj=state)
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 829, in call
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] return self.main(args, kwargs)
2023-07-28 16:31:09.167 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 782, in main
2023-07-28 16:31:09.168 [1690533000752048/model_validation/8 (pid 19599)] rv = self.invoke(ctx)
2023-07-28 16:31:09.168 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 1259, in invoke
2023-07-28 16:31:09.168 [1690533000752048/model_validation/8 (pid 19599)] return _process_result(sub_ctx.command.invoke(sub_ctx))
2023-07-28 16:31:09.169 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 1066, in invoke
2023-07-28 16:31:09.169 [1690533000752048/model_validation/8 (pid 19599)] return ctx.invoke(self.callback, ctx.params)
2023-07-28 16:31:09.169 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/core.py", line 610, in invoke
2023-07-28 16:31:09.170 [1690533000752048/model_validation/8 (pid 19599)] return callback(args, kwargs)
2023-07-28 16:31:09.170 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/_vendor/click/decorators.py", line 21, in new_func
2023-07-28 16:31:09.170 [1690533000752048/model_validation/8 (pid 19599)] return f(get_current_context(), args, kwargs)
2023-07-28 16:31:09.171 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/cli.py", line 580, in step
2023-07-28 16:31:09.171 [1690533000752048/model_validation/8 (pid 19599)] task.run_step(
2023-07-28 16:31:09.171 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/task.py", line 587, in run_step
2023-07-28 16:31:09.172 [1690533000752048/model_validation/8 (pid 19599)] self._exec_step_function(step_func)
2023-07-28 16:31:09.172 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/metaflow/task.py", line 61, in _exec_step_function
2023-07-28 16:31:09.172 [1690533000752048/model_validation/8 (pid 19599)] step_function()
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] File "/mnt/d/projects/ai/code/aizynthtrain-main/aizynthtrain/pipelines/expansion_model_pipeline.py", line 99, in model_validation
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] eval_one_step([self.config_path])
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] File "/mnt/d/projects/ai/code/aizynthtrain-main/aizynthtrain/modelling/expansion_policy/eval_one_step.py", line 194, in main
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] expander_output = _run_expander(ref_reactions_path, config_path, config)
2023-07-28 16:31:09.173 [1690533000752048/model_validation/8 (pid 19599)] File "/mnt/d/projects/ai/code/aizynthtrain-main/aizynthtrain/modelling/expansion_policy/eval_one_step.py", line 143, in _run_expander
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] expander = AiZynthExpander(configfile=config_path)
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/aizynthfinder.py", line 249, in init
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] self.config = Configuration.from_file(configfile)
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/context/config.py", line 116, in from_file
2023-07-28 16:31:09.174 [1690533000752048/model_validation/8 (pid 19599)] return Configuration.from_dict(_config)
2023-07-28 16:31:09.175 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/context/config.py", line 96, in from_dict
2023-07-28 16:31:09.175 [1690533000752048/model_validation/8 (pid 19599)] config_obj.expansion_policy.load_from_config(src_copy.get("policy", {}))
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/context/policy/policies.py", line 112, in load_from_config
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] strategy = TemplateBasedExpansionStrategy(
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/aizynthfinder/context/policy/expansion_strategies.py", line 94, in init
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] self.templates: pd.DataFrame = pd.read_hdf(templatefile, "table")
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/pandas/io/pytables.py", line 416, in read_hdf
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] store = HDFStore(path_or_buf, mode=mode, errors=errors, kwargs)
2023-07-28 16:31:09.845 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/pandas/io/pytables.py", line 578, in init
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] self.open(mode=mode, kwargs)
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/pandas/io/pytables.py", line 737, in open
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] self._handle = tables.open_file(self._path, self._mode, kwargs)
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/tables/file.py", line 300, in open_file
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] return File(filename, mode, title, root_uep, filters, kwargs)
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "/home/qa/miniconda3/envs/aizynthtrain/lib/python3.9/site-packages/tables/file.py", line 750, in init
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] self._g_new(filename, mode, params)
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "tables/hdf5extension.pyx", line 486, in tables.hdf5extension.File._g_new
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] tables.exceptions.HDF5ExtError: HDF5 error back trace
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)]
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "H5F.c", line 620, in H5Fopen
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] unable to open file
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "H5VLcallback.c", line 3501, in H5VL_file_open
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] failed to iterate over available VOL connector plugins
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] File "H5PLpath.c", line 578, in H5PLpath_table_iterate
2023-07-28 16:31:09.846 [1690533000752048/model_validation/8 (pid 19599)] can't iterate over plugins in plugin path '(null)'
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5PLpath.c", line 620, in H5PLpath_table_iterate_process_path
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] can't open directory: /usr/local/hdf5/lib/plugin
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5VLcallback.c", line 3351, in H5VLfile_open
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] open failed
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5VLnative_file.c", line 97, in H5VLnative_file_open
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] unable to open file
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5Fint.c", line 1990, in H5F_open
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] unable to read superblock
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] File "H5Fsuper.c", line 405, in H5F__super_read
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] file signature not found
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)]
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] End of HDF5 error back trace
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)]
2023-07-28 16:31:09.847 [1690533000752048/model_validation/8 (pid 19599)] Unable to open/create file 'uspto_unique_templates.csv.gz'
2023-07-28 16:31:09.848 [1690533000752048/model_validation/8 (pid 19599)]
2023-07-28 16:31:09.862 [1690533000752048/model_validation/8 (pid 19599)] Task failed.