rapidsai / dependency-file-generator

https://pypi.org/project/rapids-dependency-file-generator/
Apache License 2.0
15 stars 13 forks source link

Tool fails when using `--clean` if files with non-unicode characters exist in the repo #58

Closed mdemoret-nv closed 11 months ago

mdemoret-nv commented 11 months ago

If you run rapids-dependency-file-generator --clean with any file that has non-unicode characters in it, it will generate the following exception:

Traceback (most recent call last):
  File "${VIRTUAL_ENV}/bin/rapids-dependency-file-generator", line 8, in <module>
    sys.exit(main())
  File "${VIRTUAL_ENV}/lib/python3.10/site-packages/rapids_dependency_file_generator/cli.py", line 103, in main
    delete_existing_files(args.clean)
  File "${VIRTUAL_ENV}/lib/python3.10/site-packages/rapids_dependency_file_generator/rapids_dependency_file_generator.py", line 40, in delete_existing_files
    if HEADER in f.read():
  File "${VIRTUAL_ENV}/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xa0 in position 6038: invalid start byte

There should be a try block around reading the file to ignore files that can not be decoded. Something like this fixed it for me:

def delete_existing_files(root="."):
    """Delete any files generated by this generator.

    This function can be used to clean up a directory tree before generating a new set
    of files from scratch.

    Parameters
    ----------
    root : str
        The path to the root of the directory tree to search for files to delete.
    """
    for dirpath, _, filenames in os.walk(root):
        for fn in filter(
            lambda fn: fn.endswith(".txt") or fn.endswith(".yaml"), filenames
        ):
            try:
                with open(file_path := os.path.join(dirpath, fn)) as f:
                    if HEADER in f.read():
                        os.remove(file_path)
            except UnicodeDecodeError:
                pass
ajschmidt8 commented 11 months ago

:tada: This issue has been resolved in version 1.7.2 :tada:

The release is available on:

Your semantic-release bot :package::rocket: