search

landscapeio / prospector

Inspects Python source files and provides information about type and location of classes, methods etc
GNU General Public License v2.0
1.94k stars 171 forks source link

[BUG] prospector fails with UnicodeDecodeError on requirements.txt having DOS (CRLF) line endings #660

Open amai2012 opened 5 months ago

amai2012 commented 5 months ago

Describe the bug prospector fails with UnicodeDecodeError (see below) if the requirements.txt has DOS line endings.

dos2unix requirements.txt

cures the problem.

To Reproduce

  1. prospector .

Expected behavior Either accept the file - since pip can handle it - or show a helpful error message

Screenshots 

Traceback (most recent call last):
  File "/home/me/project/mysite/venv/bin/prospector", line 8, in <module>
    sys.exit(main())
  File "/home/me/project/mysite/venv/lib/python3.10/site-packages/prospector/run.py", line 188, in main
    config = ProspectorConfig()
  File "/home/me/project/mysite/venv/lib/python3.10/site-packages/prospector/config/__init__.py", line 36, in __init__
    self.libraries = self._find_used_libraries(self.config, self.profile)
  File "/home/me/project/mysite/venv/lib/python3.10/site-packages/prospector/config/__init__.py", line 215, in _find_used_libraries
    for found_dep in autodetect_libraries(self.workdir):
  File "/home/me/project/mysite/venv/lib/python3.10/site-packages/prospector/autodetect.py", line 89, in autodetect_libraries
    libraries = find_from_requirements(path)
  File "/home/me/project/mysite/venv/lib/python3.10/site-packages/prospector/autodetect.py", line 72, in find_from_requirements
    reqs = find_requirements(path)
  File "/home/me/project/mysite/venv/lib/python3.10/site-packages/requirements_detector/detect.py", line 83, in find_requirements
    requirements += from_requirements_txt(reqfile)
  File "/home/me/project/mysite/venv/lib/python3.10/site-packages/requirements_detector/detect.py", line 146, in from_requirements_txt
    for req in f.readlines():
  File "/usr/lib/python3.10/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xff in position 0: invalid start byte
> file requirements.txt
requirements.txt: Unicode text, UTF-16, little-endian text, with CRLF line terminators

Environment (please complete the following information):

Additional context dependencies installed, e.g. the output of pip freeze also helps.

doesn't matter imho

carlio commented 5 months ago

Additional context dependencies installed, e.g. the output of pip freeze also helps. doesn't matter imho

Actually it does, because this isn't a bug in prospector, it's a bug in one of its dependencies.

Specifically, requirements-detector. If you're not happy sharing the entire pip freeze fair enough but please share pip freeze | grep requirements-detector as that will add useful context

amai2012 commented 5 months ago

@carlio Thanks for the fast response. Here's more info:

me@computer:~/project/mysite$ pip freeze | grep requirements
pip-requirements-parser==32.0.1
requirements-detector==1.2.2