Duplicate entry

[evanxlao]

pipreqs --print output:

backoff==2.2.1
Flask==2.3.2
Flask_Cors==3.0.10
gql==3.4.1
openai==0.27.6
PyJWT==2.8.0
PyJWT==2.8.0
pymongo==4.3.3
python-dotenv==1.0.0
PyYAML==6.0.1
PyYAML==6.0.1
Requests==2.31.0
scipy==1.11.1
tiktoken==0.4.0
waitress==2.1.2
Werkzeug==2.3.6

meanwhile:

> pip freeze
aiohttp==3.8.5
aiosignal==1.3.1
async-timeout==4.0.2
attrs==23.1.0
backoff==2.2.1
blinker==1.6.2
certifi==2023.7.22
charset-normalizer==3.2.0
click==8.1.6
dnspython==2.4.1
Flask==2.3.2
Flask-Cors==3.0.10
frozenlist==1.4.0
gql==3.4.1
graphql-core==3.2.3
idna==3.4
itsdangerous==2.1.2
Jinja2==3.1.2
MarkupSafe==2.1.3
multidict==6.0.4
numpy==1.25.2
openai==0.27.6
PyJWT==2.8.0
pymongo==4.3.3
python-dotenv==1.0.0
PyYAML==6.0.1
regex==2023.6.3
requests==2.31.0
scipy==1.11.1
six==1.16.0
tiktoken==0.4.0
tqdm==4.65.0
urllib3==2.0.4
waitress==2.1.2
Werkzeug==2.3.6
yarl==1.9.2

tried using pipreqs --clean requirements.txt --force, but it actually does not clean it

[evanxlao]

https://github.com/bndr/pipreqs/pull/67 https://github.com/bndr/pipreqs/issues/65 https://github.com/bndr/pipreqs/pull/41

[DanielYang59]

The same issue in MacOS Sonoma 14.0. Python3 managed by Anaconda3.

$ pipreqs --print
WARNING: Import named "numpy" not found locally. Trying to resolve it at the PyPI server.
WARNING: Import named "numpy" was resolved to "numpy:1.26.0" package (https://pypi.org/project/numpy/).
Please, verify manually the final list of requirements.txt to avoid possible dependency confusions.
WARNING: Import named "pandas" not found locally. Trying to resolve it at the PyPI server.
WARNING: Import named "pandas" was resolved to "pandas:2.1.1" package (https://pypi.org/project/pandas/).
Please, verify manually the final list of requirements.txt to avoid possible dependency confusions.
WARNING: Import named "PyYAML" not found locally. Trying to resolve it at the PyPI server.
WARNING: Import named "PyYAML" was resolved to "PyYAML:6.0.1" package (https://pypi.org/project/PyYAML/).
Please, verify manually the final list of requirements.txt to avoid possible dependency confusions.
ase==3.22.1
matplotlib==3.8.0
numpy==1.26.0
pandas==2.1.1
pymatgen==2023.9.10
pytest==7.4.2
PyYAML==6.0.1
PyYAML==6.0.1
tqdm==4.66.1
INFO: Successfully output requirements

$ uname -a
Darwin Yang-MacMini 23.0.0 Darwin Kernel Version 23.0.0: Fri Sep 15 14:42:57 PDT 2023; root:xnu-10002.1.13~1/RELEASE_ARM64_T8112 arm64

[schollii]

This is still (Sep 2024) an issue in 0.4.13. A couple of workarounds to automate cleanup:

1. After running pipreqs, use sort -u or uniq. Eg on Ubuntu one can run sort -u -o requirements.txt requirements.txt
2. Instead of workaround 1, configure pipreqs for output to stdout, and filter. Eg on Ubuntu one can run pipreqs --print | uniq > requirements.txt. This can easily be aliased in your .bashrc.

In my .bashrc I have alias pipreqs="pipreqs --force --print | uniq > requirements.txt; echo 'Created requirements.txt'" which for me is safe (despite --force) because all my requirements.txt are always under version control.

If you want to be able to pass additional options to pipreqs, create a pipreqs() command instead of an alias:

pipreqs() {
   # Check if the first argument is --help
   if [ "$1" = "--help" ]; then
      command pipreqs "$@"
      return
   fi

   local exists="Created"
   if [ -f requirements.txt ]; then
      exists="Overwrote"
   fi

   command pipreqs --print "$@" | sort -u > requirements.txt
   echo "$exists requirements.txt"
}

Note this is actually equivalent to having --force.