search

digital-engineering / airbnb-scraper

Airbnb Scraper: Advanced Airbnb Search using Scrapy
GNU General Public License v3.0
190 stars 66 forks source link

fixed conflicts in import modules #21

Closed Mcklmo closed 2 years ago

Mcklmo commented 2 years ago

when i tried to run the pip install -Ur requirements.txt, i got an error, resulting in not being able to install the modules at all

digitalengineering commented 2 years ago

Hi @Mcklmo :wave: Thanks for the contribution!

I just tested pip install -Ur requirements.txt with a fresh environment and it completed successfully on my machine. Which version of Python are you on? What was the error message?

Mcklmo commented 2 years ago

Im using Python 3.10.4.

The error message is:

`ERROR: Cannot install -r requirements.txt (line 12), -r requirements.txt (line 37) and idna==3.1 because these package versions have conflicting dependencies.

The conflict is caused by: The user requested idna==3.1 hyperlink 21.0.0 depends on idna>=2.5 requests 2.25.1 depends on idna<3 and >=2.5

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts`

The whole installing message: `Collecting attrs==20.3.0 Using cached attrs-20.3.0-py2.py3-none-any.whl (49 kB) Collecting Automat==20.2.0 Using cached Automat-20.2.0-py2.py3-none-any.whl (31 kB) Collecting certifi==2020.12.5 Using cached certifi-2020.12.5-py2.py3-none-any.whl (147 kB) Collecting cffi==1.14.5 Using cached cffi-1.14.5-cp39-cp39-win_amd64.whl (179 kB) Collecting chardet==4.0.0 Using cached chardet-4.0.0-py2.py3-none-any.whl (178 kB) Collecting constantly==15.1.0 Using cached constantly-15.1.0-py2.py3-none-any.whl (7.9 kB) Collecting cryptography==3.4.7 Using cached cryptography-3.4.7-cp36-abi3-win_amd64.whl (1.6 MB) Collecting cssselect==1.1.0 Using cached cssselect-1.1.0-py2.py3-none-any.whl (16 kB) Collecting elasticsearch==7.12.0 Using cached elasticsearch-7.12.0-py2.py3-none-any.whl (334 kB) Collecting elasticsearch-dsl==7.3.0 Using cached elasticsearch_dsl-7.3.0-py2.py3-none-any.whl (62 kB) Collecting et-xmlfile==1.0.1 Using cached et_xmlfile-1.0.1-py3-none-any.whl Collecting hyperlink==21.0.0 Using cached hyperlink-21.0.0-py2.py3-none-any.whl (74 kB) Collecting idna==3.1 Using cached idna-3.1-py3-none-any.whl (58 kB) Collecting incremental==21.3.0 Using cached incremental-21.3.0-py2.py3-none-any.whl (15 kB) Collecting iniconfig==1.1.1 Using cached iniconfig-1.1.1-py2.py3-none-any.whl (5.0 kB) Collecting itemadapter==0.2.0 Using cached itemadapter-0.2.0-py3-none-any.whl (9.3 kB) Collecting itemloaders==1.0.4 Using cached itemloaders-1.0.4-py3-none-any.whl (11 kB) Collecting jdcal==1.4.1 Using cached jdcal-1.4.1-py2.py3-none-any.whl (9.5 kB) Collecting jmespath==0.10.0 Using cached jmespath-0.10.0-py2.py3-none-any.whl (24 kB) Collecting lxml==4.6.3 Using cached lxml-4.6.3-cp39-cp39-win_amd64.whl (3.5 MB) Collecting openpyxl==3.0.7 Using cached openpyxl-3.0.7-py2.py3-none-any.whl (243 kB) Collecting packaging==20.9 Using cached packaging-20.9-py2.py3-none-any.whl (40 kB) Collecting parsel==1.6.0 Using cached parsel-1.6.0-py2.py3-none-any.whl (13 kB) Collecting pluggy==0.13.1 Using cached pluggy-0.13.1-py2.py3-none-any.whl (18 kB) Collecting Protego==0.1.16 Using cached Protego-0.1.16-py3-none-any.whl Collecting py==1.10.0 Using cached py-1.10.0-py2.py3-none-any.whl (97 kB) Collecting pyasn1==0.4.8 Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB) Collecting pyasn1-modules==0.2.8 Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB) Collecting pycparser==2.20 Using cached pycparser-2.20-py2.py3-none-any.whl (112 kB) Collecting PyDispatcher==2.0.5 Using cached PyDispatcher-2.0.5-py3-none-any.whl Collecting PyHamcrest==2.0.2 Using cached PyHamcrest-2.0.2-py3-none-any.whl (52 kB) Collecting pyOpenSSL==20.0.1 Using cached pyOpenSSL-20.0.1-py2.py3-none-any.whl (54 kB) Collecting pyparsing==2.4.7 Using cached pyparsing-2.4.7-py2.py3-none-any.whl (67 kB) Collecting pytest==6.2.3 Using cached pytest-6.2.3-py3-none-any.whl (280 kB) Collecting python-dateutil==2.8.1 Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) Collecting queuelib==1.5.0 Using cached queuelib-1.5.0-py2.py3-none-any.whl (13 kB) Collecting requests==2.25.1 Using cached requests-2.25.1-py2.py3-none-any.whl (61 kB) Collecting Scrapy==2.4.1 Using cached Scrapy-2.4.1-py2.py3-none-any.whl (239 kB) Collecting service-identity==18.1.0 Using cached service_identity-18.1.0-py2.py3-none-any.whl (11 kB) Collecting six==1.15.0 Using cached six-1.15.0-py2.py3-none-any.whl (10 kB) Collecting toml==0.10.2 Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB) Collecting Twisted==21.2.0 Using cached Twisted-21.2.0-py3-none-any.whl (3.1 MB) Collecting urllib3==1.26.4 Using cached urllib3-1.26.4-py2.py3-none-any.whl (153 kB) Collecting w3lib==1.22.0 Using cached w3lib-1.22.0-py2.py3-none-any.whl (20 kB) Collecting zope.interface==5.3.0 Using cached zope.interface-5.3.0-cp39-cp39-win_amd64.whl (206 kB) Collecting atomicwrites>=1.0 Using cached atomicwrites-1.4.1-py2.py3-none-any.whl Collecting colorama Using cached colorama-0.4.5-py2.py3-none-any.whl (16 kB) INFO: pip is looking at multiple versions of queuelib to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of python-dateutil to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pytest to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pyparsing to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pyopenssl to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pyhamcrest to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pydispatcher to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pycparser to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pyasn1-modules to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pyasn1 to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of py to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of protego to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of pluggy to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of parsel to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of packaging to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of openpyxl to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of lxml to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of jmespath to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of jdcal to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of itemloaders to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of itemadapter to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of iniconfig to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of incremental to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of idna to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of hyperlink to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of et-xmlfile to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of elasticsearch-dsl to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of elasticsearch to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of cssselect to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of cryptography to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of constantly to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of chardet to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of cffi to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of certifi to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of automat to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of to determine which version is compatible with other requirements. This could take a while. INFO: pip is looking at multiple versions of attrs to determine which version is compatible with other requirements. This could take a while. ERROR: Cannot install -r requirements.txt (line 12), -r requirements.txt (line 37) and idna==3.1 because these package versions have conflicting dependencies.

The conflict is caused by: The user requested idna==3.1 hyperlink 21.0.0 depends on idna>=2.5 requests 2.25.1 depends on idna<3 and >=2.5

To fix this you could try to:

  1. loosen the range of package versions you've specified
  2. remove package versions to allow pip attempt to solve the dependency conflict

ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/topics/dependency-resolution/#dealing-with-dependency-conflicts`

digitalengineering commented 2 years ago

Thanks for the additional information @Mcklmo :pray:

I see the issue now. However, I would like to keep the version constraints in place. I will close this PR and push a commit to master to simplify and fix the requirements.txt file using pipreqs. I believe this will solve the issue for you also. Please let me know if you run into any further problems.