search

dimitryzub / scrape-google-scholar-py

Extract data from all Google Scholar pages from a single Python module. NOTE: I'm no longer maintaining this repo. Chrome driver/selectors might need and update.
MIT License
82 stars 16 forks source link

Wheels failed to build while pip installing #12

Closed rohanjha123 closed 1 year ago

rohanjha123 commented 1 year ago

Hi,

I'm using macOS Big Sur, python 3.11.1. Whenever I try to pip install scrape-google-scholar-py, all the requirements are satisfied but the wheels fail to build. I'm not sure what's happening, I've attached the error message below. I'd really appreciate any help, thanks!

Collecting scrape-google-scholar-py
  Using cached scrape-google-scholar-py-0.3.4.tar.gz (37 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting google-search-results>=2.4.2 (from scrape-google-scholar-py)
  Using cached google_search_results-2.4.2-py3-none-any.whl
Collecting selectolax>=0.3.12 (from scrape-google-scholar-py)
  Using cached selectolax-0.3.13-cp311-cp311-macosx_10_9_universal2.whl (4.2 MB)
Collecting parsel>=1.7.0 (from scrape-google-scholar-py)
  Using cached parsel-1.8.1-py2.py3-none-any.whl (17 kB)
Collecting selenium-stealth>=1.0.6 (from scrape-google-scholar-py)
  Using cached selenium_stealth-1.0.6-py3-none-any.whl (32 kB)
Requirement already satisfied: pandas>=1.5.3 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from scrape-google-scholar-py) (1.5.3)
Requirement already satisfied: webdriver-manager>=3.8.5 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from scrape-google-scholar-py) (3.8.6)
Requirement already satisfied: requests in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from google-search-results>=2.4.2->scrape-google-scholar-py) (2.28.1)
Requirement already satisfied: python-dateutil>=2.8.1 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from pandas>=1.5.3->scrape-google-scholar-py) (2.8.2)
Requirement already satisfied: pytz>=2020.1 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from pandas>=1.5.3->scrape-google-scholar-py) (2022.7.1)
Requirement already satisfied: numpy>=1.21.0 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from pandas>=1.5.3->scrape-google-scholar-py) (1.24.3)
Collecting cssselect>=0.9 (from parsel>=1.7.0->scrape-google-scholar-py)
  Using cached cssselect-1.2.0-py2.py3-none-any.whl (18 kB)
Collecting jmespath (from parsel>=1.7.0->scrape-google-scholar-py)
  Using cached jmespath-1.0.1-py3-none-any.whl (20 kB)
Requirement already satisfied: lxml in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from parsel>=1.7.0->scrape-google-scholar-py) (4.9.2)
Requirement already satisfied: packaging in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from parsel>=1.7.0->scrape-google-scholar-py) (22.0)
Collecting w3lib>=1.19.0 (from parsel>=1.7.0->scrape-google-scholar-py)
  Using cached w3lib-2.1.1-py3-none-any.whl (21 kB)
Requirement already satisfied: Cython>=0.29.23 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from selectolax>=0.3.12->scrape-google-scholar-py) (0.29.34)
Requirement already satisfied: selenium in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from selenium-stealth>=1.0.6->scrape-google-scholar-py) (4.9.1)
Requirement already satisfied: python-dotenv in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from webdriver-manager>=3.8.5->scrape-google-scholar-py) (1.0.0)
Requirement already satisfied: tqdm in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from webdriver-manager>=3.8.5->scrape-google-scholar-py) (4.65.0)
Requirement already satisfied: six>=1.5 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from python-dateutil>=2.8.1->pandas>=1.5.3->scrape-google-scholar-py) (1.16.0)
Requirement already satisfied: charset-normalizer<3,>=2 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from requests->google-search-results>=2.4.2->scrape-google-scholar-py) (2.1.1)
Requirement already satisfied: idna<4,>=2.5 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from requests->google-search-results>=2.4.2->scrape-google-scholar-py) (3.4)
Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from requests->google-search-results>=2.4.2->scrape-google-scholar-py) (1.26.13)
Requirement already satisfied: certifi>=2017.4.17 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from requests->google-search-results>=2.4.2->scrape-google-scholar-py) (2022.12.7)
Requirement already satisfied: trio~=0.17 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (0.22.0)
Requirement already satisfied: trio-websocket~=0.9 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (0.10.2)
Requirement already satisfied: attrs>=19.2.0 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from trio~=0.17->selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (22.2.0)
Requirement already satisfied: sortedcontainers in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from trio~=0.17->selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (2.4.0)
Requirement already satisfied: async-generator>=1.9 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from trio~=0.17->selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (1.10)
Requirement already satisfied: outcome in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from trio~=0.17->selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (1.2.0)
Requirement already satisfied: sniffio in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from trio~=0.17->selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (1.3.0)
Requirement already satisfied: exceptiongroup in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from trio-websocket~=0.9->selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (1.1.1)
Requirement already satisfied: wsproto>=0.14 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from trio-websocket~=0.9->selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (1.2.0)
Requirement already satisfied: PySocks!=1.5.7,<2.0,>=1.5.6 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from urllib3<1.27,>=1.21.1->requests->google-search-results>=2.4.2->scrape-google-scholar-py) (1.7.1)
Requirement already satisfied: h11<1,>=0.9.0 in /Users/rohanjha/.pyenv/versions/3.11.1/lib/python3.11/site-packages (from wsproto>=0.14->trio-websocket~=0.9->selenium->selenium-stealth>=1.0.6->scrape-google-scholar-py) (0.14.0)
Building wheels for collected packages: scrape-google-scholar-py
  WARNING: Building wheel for scrape-google-scholar-py failed: [Errno 13] Permission denied: '/Users/rohanjha/Library/Caches/pip/wheels/52/42'
Failed to build scrape-google-scholar-py
ERROR: Could not build wheels for scrape-google-scholar-py, which is required to install pyproject.toml-based projects
rohanjha123 commented 1 year ago

Ah, I was able to figure it out by directly editing my wheels folder, apologies for wasting your time!

dimitryzub commented 1 year ago

Ah, I was able to figure it out by directly editing my wheels folder, apologies for wasting your time!

Hi @rohanjha123, thank you for opening this issue! Could you explain in more detail how you fixed it? I don't have access to Mac and would like to add your solution to README for Mac users and tag you as a co-author in the commit.

What do you think? 🙂

rohanjha123 commented 1 year ago

Thanks for reaching out @dimitryzub! The issue comes from the following line: Permission denied: '/Users/rohanjha/Library/Caches/pip/wheels/52/42'. In my Mac, there was a directory in /Users/rohanjha/Library/Caches/pip/wheels/52, but there was no /42 subdirectory. Furthermore, the package scrape-google-scholar-py did not have permission to create this subdirectory inside the wheels directory of my Mac. So, I manually went inside the wheels directory, changed the system permissions to allow me to edit the wheels directory and manually created the /42 subdirectory. The pip install worked after that!

dimitryzub commented 1 year ago

@rohanjha123 Thank you for your thoughts 🙂

Could you tell if you used virtual environment when installing package for the first time? In other words, did you installed it globally or locally to virtual env?

I guess the CLI way of solving your issue could be by using: 

sudo chmod -R a+w /Users/rohanjha/Library/Caches/pip/wheels

which will may grant write permission to all users for the specified directory and its contents.

So CLI commands could be like so:

$ python3.11 -m venv env # or just python3
$ source env/bin/activate
$ pip install scrape-google-scholar-py

# if error above:
$ sudo chmod -R a+w /Users/rohanjha/Library/Caches/pip/wheels
$ pip install scrape-google-scholar-py

Just wanted to share a few thoughts on that. Not really sure what might caused that issue.

rohanjha123 commented 1 year ago

I installed them globally! Thanks for the suggestions