ValueError when using BeautifulSoup in lab.py

[SPN11]

Experiencing a ValueError when I attempt to use the PittAPI to retrieve a dictionary from lab.py for computer lab availability. I've narrowed it down to a potential issue with BeautifulSoup in _fetch_labs(): soup = BeautifulSoup(page.text, 'lxml') I added a print statement before and after this line. Print statement before checks out, the one after never prints.

Steps to reproduce: -Activate the virtual environment, and run server.py -Make a request to the API for a lab's status.

It will reach the appropriate endpoint, find a match in locations, and attempt to fetch the lab info, but will encounter a ValueError when attempting to parse the span section of the page response's text.

[RitwikGupta]

Could you please paste the entire stack trace?

[RitwikGupta]

Also, please be aware that I cannot repro this error.

[Rahi374]

Same.

[azharichenko]

@SPN11 Can you give us more details on what the ValueError is telling you?

[SPN11]

I'm sorry for just responding to this. I'm no longer receiving the ValueError, but now I'm receiving the following error from the same line:

Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library? You might have seen it in the Slack.

I don't believe there is an issue with the WebWrapper. I am hosting and testing this on my MacBook Pro which might be interfering with the Python built-in HTML parser. I have the lxml package installed though and the virtual environment is enabled. No one else able to reproduce the ValueError, but is anyone able to reproduce the above message?

[azharichenko]

@SPN11 Yeah, I saw that in the Slack Chat. So this is exclusively just an issue with BeautifulSoup not being able to find the lxml package. I'm going to see if I can able reproduce this, and I'll give you my exact steps of how I setup it up.

[azharichenko]

I like @RitwikGupta and @Rahi374 wasn't able to reproduce that same error. Here is the log of the commands I used.

cd Documents/
git clone https://github.com/Pitt-CSC/PittAPIWebWrapper.git --recurse-submodules
cd PittAPIWebWrapper/
git submodule update --remote --merge
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt
pip install -r apiwrapper/PittAPI/requirements.txt
python server.py

I'm going to try rerunning this on a fresh installation of Ubuntu to see if I can reproduce the same error.

[azharichenko]

I went through rerunning this again on a new installation of Ubuntu with theses commands...

sudo add-apt-repository universe
sudo apt-get update
sudo apt-get install git
sudo apt-get install python3-pip

cd Documents/
git clone https://github.com/Pitt-CSC/PittAPIWebWrapper.git 
cd PittAPIWebWrapper/apiwrapper
git clone https://github.com/Pitt-CSC/PittAPI.git 
cd ..

pip3 install virtualenv --user
virtualenv venv
source venv/bin/activate
pip3 install -r requirements.txt
pip3 install -r apiwrapper/PittAPI/requirements.txt
python3 server.py

and still wasn't able to reproduce the same error. Currently, all I can think that could be messing you up is using commands that use Python 2.7 instead of Python 3.5. You may be using the 2.7 of pip which doesn't install all the packages properly for 3.5 and that might be messing it up. But, not quite sure if you are able to, can you provide all the commands used to perform the error and the version of pip and python you are using?

python --version
pip --version

---

Update: Just reproduced same error after uninstalling lxml.

{"error": "Couldn't find a tree builder with the features you requested: lxml. Do you need to install a parser library?"}

[RitwikGupta]

@azharichenko lxml needs to be installed, so uninstalling it will give you that error. The requirements file should install lxml without an issue.

@SPN11 Can you check that you're using Python 3 and not 2.7?

[azharichenko]

I just wanted to see if I could reproduce the error, hence why I uninstalled it. Otherwise doing pip install on the requirements from PittAPI installs it properly.

[SPN11]

I imagined it had something to do with lxml not being properly and BeautifulSoup not being able to find the package. Here is my log of commands: git submodule update --remote --merge No changes source bin/activate python --version 3.5 pip --version 9.0.1 pip install -r requirements.txt pip install -r apiwrapper/PittAPI/requirements.txt No changes anywhere pip uninstall lxml Uninstalls successfully pip install lxmlInstalls successfully (but is using cached data... would that be an issue?) python server.py I type into my address bar http://localhost:5000/lab_status/hillman and I still receive the same exact error.

If no one else is able to reproduce the error (except by uninstalling lxml), then it must be an issue on my end... I'm sorry if I'm wasting people's time with this.

[azharichenko]

Can you give me the full response that pip --version gives because at the end will tell me whether it's pip for Python 3 or 2

[SPN11]

I should have specified that it was for Python 3.5. Here's the full response: pip 9.0.1 from /Users/snigh16/Desktop/PittAPIWebWrapper/lib/python3.5/site-packages (python 3.5)

[azharichenko]

@SPN11 This looks like it's an issue with lxml, http://lxml.de/installation.html

[RitwikGupta]

Seems like you have to run STATIC_DEPS=true sudo pip install lxml on macOS

[SPN11]

I tried that too. I tried uninstalling it. I tried re-installing python, re-cloning the repository, re-installing the dependencies, and everything. I don't know why it doesn't recognize it. If no one else can reproduce the issue it must be me.

[SPN11]

In the end it all boiled down to testing on Mac OSX. I'm retrieving the data and all is working as intended. Sorry, everyone.

[RitwikGupta]

No worries! It was a learning experience for us all 😄