search

Opentrons / opentrons

Software for writing protocols and running them on the Opentrons Flex and Opentrons OT-2
https://opentrons.com
Apache License 2.0
415 stars 177 forks source link

bug: installing packages from PyPi is broken upon upgrading from v7.1.0 to v7.2.2 #15110

Open ahmadomira opened 4 months ago

ahmadomira commented 4 months ago

Overview

After updating our OT-2 robot's software from v7.1.0 to v7.2.2, installing a Python package (from outside Python's standard library) using pip doesn't work. The installed packages do exist in Python's site-packages directory, yet importing them inside the Python interpreter results a Module Not Found error.

Please note that this problem goes away when downgrading the software to v7.1.0.

Steps to reproduce

Note: the following problem can be reproduced by installing any PyPi package. The packages are installed following this Opentrons tutorial.

After completing the initial SSH-setup:

  1. connect the robot to our local network
  2. open a Powershell window, and do: 
    ssh -i ot2_ssh_key root@OUR_ROBOT_ID
  3. after successful login to the robot: 
    pip3 install openpyxl
  4. after a successful package installation, restart the robot.
  5. login again into the robot over SSH (repeat Step 1)
  6. doing, for example: 
    python3
>>> import openpyxl

Current behavior

Doing, for example, this:

python3
>>> import openpyxl

outputs: Module Not Found Error

or simply attempting to uninstall the same package, with no further steps in between (as shown in the screenshot below):

pip3 uninstall openpyxl

outputs: WARNING: Skipping openpyxl as it is not installed.

We have tried to install packages from PyPi into Python virtual environemts (venv), but the problem persists.

Problem Screenshot

image

The installed packages do exist in Python's system directory

Navigating to Python's site-packages directory, after package installation, shows that the packages exist within the Python directory. Doing in the robots terminal, for example: 

which python3

outputs: /usr/bin/python3

Now navigating to /usr/bin/python3/.../site-packages shows that the installed package (and its submodules) exists, as shown in these screenshots:

image image image

Expected behavior

Before upgrading the software, which we had to do because of other problem, we used to install our Python packages from within our Jupyter notebooks by doing:

!pip install openpyxl

then import openpyxl in next cells with no issues. Now, doing the same results Module Not Found errors

Operating system

Windows

System and robot setup or anything else?

aschleck commented 1 month ago

Experiencing this as well on an OT-3 at v7.3.1. Presumably a breakage caused on the OT-2 by https://github.com/Opentrons/buildroot/commit/4d580d8b4cff302d4d7cc4170d5c4d6ff77286fb / https://github.com/Opentrons/buildroot/pull/231 but I don't really understand the mechanism and I know the OT-3 is pulling from eg https://github.com/Opentrons/oe-core/commit/9eef29d03640d371186254d04136527d37261f2a .

Anyway you can set PYTHONPATH to make this work-ish. Opentrons should still fix this.

root@41523d:~# cd /home/root
root@41523d:~# python3 -m venv venv
root@41523d:~# source venv/bin/activate
(venv) root@41523d:~# pip install -r requirements.txt --verbose                                       
Using pip 22.3.1 from /home/root/venv/lib/python3.10/site-packages/pip (python 3.10)
Requirement already satisfied: opentrons==7.3.1 in /opt/opentrons-robot-server (from -r requirements.txt (line 1)) (7.3.1)
Collecting pandas==1.5.3
  Using cached pandas-1.5.3-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (11.4 MB)
Requirement already satisfied: opentrons-shared-data==7.3.1 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (7.3.1)
Requirement already satisfied: aionotify==0.2.0 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (0.2.0)
Requirement already satisfied: anyio<4.0.0,>=3.6.1 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (3.7.1)
Requirement already satisfied: jsonschema<4.18.0,>=3.0.1 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (4.17.3)
Collecting numpy<2,>=1.20.0
  Using cached numpy-1.26.4-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl (14.2 MB)
Requirement already satisfied: pydantic<2.0.0,>=1.10.9 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (1.10.12)
Requirement already satisfied: pyserial>=3.5 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (3.5)
Requirement already satisfied: typing-extensions<5,>=4.0.0 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (4.11.0)
Requirement already satisfied: click<9,>=8.0.0 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (8.1.2)
Requirement already satisfied: packaging>=21.0 in /opt/opentrons-robot-server (from opentrons==7.3.1->-r requirements.txt (line 1)) (24.0)
Collecting pytz>=2020.1
  Using cached pytz-2024.1-py2.py3-none-any.whl (505 kB)
Collecting python-dateutil>=2.8.1
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Requirement already satisfied: idna>=2.8 in /opt/opentrons-robot-server (from anyio<4.0.0,>=3.6.1->opentrons==7.3.1->-r requirements.txt (line 1)) (3.3)
Requirement already satisfied: sniffio>=1.1 in /opt/opentrons-robot-server (from anyio<4.0.0,>=3.6.1->opentrons==7.3.1->-r requirements.txt (line 1)) (1.3.1)
Requirement already satisfied: exceptiongroup in /opt/opentrons-robot-server (from anyio<4.0.0,>=3.6.1->opentrons==7.3.1->-r requirements.txt (line 1)) (1.2.0)
Requirement already satisfied: attrs>=17.4.0 in /opt/opentrons-robot-server (from jsonschema<4.18.0,>=3.0.1->opentrons==7.3.1->-r requirements.txt (line 1)) (23.2.0)
Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /opt/opentrons-robot-server (from jsonschema<4.18.0,>=3.0.1->opentrons==7.3.1->-r requirements.txt (line 1)) (0.20.0)
Collecting six>=1.5
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, six, numpy, python-dateutil, pandas
  Creating /var/user-packages/home/root/venv/bin
  changing mode of /var/user-packages/home/root/venv/bin/f2py to 755
  WARNING: The script f2py is installed in '/var/user-packages/home/root/venv/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed numpy-1.26.4 pandas-1.5.3 python-dateutil-2.9.0.post0 pytz-2024.1 six-1.16.0

[notice] A new release of pip available: 22.3.1 -> 24.1.2
[notice] To update, run: pip install --upgrade pip
(venv) root@41523d:~# ls venv/lib/python3.10/site-packages/
_distutils_hack  distutils-precedence.pth  pip  pip-22.3.1.dist-info  pkg_resources  setuptools  setuptools-65.5.0.dist-info
(venv) root@41523d:~# ls /var/user-packages/home/root/venv/lib/python3.10/site-packages/
__pycache__  dateutil  numpy  numpy-1.26.4.dist-info  numpy.libs  pandas  pandas-1.5.3.dist-info  python_dateutil-2.9.0.post0.dist-info  pytz  pytz-2024.1.dist-info  six-1.16.0.dist-info  six.py
(venv) root@41523d:~# PYTHONPATH=/var/user-packages/home/root/venv/lib/python3.10/site-packages/:/opt/opentrons-robot-server:/usr/lib/python3.10/site-packages python3          
Python 3.10.9 (main, Dec  6 2022, 18:44:57) [GCC 11.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas
>>> import opentrons
>>>
aschleck commented 5 days ago

Better solution: run the venv's pip with PIP_CONFIG_FILE=/dev/null like PIP_CONFIG_FILE=/dev/null pip install pandas.

The breakage on Flex is caused by https://github.com/Opentrons/oe-core/blob/2ad1ac910d393c84b7f363ae3cca23786d290935/layers/meta-opentrons/recipes-devtools/python3-pip/files/pip.conf#L1