Use uv to manage dependencies

fyliu commented 11 months ago

Overview

We should convert our project to use poetry so we can use it to easily upgrade packages when the github bot tells us there's new vulnerabilities in old packages.

Action Items

[x] Create requirements.in and add all our dependencies
[x] Generate requirements.txt using uv
[x] Write script to make the update process easier
[x] Document how to update package dependencies

After merging

[ ] Release #183 from ice box

Resources/Instructions

See docker-mkdocs code for ideas.
If for some reason Poetry does not work, we could try using Hatch. See Fang's writeup about it
Hatch writeup

I was exploring using hatch over poetry because it supports scripts and environments, among many other things. The advantage is we can move the run scripts into the toml file rather than have them as separate files. Also, we could have dev and docs environments where we can install different sets of packages and python versions if we wanted.

But I found that it’s missing a piece that I thought was important and that poetry has, which is the ability to upgrade packages to the latest version. That saves us time so we don’t have to look through pypi ourselves.

It’s a young project with great ideas and the maintainer seems willing to add missing features including this one. I think we should revisit it later and consider using it.

Many popular and progressive python projects already switched to it. A major draw of hatch is it support matrix builds with different python version combinations that a package wants to support. It's a feature that used to require running a separate tool called tox.

fyliu commented 10 months ago

Updating milestone to security since this will make it straightforward to update dependency versions and resolve conflicts with other packages automatically when there's a fixable security advisory.

The simplest thing would be to do a poetry update to use the latest everything if it doesn't break the code.

fyliu commented 9 months ago

I was exploring using hatch over poetry because it supports scripts and environments, among many other things. The advantage is we can move the run scripts into the toml file rather than have them as separate files. Also, we could have dev and docs environments where we can install different sets of packages and python versions if we wanted.

But I found that it’s missing a piece that I thought was important and that poetry has, which is the ability to upgrade packages to the latest version. That saves us time so we don’t have to look through pypi ourselves.

It’s a young project with great ideas and the maintainer seems willing to add missing features including this one. I think we should revisit it later and consider using it.

Many popular and progressive python projects already switched to it. A major draw of hatch is it support matrix builds with different python version combinations that a package wants to support. It's a feature that used to require running a separate tool called tox.

fyliu commented 4 months ago

Progress:
- I set up pyproject.toml and am able to build the project with poetry and with pip inside the poetry virtual environment. Building with pip within a running docker compose session also works.
Blockers:
- Automated build with pip within Dockerfile via docker-compose up does not work because of package directory naming requirements. It wants a sibling directory to pyproject.toml to be named peopledepot, which contains all the source code. Maybe we need to COPY the source code in the Dockerfile rather than just mapping it in docker-compose. Mapping is preferred since it would be the latest WIP code.
Availability:
- Mon-Thurs 2-4pm
ETA:
- Next week
Problem output:

click to expand

❯ ./scripts/buildrun.sh                                                                                                                                                                                                                        
++ dirname ./scripts/buildrun.sh                                                                                                                                                                                                               
+ SCRIPT_DIR=./scripts                                                                                                                                                                                                                         
+ ./scripts/run.sh -c -o -d -b -m                                                                                                                                                                                                              
+ docker-compose down --remove-orphans                                                                                                                                                                                                         
Removing network peopledepot_default                                                                                                                                                                                                           
+ set +x                                                                                                                                                                                                                                       
+ docker-compose up -d --build                                                                                                                                                                                                                 
Creating network "peopledepot_default" with the default driver                                                                                                                                                                                 
Building web                                                                                                                                                                                                                                   
[+] Building 59.9s (10/13)                                                                              docker:default                                                                                                                         
 => [internal] load build definition from Dockerfile                                                              0.0s                                                                                                                         
 => => transferring dockerfile: 873B                                                                              0.0s                                                                                                                         
 => [internal] load .dockerignore                                                                                 0.0s                                                                                                                         
 => => transferring context: 100B                                                                                 0.0s                                                                                                                         
 => [internal] load metadata for docker.io/library/python:3.10-bullseye                                           2.9s                                                                                                                         
 => [internal] load build context                                                                                 0.0s                                                                                                                         
 => => transferring context: 410.11kB                                                                             0.0s                                                                                                                         
 => [1/9] FROM docker.io/library/python:3.10-bullseye@sha256:02c7cb92b8f23908de6457f7800c93b84ed8c6e7201da793544  0.0s                                                                                                                         
 => CACHED [2/9] WORKDIR /usr/src/app                                                                             0.0s                                                                                                                         
 => CACHED [3/9] RUN apt-get update   && apt-get --no-install-recommends install -y   netcat=1.10-46   gcc=4:10.  0.0s                                                                                                                         
 => CACHED [4/9] RUN python -m pip install --no-cache-dir pip~=24.0                                               0.0s                                                                                                                         
 => CACHED [5/9] COPY pyproject.toml .                                                                            0.0s                                                                                                                         
 => ERROR [6/9] RUN python -m pip install --no-cache-dir .                                                       56.9s                                                                                                                         
------                                                                                                                                                                                                                                         
 > [6/9] RUN python -m pip install --no-cache-dir .:                                                                                                                                                                                           
1.153 Processing /usr/src/app                                                                                                                                                                                                                  
1.155   Installing build dependencies: started                                                                                                                                                                                                 
10.927   Installing build dependencies: finished with status 'done'                                                                                                                                                                             
7.928   Getting requirements to build wheel: started                                                                                                                                                                                           
11.193   Getting requirements to build wheel: finished with status 'done'                                                                                                                                                                       
8.195   Preparing metadata (pyproject.toml): started                                                                                                                                                                                           
8.803   Preparing metadata (pyproject.toml): finished with status 'done'                                                                                                                                                                       
12.48 Collecting django<4.1.0,>=4.0.1 (from peopledepot==0.1.0)                                                                                                                                                                                
13.53   Downloading Django-4.0.10-py3-none-any.whl (8.0 MB)                                                                                                                                                                                    
16.14      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 8.0/8.0 MB 1.7 MB/s eta 0:00:00                                                                                                                                                            
16.70 Collecting django-extensions<3.2.0,>=3.1.5 (from peopledepot==0.1.0)                                                                                                                                                                     
16.94   Downloading django_extensions-3.1.5-py3-none-any.whl (224 kB)                                                                                                                                                                          
16.99      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 224.2/224.2 kB 4.3 MB/s eta 0:00:00                                                                                                                                                         
17.37 Collecting django-linear-migrations<2.13.0,>=2.12.0 (from peopledepot==0.1.0)                                                                                                                                                            
17.62   Downloading django_linear_migrations-2.12.0-py3-none-any.whl.metadata (13 kB)                                                                                                                                                          
17.90 Collecting django-phonenumber-field<6.4.0,>=6.3.0 (from peopledepot==0.1.0)                                                                                                                                                              
18.19   Downloading django_phonenumber_field-6.3.0-py3-none-any.whl (65 kB)
18.19      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 65.6/65.6 kB 39.7 MB/s eta 0:00:00
18.54 Collecting django-timezone-field==5.0 (from peopledepot==0.1.0)
18.79   Downloading django_timezone_field-5.0-py3-none-any.whl (10 kB)
19.32 Collecting djangorestframework<3.14.0,>=3.13.1 (from peopledepot==0.1.0)
19.54   Downloading djangorestframework-3.13.1-py3-none-any.whl (958 kB)
19.82      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 958.3/958.3 kB 3.5 MB/s eta 0:00:00
20.33 Collecting drf-jwt<1.20.0,>=1.19.2 (from peopledepot==0.1.0)
20.66   Downloading drf_jwt-1.19.2-py2.py3-none-any.whl (21 kB)
21.12 Collecting drf-spectacular==0.22.1 (from peopledepot==0.1.0)
21.37   Downloading drf_spectacular-0.22.1-py3-none-any.whl (84 kB)
21.38      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 84.2/84.2 kB 49.2 MB/s eta 0:00:00
21.82 Collecting markdown<3.5.0,>=3.4.1 (from peopledepot==0.1.0)
22.13   Downloading Markdown-3.4.4-py3-none-any.whl.metadata (6.9 kB)
22.79 Collecting psycopg2-binary<2.10.0,>=2.9.3 (from peopledepot==0.1.0)
23.22   Downloading psycopg2_binary-2.9.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.4 kB)
23.62 Collecting pydot<1.5.0,>=1.4.2 (from peopledepot==0.1.0)
23.95   Downloading pydot-1.4.2-py2.py3-none-any.whl (21 kB)
24.39 Collecting pytest-cov<3.1.0,>=3.0.0 (from peopledepot==0.1.0)
24.75   Downloading pytest_cov-3.0.0-py3-none-any.whl (20 kB)
25.19 Collecting pytest-django<4.6.0,>=4.5.2 (from peopledepot==0.1.0)
25.40   Downloading pytest_django-4.5.2-py3-none-any.whl (20 kB)
26.07 Collecting pytz (from django-timezone-field==5.0->peopledepot==0.1.0)
26.43   Downloading pytz-2024.1-py2.py3-none-any.whl.metadata (22 kB)
26.85 Collecting uritemplate>=2.0.0 (from drf-spectacular==0.22.1->peopledepot==0.1.0)
27.10   Downloading uritemplate-4.1.1-py2.py3-none-any.whl (10 kB)
27.53 Collecting PyYAML>=5.1 (from drf-spectacular==0.22.1->peopledepot==0.1.0)
27.91   Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (2.1 kB)
28.31 Collecting jsonschema>=2.6.0 (from drf-spectacular==0.22.1->peopledepot==0.1.0)
28.57   Downloading jsonschema-4.21.1-py3-none-any.whl.metadata (7.8 kB)
28.90 Collecting inflection>=0.3.1 (from drf-spectacular==0.22.1->peopledepot==0.1.0)
29.27   Downloading inflection-0.5.1-py2.py3-none-any.whl (9.5 kB)
29.74 Collecting asgiref<4,>=3.4.1 (from django<4.1.0,>=4.0.1->peopledepot==0.1.0)
29.94   Downloading asgiref-3.7.2-py3-none-any.whl.metadata (9.2 kB)
30.36 Collecting sqlparse>=0.2.2 (from django<4.1.0,>=4.0.1->peopledepot==0.1.0)
30.69   Downloading sqlparse-0.4.4-py3-none-any.whl (41 kB) 
30.69      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 41.2/41.2 kB 36.5 MB/s eta 0:00:00
31.08 Collecting PyJWT<3.0.0,>=1.5.2 (from PyJWT[crypto]<3.0.0,>=1.5.2->drf-jwt<1.20.0,>=1.19.2->peopledepot==0.1.0)
31.36   Downloading PyJWT-2.8.0-py3-none-any.whl.metadata (4.2 kB)
31.80 Collecting pyparsing>=2.1.4 (from pydot<1.5.0,>=1.4.2->peopledepot==0.1.0)
32.15   Downloading pyparsing-3.1.1-py3-none-any.whl.metadata (5.1 kB)
32.72 Collecting pytest>=4.6 (from pytest-cov<3.1.0,>=3.0.0->peopledepot==0.1.0)
33.00   Downloading pytest-8.0.0-py3-none-any.whl.metadata (7.8 kB)
34.69 Collecting coverage>=5.2.1 (from coverage[toml]>=5.2.1->pytest-cov<3.1.0,>=3.0.0->peopledepot==0.1.0)
34.94   Downloading coverage-7.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (8.2 kB)                                                                                         
35.23 Collecting typing-extensions>=4 (from asgiref<4,>=3.4.1->django<4.1.0,>=4.0.1->peopledepot==0.1.0)
35.61   Downloading typing_extensions-4.9.0-py3-none-any.whl.metadata (3.0 kB)
35.98 Collecting tomli (from coverage[toml]>=5.2.1->pytest-cov<3.1.0,>=3.0.0->peopledepot==0.1.0)
36.19   Downloading tomli-2.0.1-py3-none-any.whl (12 kB)
36.50 Collecting attrs>=22.2.0 (from jsonschema>=2.6.0->drf-spectacular==0.22.1->peopledepot==0.1.0)
36.75   Downloading attrs-23.2.0-py3-none-any.whl.metadata (9.5 kB)
37.02 Collecting jsonschema-specifications>=2023.03.6 (from jsonschema>=2.6.0->drf-spectacular==0.22.1->peopledepot==0.1.0)
37.32   Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl.metadata (3.0 kB)
37.75 Collecting referencing>=0.28.4 (from jsonschema>=2.6.0->drf-spectacular==0.22.1->peopledepot==0.1.0)
38.00   Downloading referencing-0.33.0-py3-none-any.whl.metadata (2.7 kB)
39.28 Collecting rpds-py>=0.7.1 (from jsonschema>=2.6.0->drf-spectacular==0.22.1->peopledepot==0.1.0)
39.60   Downloading rpds_py-0.18.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (4.1 kB)
40.74 Collecting cryptography>=3.4.0 (from PyJWT[crypto]<3.0.0,>=1.5.2->drf-jwt<1.20.0,>=1.19.2->peopledepot==0.1.0)
40.96   Downloading cryptography-42.0.2-cp39-abi3-manylinux_2_28_x86_64.whl.metadata (5.3 kB)
41.23 Collecting iniconfig (from pytest>=4.6->pytest-cov<3.1.0,>=3.0.0->peopledepot==0.1.0)
41.53   Downloading iniconfig-2.0.0-py3-none-any.whl (5.9 kB)
41.86 Collecting packaging (from pytest>=4.6->pytest-cov<3.1.0,>=3.0.0->peopledepot==0.1.0)
42.04   Downloading packaging-23.2-py3-none-any.whl.metadata (3.2 kB)
42.46 Collecting pluggy<2.0,>=1.3.0 (from pytest>=4.6->pytest-cov<3.1.0,>=3.0.0->peopledepot==0.1.0)
42.77   Downloading pluggy-1.4.0-py3-none-any.whl.metadata (4.3 kB)
43.12 Collecting exceptiongroup>=1.0.0rc8 (from pytest>=4.6->pytest-cov<3.1.0,>=3.0.0->peopledepot==0.1.0)
43.50   Downloading exceptiongroup-1.2.0-py3-none-any.whl.metadata (6.6 kB)
44.38 Collecting cffi>=1.12 (from cryptography>=3.4.0->PyJWT[crypto]<3.0.0,>=1.5.2->drf-jwt<1.20.0,>=1.19.2->peopledepot==0.1.0)
44.68   Downloading cffi-1.16.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (1.5 kB)
45.12 Collecting pycparser (from cffi>=1.12->cryptography>=3.4.0->PyJWT[crypto]<3.0.0,>=1.5.2->drf-jwt<1.20.0,>=1.19.2->peopledepot==0.1.0)
45.54   Downloading pycparser-2.21-py2.py3-none-any.whl (118 kB)
45.57      ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 118.7/118.7 kB 4.0 MB/s eta 0:00:00
45.83 Downloading django_linear_migrations-2.12.0-py3-none-any.whl (19 kB)
46.24 Downloading Markdown-3.4.4-py3-none-any.whl (94 kB)
46.25    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 94.2/94.2 kB 21.1 MB/s eta 0:00:00
46.59 Downloading psycopg2_binary-2.9.9-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)
47.46    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 3.0/3.0 MB 3.5 MB/s eta 0:00:00
47.85 Downloading asgiref-3.7.2-py3-none-any.whl (24 kB)
48.15 Downloading coverage-7.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (233 kB)
48.18    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 234.0/234.0 kB 8.1 MB/s eta 0:00:00
48.55 Downloading jsonschema-4.21.1-py3-none-any.whl (85 kB)
48.57    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 85.5/85.5 kB 4.6 MB/s eta 0:00:00
48.91 Downloading PyJWT-2.8.0-py3-none-any.whl (22 kB)
49.22 Downloading pyparsing-3.1.1-py3-none-any.whl (103 kB) 
49.25    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 103.1/103.1 kB 4.8 MB/s eta 0:00:00
49.66 Downloading pytest-8.0.0-py3-none-any.whl (334 kB)
49.73    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 334.0/334.0 kB 4.7 MB/s eta 0:00:00
49.94 Downloading PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)
50.07    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 705.5/705.5 kB 5.4 MB/s eta 0:00:00
50.31 Downloading pytz-2024.1-py2.py3-none-any.whl (505 kB)                                                                                                                                                                                    
50.45    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 505.5/505.5 kB 3.8 MB/s eta 0:00:00
50.73 Downloading attrs-23.2.0-py3-none-any.whl (60 kB)
50.74    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 60.8/60.8 kB 45.4 MB/s eta 0:00:00
50.94 Downloading cryptography-42.0.2-cp39-abi3-manylinux_2_28_x86_64.whl (4.7 MB)
51.98    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 4.7/4.7 MB 4.5 MB/s eta 0:00:00
52.32 Downloading exceptiongroup-1.2.0-py3-none-any.whl (16 kB)
52.62 Downloading jsonschema_specifications-2023.12.1-py3-none-any.whl (18 kB)
52.97 Downloading pluggy-1.4.0-py3-none-any.whl (20 kB)
53.37 Downloading referencing-0.33.0-py3-none-any.whl (26 kB)
53.73 Downloading rpds_py-0.18.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
53.94    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.1/1.1 MB 5.3 MB/s eta 0:00:00
54.28 Downloading typing_extensions-4.9.0-py3-none-any.whl (32 kB)
54.52 Downloading packaging-23.2-py3-none-any.whl (53 kB)
54.57    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 53.0/53.0 kB 21.4 MB/s eta 0:00:00
54.79 Downloading cffi-1.16.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (443 kB)
54.88    ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 443.9/443.9 kB 5.0 MB/s eta 0:00:00
54.98 Building wheels for collected packages: peopledepot
54.98   Building wheel for peopledepot (pyproject.toml): started
55.45   Building wheel for peopledepot (pyproject.toml): finished with status 'error'
55.45   error: subprocess-exited-with-error
55.45                                                      
55.45   × Building wheel for peopledepot (pyproject.toml) did not run successfully.
55.45   │ exit code: 1                                     
55.45   ╰─> [24 lines of output]
55.45       Traceback (most recent call last):
55.45         File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
55.45           main()                                     
55.45         File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
55.45           json_out['return_val'] = hook(**hook_input['kwargs'])
55.45         File "/usr/local/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
55.45           return _build_backend().build_wheel(wheel_directory, config_settings,
55.45         File "/tmp/pip-build-env-ezxinv6k/overlay/lib/python3.10/site-packages/poetry/core/masonry/api.py", line 58, in build_wheel
55.45           return WheelBuilder.make_in(
55.45         File "/tmp/pip-build-env-ezxinv6k/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 88, in make_in
55.45           wb.build(target_dir=directory)
55.45         File "/tmp/pip-build-env-ezxinv6k/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 124, in build
55.45           self._copy_module(zip_file)
55.45         File "/tmp/pip-build-env-ezxinv6k/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/wheel.py", line 265, in _copy_module
55.45           to_add = self.find_files_to_add()
55.45         File "/tmp/pip-build-env-ezxinv6k/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/builder.py", line 171, in find_files_to_add
55.45           for include in self._module.includes:
55.45         File "/usr/local/lib/python3.10/functools.py", line 981, in __get__
55.45           val = self.func(instance)
55.45         File "/tmp/pip-build-env-ezxinv6k/overlay/lib/python3.10/site-packages/poetry/core/masonry/builders/builder.py", line 97, in _module
55.45           return Module(
55.45         File "/tmp/pip-build-env-ezxinv6k/overlay/lib/python3.10/site-packages/poetry/core/masonry/utils/module.py", line 65, in __init__
55.45           raise ModuleOrPackageNotFound(
55.45       poetry.core.masonry.utils.module.ModuleOrPackageNotFound: No file/folder found for package peopledepot
55.45       [end of output]                                
55.45                                                      
55.45   note: This error originates from a subprocess, and is likely not a problem with pip.
55.46   ERROR: Failed building wheel for peopledepot
55.46 Failed to build peopledepot
55.46 ERROR: Could not build wheels for peopledepot, which is required to install pyproject.toml-based projects
------                                                     
Dockerfile:25                                              
--------------------                                       
  23 |     # pyproject.toml needs to be in subdirectory
  24 |     COPY pyproject.toml .
  25 | >>> RUN python -m pip install --no-cache-dir .
  26 |                                                     
  27 |     # copy entrypoint.sh
--------------------                                       
ERROR: failed to solve: process "/bin/sh -c python -m pip install --no-cache-dir ." did not complete successfully: exit code: 1
ERROR: Service 'web' failed to build : Build failed

fyliu commented 4 months ago

I found that other projects tend to use poetry.core rather than poetry. It makes the configuration pip-compatible. So we can skip the step to generate the requirements.txt and have pip read from pyproject.toml directly.

The problem is that pip wants to build peopledepot as a package, which has some requirements we don't meet, since package building is not our goal.

Potential solutions:

Take steps to satisfy the packaging requirements.
Find a way to not build the package. There's a flag mentioned, but I'm not able to make it work. Maybe I'm not using it correctly.
Learn from other projects using poetry.core to see how they organize their project.

fyliu commented 4 months ago

Update:

Looks like pip cannot install the python dependencies from just the pyproject.toml file without also packaging the project. Since we're running the code as an application and not trying to create a python package, we don't want to have the code packaged. So running pip on pyproject.toml is a NO GO.

Our current options are:

Run poetry export to create a requirement file for pip
Don't use poetry and opt for something like pip-tools to manage dependencies
Another option is uv, which is a newer tool that's supposed to be a drop-in replacement for pip, pip-tools, and does the job of venv
hatch is not being considered for reasons below

Quick comparison:

poetry
- manages venv for non-docker (not our main usage but nice to have)
- exports requirements for pip use in Dockerfile
- can sync venv and requirements via poetry lock
- can update requirements via poetry update
- must install poetry, just using pip "should work" but doesn't really do what we want
pip-tools
- can sync venv and requirements via pip-sync
- can generate requirement file via pip-compile
- can update requirements via pip-compile --upgrade
- need to install pip-tools, which is smaller than poetry I think, since it does less
- does not manage venv
uv
- can do what pip-tools does plus manages venv
- drop-in replacement for pip, pip-tools, meaning we can go back to those tools if uv becomes unusable for some reason
- much faster
- not dependent on python version, since it's written in rust
- backed by committed company that also maintains ruff linter/formatter
  - ruff is well-received by projects in the community, so I'd say the company has a good track record for creating useful and fast projects
- ambitious but realistic roadmap to expand capability to surpass poetry
  - they took over maintenance of a similar tool and will slowly migrate its features into uv
hatch
- comparable to poetry as a build tool and has nice features such as matrix builds and scripting
- doesn't have ability to upgrade dependencies, a deal-breaker for the current issue
- can be used in conjunction with pip-tools and uv to provide venv and scripting abilities, which is nice to have but not relevant to this issue

Recommendation:

Given the main purpose of this issue is to have our dependencies managed, any of the 3 options should work fine for us.
I like the simplicity of pip-tools
poetry and uv being able to manage venv is a plus
the speed improvement that uv promises sounds really good
Since we're still a young project, I'd say we can choose a shiny and promising tool like uv, and hope it'll mature as we do.

fyliu commented 4 months ago

After trying out pip-tools and then uv, I can confirm that uv is much faster. a pip-compile --upgrade operation that takes 50 seconds took uv 10 seconds.

The only thing that concerns me is uv not supporting hash checking packages. They do plan to add support.

We don't currently do anything with hashes. To add it to our workflow, add --generate-hashes to the pip-compile command and --require-hashes to the pip install command. uv pip compile will also generate hashes, uv pip install just won't verify them right now. We can just use pip install in any CI/CD workflows if we decode to use hashes to improve security.

fyliu commented 4 months ago

I converted the issue to one that uses uv, since it's better suited to what we want to do. Poetry is more for creating packages, which we aren't doing.

All the tasks are done in my fork. I need to clean up the commits before doing the PR.

hackforla / peopledepot