Split requirements.txt into production and dev requirements

[jzohrab]

Dev requirements shouldn't be included in the docker image, adds space.

The pyproject.toml currently lists production requirements, that should suffice for prod builds.

• [x] figure out the correct split of requirements
• [x] create new requirements-dev.txt file with dev-only requirements
• [x] remove dev-only reqs from requirements.txt
• [x] add optional reqs to pyproject.toml???
• [x] update dev wiki, devs need all
• [x] ensure CI works -- add dev reqs for CI runs
• [x] ensure docker build works
• [x] note that the flit build will work, since it uses the pyproject.toml

[jzohrab]

I started this work in the branch wip_split_dependencies, but am stopping for now:

• I don't have a good way to manage adding new dependencies. pip freeze > requirements.txt would include everything, so dev has to be careful not to commit extra stuff to prod
• this only affects docker builds, really, and those only happen during launches. It would be nice to not download heavy dependencies like selenium, but is it a dealbreaker?

[jzohrab]

Should just split out the pytest and selenium/splinter stuff for now, I think those are fairly heavy.

pytest-splinter added in commit e7ebc0960

git log -p e7ebc0960 -n 1 -- requirements.txt

commit e7ebc0960e8643b9f3d1c734d2e6610a9db24109
...

    Add pytest-splinter.

diff --git a/requirements.txt b/requirements.txt
index c1ed9342..2899ab87 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,4 +1,5 @@
 astroid==2.15.6
+attrs==23.1.0
 beautifulsoup4==4.12.2
 blinker==1.6.2
 certifi==2023.7.22
@@ -10,6 +11,7 @@ dill==0.3.7
 Flask==2.3.3
 Flask-SQLAlchemy==3.1.1
 Flask-WTF==1.2.1
+h11==0.14.0
 idna==3.4
 iniconfig==2.0.0
 invoke==2.2.0
@@ -21,6 +23,7 @@ Mako==1.2.4
 MarkupSafe==2.1.3
 mccabe==0.7.0
 natto-py==1.0.1
+outcome==1.3.0.post0
 packaging==23.1
 parse==1.19.1
 parse-type==0.6.2
@@ -28,17 +31,26 @@ platformdirs==3.10.0
 pluggy==1.3.0
 pycparser==2.21
 pylint==2.17.5
+PySocks==1.7.1
 pytest==7.4.2
 pytest-bdd==7.0.0
+pytest-splinter==3.3.2
 PyYAML==6.0.1
 requests==2.31.0
+selenium==4.14.0
 six==1.16.0
+sniffio==1.3.0
+sortedcontainers==2.4.0
 soupsieve==2.5
+splinter==0.19.0
 SQLAlchemy==2.0.21
 tomlkit==0.12.1
+trio==0.22.2
+trio-websocket==0.11.1
 typing_extensions==4.8.0
-urllib3==2.0.7
+urllib3==1.26.18
 waitress==2.1.2
 Werkzeug==2.3.7
 wrapt==1.15.0
+wsproto==1.2.0
 WTForms==3.0.1

pytest stuff added at the start in 748dc339e

$ git log -p 748dc339 -- requirements.txt
...
    Initial commit, demo test.

diff --git a/requirements.txt b/requirements.txt
new file mode 100644
index 00000000..92a18d5f
--- /dev/null
+++ b/requirements.txt
@@ -0,0 +1,14 @@
+astroid==2.15.6
+coverage==7.3.1
+dill==0.3.7
+iniconfig==2.0.0
+isort==5.12.0
+lazy-object-proxy==1.9.0
+mccabe==0.7.0
+packaging==23.1
+platformdirs==3.10.0
+pluggy==1.3.0
+pylint==2.17.5
+pytest==7.4.2
+tomlkit==0.12.1
+wrapt==1.15.0

[jzohrab]

Done split. Have to document management.

Docker image sizes shrunk by 200 mb:

# pre split
lute3-lean   latest               2ebf3abc013b   4 days ago      462MB
lute3        latest               5e3b1b50b46a   4 days ago      1.07GB

# post split
lute3-lean   latest               ac52feff0ea5   8 seconds ago    240MB
lute3        latest               a85c24ae1901   19 seconds ago   851MB

Worth it just for reduced image build times.

Updating todo items.

[jzohrab]

Launched in 3.5.6.