search

edraj / dmart

Headless information management system
https://dmart.cc
GNU Affero General Public License v3.0
10 stars 7 forks source link

Reduce and organize dependencies #482

Closed kefahi closed 1 month ago

kefahi commented 2 months ago

Over the time the list of dependencies is growing out of control; currently we have the following (upon a fresh install)

Installing collected packages: webencodings, passlib, brotli, altgraph, zopfli, wrapt, websockets, uvloop, urllib3, uritemplate, ujson, typing-extensions, types-setuptools, types-pyasn1, types-psutil, types-aiofiles, tinycss2, sniffio, six, shellingham, setuptools, segno, ruff, ruamel.yaml.clib, rstr, rpds-py, regress, redis, pyzmq, pyyaml, python-multipart, python-dotenv, Pyphen, pypdf, pyparsing, pyjwt, pygments, pydyf, pycparser, pyasn1, psutil, protobuf, priority, portalocker, pluggy, pillow, packaging, orjson, oauthlib, nodeenv, mypy-extensions, multidict, msgpack, mdurl, MarkupSafe, jq, itsdangerous, iniconfig, idna, hyperframe, httptools, hpack, h11, grpcio, greenlet, google-crc32c, frozenlist, fonttools, duckdb, dnspython, dicttoxml, ConfigArgParse, click, charset-normalizer, certifi, cachetools, blinker, bcrypt, attrs, annotated-types, aiofiles, zope.interface, zope.event, yarl, wsproto, Werkzeug, uvicorn, types-pyinstaller, types-ldap3, smart-open, ruamel.yaml, rsa, requests, referencing, python-dateutil, pytest, pyright, pyinstaller-hooks-contrib, pydantic-core, pyasn1-modules, proto-plus, pdf2image, mypy, markdown-it-py, ldap3, jinja2, httplib2, httpcore, html5lib, h2, googleapis-common-protos, google-resumable-media, email-validator, cssselect2, concurrent-log-handler, cffi, anyio, aiosignal, weasyprint, watchfiles, types-jsonschema, starlette, rich, pytest-ordering, pytest-mock, pytest-asyncio, pyinstaller, pydantic, jsonschema-specifications, hypercorn, httpx, grpcio-status, google-auth, gevent, flask, faker, cryptography, cachecontrol, aiohttp, typer, pydantic-settings, jsonschema, google-auth-httplib2, google-api-core, geventhttpclient, Flask-Login, Flask-Cors, asgi-correlation-id, locust, jsf, google-cloud-core, google-api-python-client, fastapi-cli, check-jsonschema, google-cloud-storage, google-cloud-firestore, fastapi, firebase-admin, fastapi-sso Successfully installed ConfigArgParse-1.7 Flask-Cors-4.0.1 Flask-Login-0.6.3 MarkupSafe-2.1.5 Pyphen-0.15.0 Werkzeug-3.0.3 aiofiles-24.1.0 aiohttp-3.9.5 aiosignal-1.3.1 altgraph-0.17.4 annotated-types-0.7.0 anyio-4.4.0 asgi-correlation-id-4.3.1 attrs-23.2.0 bcrypt-4.1.3 blinker-1.8.2 brotli-1.1.0 cachecontrol-0.14.0 cachetools-5.3.3 certifi-2024.7.4 cffi-1.16.0 charset-normalizer-3.3.2 check-jsonschema-0.28.6 click-8.1.7 concurrent-log-handler-0.9.25 cryptography-42.0.8 cssselect2-0.7.0 dicttoxml-1.7.16 dnspython-2.6.1 duckdb-1.0.0 email-validator-2.2.0 faker-26.0.0 fastapi-0.111.0 fastapi-cli-0.0.4 fastapi-sso-0.15.0 firebase-admin-6.5.0 flask-3.0.3 fonttools-4.53.1 frozenlist-1.4.1 gevent-24.2.1 geventhttpclient-2.3.1 google-api-core-2.19.1 google-api-python-client-2.136.0 google-auth-2.31.0 google-auth-httplib2-0.2.0 google-cloud-core-2.4.1 google-cloud-firestore-2.16.1 google-cloud-storage-2.17.0 google-crc32c-1.5.0 google-resumable-media-2.7.1 googleapis-common-protos-1.63.2 greenlet-3.0.3 grpcio-1.64.1 grpcio-status-1.62.2 h11-0.14.0 h2-4.1.0 hpack-4.0.0 html5lib-1.1 httpcore-1.0.5 httplib2-0.22.0 httptools-0.6.1 httpx-0.27.0 hypercorn-0.17.3 hyperframe-6.0.1 idna-3.7 iniconfig-2.0.0 itsdangerous-2.2.0 jinja2-3.1.4 jq-1.7.0 jsf-0.11.2 jsonschema-4.22.0 jsonschema-specifications-2023.12.1 ldap3-2.9.1 locust-2.29.1 markdown-it-py-3.0.0 mdurl-0.1.2 msgpack-1.0.8 multidict-6.0.5 mypy-1.10.1 mypy-extensions-1.0.0 nodeenv-1.9.1 oauthlib-3.2.2 orjson-3.10.6 packaging-24.1 passlib-1.7.4 pdf2image-1.17.0 pillow-10.4.0 pluggy-1.5.0 portalocker-2.10.0 priority-2.0.0 proto-plus-1.24.0 protobuf-4.25.3 psutil-6.0.0 pyasn1-0.6.0 pyasn1-modules-0.4.0 pycparser-2.22 pydantic-2.8.2 pydantic-core-2.20.1 pydantic-settings-2.3.4 pydyf-0.10.0 pygments-2.18.0 pyinstaller-6.9.0 pyinstaller-hooks-contrib-2024.7 pyjwt-2.8.0 pyparsing-3.1.2 pypdf-4.2.0 pyright-1.1.370 pytest-8.2.2 pytest-asyncio-0.23.7 pytest-mock-3.14.0 pytest-ordering-0.6 python-dateutil-2.9.0.post0 python-dotenv-1.0.1 python-multipart-0.0.9 pyyaml-6.0.1 pyzmq-26.0.3 redis-5.0.7 referencing-0.35.1 regress-0.4.5 requests-2.32.3 rich-13.7.1 rpds-py-0.18.1 rsa-4.9 rstr-3.2.2 ruamel.yaml-0.18.6 ruamel.yaml.clib-0.2.8 ruff-0.5.1 segno-1.6.1 setuptools-70.2.0 shellingham-1.5.4 six-1.16.0 smart-open-7.0.4 sniffio-1.3.1 starlette-0.37.2 tinycss2-1.3.0 typer-0.12.3 types-aiofiles-24.1.0.20240626 types-jsonschema-4.22.0.20240610 types-ldap3-2.9.13.20240205 types-psutil-6.0.0.20240621 types-pyasn1-0.6.0.20240402 types-pyinstaller-6.8.0.20240626 types-setuptools-70.2.0.20240704 typing-extensions-4.12.2 ujson-5.10.0 uritemplate-4.1.1 urllib3-2.2.2 uvicorn-0.30.1 uvloop-0.19.0 watchfiles-0.22.0 weasyprint-62.3 webencodings-0.5.1 websockets-12.0 wrapt-1.16.0 wsproto-1.2.0 yarl-1.9.4 zope.event-5.0 zope.interface-6.4.post2 zopfli-0.2.3

Which consumes 540MBs.

We need to modularize those dependencies, so they need to be installed, only when the respective subfeatures / plugs are needed.

E.g. Data entry types (like duckdb) should be modularized (made optional upon configuration). Same for firebase and ldap ...etc.

This way, the admin who only needs the core dmart, can install minimum python packages and run pip install -r xxx-requirements.txt for any additional sub-features they want to enable.

kefahi commented 1 month ago

This was resolved in both the docker file and requirements