Make some module dependencies optional

[resokou]

Consider making dependencies that are not stricly required as optional. In particular lmxl and ujson. You can test if the modules are installed on the user system with a simple try/except and fallback to the traditional modules if they are not available:

try:
    import lxml as unused_lxml  # noqa
    html_parser = 'lxml'
except ImportError:
    html_parser = 'html.parser'

# ... code

string = BeautifulSoup(string, html_parser).get_text()

and

try:
    import ujson
except ImportError:
    import json as ujson

Then remove them from requirements.txt. If you think it might be worth it for the end user to install those packages, create a new .txt with the optional modules instead, i.e. requirements-optional.txt with content:

lxml
ujson

and mention it in the README.

[stranger-danger-zamu]

Use pyproject.toml instead of separate requirements.txt.

[DIGITALCRIMINAL]

I'll look into pyproject.toml optional dependencies.

[stranger-danger-zamu]

[tool.poetry]
name = "ultimascraper"
version = "7.7.0"
description = ""
authors = ["DIGITALCRIMINAL <89371864+DIGITALCRIMINALS@users.noreply.github.com>"]
license = "GPL-3.0"

[tool.poetry.dependencies]
python = "^3.10.1"
requests = "^2.26.0"
beautifulsoup4 = "^4.10.0"
win32-setctime = "^1.0.4"
python-socks = {extras = ["asyncio"], version = "^2.0.1"}
python-dateutil = "^2.8.2"
lxml = { version = "^4.7.1", optional = true }
mergedeep = "^1.3.4"
jsonpickle = "^2.0.0"
ujson = { version = "^5.1.0", optional = true }
SQLAlchemy = "^1.4.20"
alembic = "^1.7.5"
tqdm = "^4.62.3"
selenium = "^4.1.0"
selenium-wire = "2.1.2"
user-agent = "^0.1.10"
aiohttp = "^3.8.1"
aiohttp-socks = "^0.7.1"
rich = "^10.16.1"

[tool.poetry.dev-dependencies]

[tool.poetry.extras]
performance = ["ujson","lxml"]

[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

Changing the pyproject.toml to something like this and changing imports to the try-ImportError blocks (like those mentioned by resokou) will allow for optional dependencies.

[DIGITALCRIMINAL]

Thanks, I added in latest commit.

https://github.com/DIGITALCRIMINALS/OnlyFans/commit/f3383b52b201602e9a54cb672dc8adf6079587d8