twintproject / twint

An advanced Twitter scraping & OSINT tool written in Python that doesn't use Twitter's API, allowing you to scrape a user's followers, following, Tweets and more while evading most API limitations.
MIT License
15.86k stars 2.73k forks source link

dataclasses dependency should be declared only for Python 3.6 #1000

Open johnthagen opened 4 years ago

johnthagen commented 4 years ago

Issue Template

Initial Check

Make sure you've checked the following:

Command Ran

Please provide the exact command ran including the username/search/code so I may reproduce the issue.

# pyproject.toml
# ...
[tool.poetry.dependencies]
python = "^3.7"
twint = {git = "https://github.com/twintproject/twint.git", rev = "master"}
$ poetry install
Installing dependencies from lock file

  SolverProblemError

  Because twint (2.1.21 git rev master) depends on dataclasses (*) which doesn't match any versions, twint is forbidden.
  So, because my-package depends on twint (2.1.21 git rev master), version solving failed.

  at ~/.poetry/lib/poetry/puzzle/solver.py:241 in _solve
      237│             packages = result.packages
      238│         except OverrideNeeded as e:
      239│             return self.solve_in_compatibility_mode(e.overrides, use_latest=use_latest)
      240│         except SolveFailure as e:
    → 241│             raise SolverProblemError(e)
      242│ 
      243│         results = dict(
      244│             depth_first_search(
      245│                 PackageNode(self._package, packages), aggregate_package_nodes

Description of Issue

The dataclasses PyPI package is only available for Python 3.6: https://pypi.org/project/dataclasses/

Requires: Python >=3.6, <3.7

This requirement should be specified as:

dataclasses; python_version < '3.7'

This broke in #955

Environment Details

Greatdane commented 4 years ago

Is there a way to fix this?

On Python 3.8 I keep getting this error relating to dataclasses;

Traceback (most recent call last):
  File "/layers/google.python.webserver/gunicorn/gunicorn/arbiter.py", line 583, in spawn_worker
    worker.init_process()
  File "/layers/google.python.webserver/gunicorn/gunicorn/workers/gthread.py", line 92, in init_process
    super().init_process()
  File "/layers/google.python.webserver/gunicorn/gunicorn/workers/base.py", line 119, in init_process
    self.load_wsgi()
  File "/layers/google.python.webserver/gunicorn/gunicorn/workers/base.py", line 144, in load_wsgi
    self.wsgi = self.app.wsgi()
  File "/layers/google.python.webserver/gunicorn/gunicorn/app/base.py", line 67, in wsgi
    self.callable = self.load()
  File "/layers/google.python.webserver/gunicorn/gunicorn/app/wsgiapp.py", line 49, in load
    return self.load_wsgiapp()
  File "/layers/google.python.webserver/gunicorn/gunicorn/app/wsgiapp.py", line 39, in load_wsgiapp
    return util.import_app(self.app_uri)
  File "/layers/google.python.webserver/gunicorn/gunicorn/util.py", line 358, in import_app
    mod = importlib.import_module(module)
  File "/opt/python3.8/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1014, in _gcd_import
  File "<frozen importlib._bootstrap>", line 991, in _find_and_load
  File "<frozen importlib._bootstrap>", line 975, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 671, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "/srv/main.py", line 1, in <module>
    from ElonTweets.wsgi import application
  File "/srv/ElonTweets/wsgi.py", line 16, in <module>
    application = get_wsgi_application()
  File "/layers/google.python.pip/pip/django/core/wsgi.py", line 12, in get_wsgi_application
    django.setup(set_prefix=False)
  File "/layers/google.python.pip/pip/django/__init__.py", line 24, in setup
    apps.populate(settings.INSTALLED_APPS)
  File "/layers/google.python.pip/pip/django/apps/registry.py", line 122, in populate
    app_config.ready()
  File "/srv/website/apps.py", line 8, in ready
    from website import updater
  File "/srv/website/updater.py", line 5, in <module>
    import twint
  File "/layers/google.python.pip/pip/twint/__init__.py", line 12, in <module>
    from .config import Config
  File "/layers/google.python.pip/pip/twint/config.py", line 5, in <module>
    class Config:
  File "/layers/google.python.pip/pip/dataclasses.py", line 958, in dataclass
    return wrap(_cls)
  File "/layers/google.python.pip/pip/dataclasses.py", line 950, in wrap
    return _process_class(cls, init, repr, eq, order, unsafe_hash, frozen)
  File "/layers/google.python.pip/pip/dataclasses.py", line 800, in _process_class
    cls_fields = [_get_field(cls, name, type)
  File "/layers/google.python.pip/pip/dataclasses.py", line 800, in <listcomp>
    cls_fields = [_get_field(cls, name, type)
  File "/layers/google.python.pip/pip/dataclasses.py", line 659, in _get_field
    if (_is_classvar(a_type, typing)
  File "/layers/google.python.pip/pip/dataclasses.py", line 550, in _is_classvar
    return type(a_type) is typing._ClassVar
AttributeError: module 'typing' has no attribute '_ClassVar'

It appears that dataclasses should not even be a requirement. Yet I cant fix these even by uninstalling dataclasses module.

himanshudabas commented 4 years ago

@Greatdane @johnthagen I apologize for adding dataclasses as a dependency. I must admit I had no idea that it isn't required for Python >= 3.7.

Anyhow, I'll quickly put up a patch for it. while the PR gets merged, you can use it directly from my branch.

johnthagen commented 4 years ago

@himanshudabas Could you open a PR to this repo so it could be merged into master? Thanks!

himanshudabas commented 4 years ago

@johnthagen Sure. Will do.

johnthagen commented 4 years ago

@himanshudabas And for reference, dataclasses were added to the Python stdlib in 3.7: https://docs.python.org/3/library/dataclasses.html

The dataclasses module on PyPI is a backport so that Python 3.6 environments can use it.