Package cannot be built from source without installing Django first

[niklasb]

On systems where no binary build is available, django-picklefield cannot be installed in a clean virtualenv, because it relies on Django being installed at build time

To reproduce:

$ mkvirtualenv -p python3.8 test-django-picklefield-build
$ workon test-django-picklefield-build
$ pip install --no-binary django-picklefield django-picklefield
Collecting django-picklefield
  Using cached django-picklefield-3.0.1.tar.gz (9.5 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/niklas/.virtualenvs/test-django-picklefield/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/setup.py'"'"'; __file__='"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-pip-egg-info-m4z3m_ay
         cwd: /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/setup.py", line 5, in <module>
        import picklefield
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-7k5uva3e/django-picklefield/picklefield/__init__.py", line 3, in <module>
        import django.utils.version
    ModuleNotFoundError: No module named 'django'
    ----------------------------------------
ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

The obvious fix is to maintain the version differently, e.g. https://github.com/niklasb/django-picklefield/commit/dc2979840cdb0862ac22ca4b0ab3dc41684720f3

[carltongibson]

Hi. I'm bit really following what you're trying to do here? I can't quite see why you'd object to installing Django? 🤔

[niklasb]

It's not a matter of installing Django or not. The problem is that there is a build-time dependency on Django, which is not something that pip supports to my knowledge (at least not when imported from setup.py). This means you can't install the most recent versions of the package if you disallow binary builds:

$ mkvirtualenv -p python3.8 test-django-picklefield-build
$ cat requirements.txt
django-picklefield
Django
$ pip install --no-binary :all: -r requirements.txt
Collecting django-picklefield
  Using cached django-picklefield-3.0.1.tar.gz (9.5 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/niklas/.virtualenvs/test-django-picklefield-build/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_b21642216cf64a79904769944e1fab7b/setup.py'"'"'; __file__='"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_b21642216cf64a79904769944e1fab7b/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-pip-egg-info-6tbagou7
         cwd: /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_b21642216cf64a79904769944e1fab7b/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_b21642216cf64a79904769944e1fab7b/setup.py", line 5, in <module>
        import picklefield
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_b21642216cf64a79904769944e1fab7b/picklefield/__init__.py", line 3, in <module>
        import django.utils.version
    ModuleNotFoundError: No module named 'django'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/3d/3d/5e8b2ce0be78427ca91e63fcbdce2b98ea356c7c15b1886055b7ad35b959/django-picklefield-3.0.1.tar.gz#sha256=15ccba592ca953b9edf9532e64640329cd47b136b7f8f10f2939caa5f9ce4287 (from https://pypi.org/simple/django-picklefield/) (requires-python:>=3). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached django-picklefield-3.0.tar.gz (9.7 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/niklas/.virtualenvs/test-django-picklefield-build/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_7be045266227404f9f72073df5ee1798/setup.py'"'"'; __file__='"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_7be045266227404f9f72073df5ee1798/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-pip-egg-info-ma4apxz1
         cwd: /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_7be045266227404f9f72073df5ee1798/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_7be045266227404f9f72073df5ee1798/setup.py", line 5, in <module>
        import picklefield
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_7be045266227404f9f72073df5ee1798/picklefield/__init__.py", line 3, in <module>
        import django.utils.version
    ModuleNotFoundError: No module named 'django'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/11/2d/426938ccb9e6460d6427641d4ec0ef35bce6dbf962f711d7be8d6e82fbfa/django-picklefield-3.0.tar.gz#sha256=28aa97dded0bfd14f067462638bd8c0c70c50da17ab265aee74a30f47fc2b024 (from https://pypi.org/simple/django-picklefield/) (requires-python:>=3). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached django-picklefield-2.1.1.tar.gz (9.4 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/niklas/.virtualenvs/test-django-picklefield-build/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_938706fc5dd44052a5ccef8d52c34742/setup.py'"'"'; __file__='"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_938706fc5dd44052a5ccef8d52c34742/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-pip-egg-info-1c2tdzpk
         cwd: /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_938706fc5dd44052a5ccef8d52c34742/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_938706fc5dd44052a5ccef8d52c34742/setup.py", line 5, in <module>
        import picklefield
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_938706fc5dd44052a5ccef8d52c34742/picklefield/__init__.py", line 3, in <module>
        import django.utils.version
    ModuleNotFoundError: No module named 'django'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/df/48/3c8a333b11312d73fb93dce589ae5a979e5334a5f5ee9918e580b88ddd96/django-picklefield-2.1.1.tar.gz#sha256=67a5e156343e3b032cac2f65565f0faa81635a99c7da74b0f07a0f5db467b646 (from https://pypi.org/simple/django-picklefield/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached django-picklefield-2.1.tar.gz (9.6 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/niklas/.virtualenvs/test-django-picklefield-build/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_4ff6b859102d43b1a1080078794f36f4/setup.py'"'"'; __file__='"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_4ff6b859102d43b1a1080078794f36f4/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-pip-egg-info-0nvi4ew5
         cwd: /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_4ff6b859102d43b1a1080078794f36f4/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_4ff6b859102d43b1a1080078794f36f4/setup.py", line 5, in <module>
        import picklefield
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_4ff6b859102d43b1a1080078794f36f4/picklefield/__init__.py", line 3, in <module>
        import django.utils.version
    ModuleNotFoundError: No module named 'django'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/d4/ee/1dab7cb639ff6fbf8f64259fb68f8a396ef5482eda0d64daee6161610953/django-picklefield-2.1.tar.gz#sha256=9f184548e98b7cb351bfacf16441f2760738eac8273a1e6088dd2b563a91e0d5 (from https://pypi.org/simple/django-picklefield/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached django-picklefield-2.0.tar.gz (10 kB)
    ERROR: Command errored out with exit status 1:
     command: /Users/niklas/.virtualenvs/test-django-picklefield-build/bin/python -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_0acfa97ead7247ad8970ea716f7b7496/setup.py'"'"'; __file__='"'"'/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_0acfa97ead7247ad8970ea716f7b7496/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-pip-egg-info-zh_eb763
         cwd: /private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_0acfa97ead7247ad8970ea716f7b7496/
    Complete output (7 lines):
    Traceback (most recent call last):
      File "<string>", line 1, in <module>
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_0acfa97ead7247ad8970ea716f7b7496/setup.py", line 5, in <module>
        import picklefield
      File "/private/var/folders/zy/5k15xt89127ck1l3sk615l5m0000gn/T/pip-install-5rjby5hq/django-picklefield_0acfa97ead7247ad8970ea716f7b7496/picklefield/__init__.py", line 3, in <module>
        import django.utils.version
    ModuleNotFoundError: No module named 'django'
    ----------------------------------------
WARNING: Discarding https://files.pythonhosted.org/packages/5e/4d/5732084d462b428325d98d8409e40215344098e39366b27d3af8cbee4d33/django-picklefield-2.0.tar.gz#sha256=f1733a8db1b6046c0d7d738e785f9875aa3c198215de11993463a9339aa4ea24 (from https://pypi.org/simple/django-picklefield/). Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.
  Using cached django-picklefield-1.1.0.tar.gz (13 kB)
Collecting Django
  Using cached Django-3.2.3.tar.gz (9.8 MB)
Collecting asgiref<4,>=3.3.2
  Using cached asgiref-3.3.4.tar.gz (30 kB)
Collecting pytz
  Using cached pytz-2021.1.tar.gz (317 kB)
Collecting sqlparse>=0.2.2
  Using cached sqlparse-0.4.1.tar.gz (67 kB)
Skipping wheel build for Django, due to binaries being disabled for it.
Skipping wheel build for asgiref, due to binaries being disabled for it.
Skipping wheel build for sqlparse, due to binaries being disabled for it.
Skipping wheel build for django-picklefield, due to binaries being disabled for it.
Skipping wheel build for pytz, due to binaries being disabled for it.
Installing collected packages: sqlparse, pytz, asgiref, django-picklefield, Django
    Running setup.py install for sqlparse ... done
    Running setup.py install for pytz ... done
    Running setup.py install for asgiref ... done
    Running setup.py install for django-picklefield ... done
    Running setup.py install for Django ... done
Successfully installed Django-3.2.3 asgiref-3.3.4 django-picklefield-1.1.0 pytz-2021.1 sqlparse-0.4.1

As you can see, pip went back all the way to django-picklefield-1.1.0 to find a version which has a setup.py that can actually run in this environment. My understanding is that setup.py should not import non-setuptools packages like this

[carltongibson]

Hi @niklasb — Interesting. I need to have a play with this.

It's pretty standard to import the package to access the version, so you don't need to duplicate the version declaration.

Just out of interest...

$ cat requirements.txt
django-picklefield
Django

... If you swap these round, so Django comes first, do you still get the same issue?

[niklasb]

@carltongibson no, I think if you swap it around Django will be available once django-picklefield setup.py runs. Unfortunately in many scenarios we don't control the order, e.g. we hit this issue with poetry (which for some reason doesn't use binary wheels in some scenarios)

[carltongibson]

OK, thanks for the confirmation @niklasb.

As I say, I'm not sure what to conclude immediately — that pattern is pretty standard — e.g. picking one from the hat crispy forms … — I'm pretty sure that duplicating the version declaration isn't the way to go 🤔

[niklasb]

@carltongibson since this package has an arch-independent binary wheel we can work around it. In other cases, where native components are involved, it's a big issue because it can make packages uninstallable via package managers on architectures for which no wheel is available.

To be honest the logic you pointed to in django-crispy-forms seems even more broken: If you install the package like that it will always be tagged with the version of the previously installed crispy, completely independent of what version of django-crispy-forms you are even installing? So you have to update crispy first, then django-crispy-forms, or else it will "work" but the version information will be wrong

[carltongibson]

To be honest the logic you pointed to in django-crispy-forms seems even more broken

It's been working fine for about a dozen years... 😀

I'll have a think about it. That's all I can say currently.

[niklasb]

@carltongibson I definitely don't mean to come across as criticizing your work, sorry :)

To give some context into why we care about this: Currently our entire Python-based infrastructure is dependent on x86_64, due to the fact that a lot of packages don't have source wheels on PyPI at all, or those don't actually compile properly because everybody uses x86_64 and errors go unnoticed. A lot of packages don't compile on linux/arm64, let alone on arm64 MacBooks (M1). Thus I started to raise issues like this against all packages that we use that do not properly install from source. In some cases those are blocking, in others they are not (like in this case, because arch-independent binary wheel is on PyPI).

[niklasb]

I also misunderstood the construct in crispy_forms, I thought crispy_forms was from a different package, but it is not

[niklasb]

As an example of what other packages often do to avoid duplciation, I can point to https://github.com/StellarCN/py-stellar-base/blob/master/setup.py#L12

[carltongibson]

OK, thanks, plenty to go on there. (With recent changes to Python packaging, it's likely setup.py's days are numbered anyhow – but …)