try to build a C module, but don't worry if it doesn't work

[182980c4-e4f4-4e83-8e3e-662d412bbf71]

BPO	4706
Nosy	@tarekziade
Superseder	bpo-5583: Optional extensions in setup.py
Files	build-with-no-Python.h.txt build-with-no-gcc.txt Twisted-build-with-no-gcc.txt Twisted-build-no-Python.h.txt

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = 'https://github.com/tarekziade' closed_at = created_at = labels = ['type-feature', 'library'] title = "try to build a C module, but don't worry if it doesn't work" updated_at = user = 'https://bugs.python.org/zooko' ``` bugs.python.org fields: ```python activity = actor = 'tarek' assignee = 'tarek' closed = True closed_date = closer = 'tarek' components = ['Distutils'] creation = creator = 'zooko' dependencies = [] files = ['12910', '12911', '12912', '12913'] hgrepos = [] issue_num = 4706 keywords = [] message_count = 8.0 messages = ['78106', '79047', '80681', '80682', '80896', '80897', '80898', '80899'] nosy_count = 3.0 nosy_names = ['zooko', 'exarkun', 'tarek'] pr_nums = [] priority = 'normal' resolution = 'duplicate' stage = None status = 'closed' superseder = '5583' type = 'enhancement' url = 'https://bugs.python.org/issue4706' versions = ['Python 3.1', 'Python 2.7'] ```

[182980c4-e4f4-4e83-8e3e-662d412bbf71]

I was trying to install Twisted on my son's OLPC. Twisted comes with a handful of C extensions which are all optional and most of which are not expected to compile on this platform anyway (they are platform-specific for Mac or Windows). If my son's OLPC had a C compiler installed, then it would attempt to build those modules, fail, and proceed to install Twisted. However, since it doesn't have a C compiler, then instead it stops with an error.

It would be nice if Twisted could tell distutils to try to build a module, and report whether the build succeeded, but don't stop the entire setup().

http://twistedmatrix.com/trac/ticket/3586

[508f7627-3797-4332-8a63-a38898672f33]

I can see two cases here:

1/ a compiler is not found, and build_ext can't proceed 2/ a compiler is found but it fails

For 2/, I don't think it's wise to fail silently if a compiler is found and the build fails, because that can hide an unexpected problem.

For 1/ I would go for a binary distribution, or for a source distribution that is specific to Linux in your case, if the extensions are specific to other platforms. The latter could be done in the same setup.py if needed by declaring the extensions only on matching platforms.

But I can see the benefit of a source distribution with optional extensions in general. In that case, I think this option would have to be declared optionally for each extension to be able to fail properly if one extension is mandatory.

[182980c4-e4f4-4e83-8e3e-662d412bbf71]

Tarek:

Yes, failure to build an extension module for any reason should definitely cause "build" to fail by default. However, many Python packages seem to come with optional extension modules, typically for a performance speed-up. simplejson, zope.interface, and twisted all do this (these are three packages that allmydata-tahoe depends on). My own pyutil package used to do this, but I eventually just removed the optional C extension module in order to avoid this problem. So, for those it would be good if the package setup.py could specify that an extension module is optional and that failure to build it for any reason does not mean failure of the "build".

Note that twisted's setup.py already attempts to do this for certain of its extension modules, and if I understand correctly it is able to detect and handle failure to build if gcc fails, but not if gcc is absent.

[8726d1eb-a365-45b6-b81d-c75988975e5a]

As far as Twisted is concerned, the two cases Tarek mentioned aren't really relevant. In order for Zooko's use case to be handled ("install Twisted on an OLPC"), what would be ideal is if there were a way to tell distutils about an extension which is optional. It doesn't matter if it can be built or not. Any failure is irrelevant.

This doesn't necessarily have to mean that errors pass silently. The build failure can still be reported. The build could even fail by default and distutils could gain a new option for switching the behavior to succeeding if only optional extensions fail to build. Or it could be the other way around, with an option to make the build fail if any optional extensions cannot be built (this approach favors people who don't really know what a "build" is and just want something installed, leaving the more advanced option to people who are able to figure out that there are options that control build behavior).

[182980c4-e4f4-4e83-8e3e-662d412bbf71]

Hey check it out -- simplejson can try to build its extension module, and when it fails to compile (in this case because there is no Python.h), then it prints out a warning message and finishes a successful build:

[182980c4-e4f4-4e83-8e3e-662d412bbf71]

Oh! And simplejson is able to cleanly fall back to pure Python when gcc is not found, as well. Perhaps Twisted http://twistedmatrix.com/trac/ticket/3586 could use simplejson's approach.

[182980c4-e4f4-4e83-8e3e-662d412bbf71]

Here's Twisted failing to build when gcc is not installed:

[182980c4-e4f4-4e83-8e3e-662d412bbf71]

Here's Twisted failing to build because Python.h isn't found: