Python.h should include system headers properly [POSIX]

[d15ce025-cf0a-4385-9ce9-e8874168e077]

BPO	1337400
Nosy	@loewis, @smontanaro

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core'] title = 'Python.h should include system headers properly [POSIX]' updated_at = user = 'https://bugs.python.org/papadopo' ``` bugs.python.org fields: ```python activity = actor = 'loewis' assignee = 'none' closed = True closed_date = None closer = None components = ['Interpreter Core'] creation = creator = 'papadopo' dependencies = [] files = [] hgrepos = [] issue_num = 1337400 keywords = [] message_count = 8.0 messages = ['26696', '26697', '26698', '26699', '26700', '26701', '26702', '26703'] nosy_count = 3.0 nosy_names = ['loewis', 'skip.montanaro', 'papadopo'] pr_nums = [] priority = 'normal' resolution = 'works for me' stage = None status = 'closed' superseder = None type = None url = 'https://bugs.python.org/issue1337400' versions = ['Python 2.4'] ```

[d15ce025-cf0a-4385-9ce9-e8874168e077]

In Python 2.4.2, Python.h looks like this:

    #include <limits.h>
    [...]
    #include <stdio.h>
    [...]
    #include <string.h>
    #include <errno.h>
    #include <stdlib.h>
    #ifdef HAVE_UNISTD_H
    #include <unistd.h>
    #endif

On POSIX platforms \<unistd.h> should be included first!

Indeed it includes headers such as \<sys/feature_tests.h> on Solaris, \<standards.h> on Irix, or \<features.h> on GNU systems, which define macros that specify the system interfaces to use, possibly depending on compiler options, which in turn may enable/disable/modify parts of other system headers such as \<limits.h> or \<errno.h>.

By including \<unistd.h>, you ensure consistent systems interfaces are specified in all system headers included by Python sources.

This may seem rather academic, but it actually breaks my Solaris builds: I need to compile Python using Sun's C compiler when building Python for performance and GNU's C++ compiler when building Python modules written in C++ for compatibility with C++ libraries used by these modules that can't be compiled with Sun's C++ compiler. So the same Python.h is used by Sun's C compiler (which it was created for in the first place) and GNU's C++ compiler. GNU's C++ compiler fails to compile some modules. Unfortunately I can't recall the exact modules and error messages right now, but including \<unistd.h> fixes the problem.

[d15ce025-cf0a-4385-9ce9-e8874168e077]

Logged In: YES user_id=52414

Oops... Instead of including \<unistd.h> fixes the problem. please read including \<unistd.h> first fixes the problem.

Here is an example to reproduce the problem:

$ cat > foo.cpp
#include <Python.h>
#include <cwchar>
$
$ g++ -I/usr/local/python/include/python2.4 -c foo.cpp
[...]
/usr/local/gcc-4.0.2/lib/gcc/sparc-sun-solaris2.8/4.0.2/../../../../include/c++/4.0.2/cwchar:145:
error: '::btowc' has not been declared
[...]
$
$ cat > foo.cpp
#include <unistd.h>
#include <Python.h>
#include <cwchar>
$
$ g++ -I/usr/local/python/include/python2.4 -c foo.cpp
[...]
$

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Logged In: YES user_id=21627

Can you please point to the relevant section of the POSIX specification that states that unistdh.h must be included before stdlib.h?

As for the specific problem: it may be that you are somehow working around the real problem by including unistd.h before Python.h. Python.h *must* be included before any system headers, as pyconfig.h defines certain compilation options which affect the feature tests. Including unistd.h before can actually break things, as structs may get defined differently depending on whether pyconfig.h was included first or not.

So in the specific example, it would help if you could determine why ::btowc is defined in one case but not in the other.

[d15ce025-cf0a-4385-9ce9-e8874168e077]

Logged In: YES user_id=52414

Ah, I didn't explain myself clearly.

I meant to say that \<unistd.h> must be included before other system headers such as \<limits.h>, \<stdio.h>, \<string.h>, \<errno.h> and \<stdlib.h> in this specific case.

I totally agree it has to be included after "pyconfig.h". For example if "pyconfig.h" defined _HPUX_SOURCE and \<unistd.h> was included before "pyconfig.h", then wrong system APIs may be triggered (or at least system APIs that were not intended to be specified).

Now why \<unistd.h> should be included in front of other system headers? This is because: 1) \<unistd.h> triggers specific POSIX or Single UNIX Specification APIs 2) most if not all system headers do not include \<unistd.h>, so different APIs may be triggered before including \<unistd.h> and after including \<unistd.h>

I can't provide a section of the POSIX specification that explictly states that \<unistd.h> must be included before \<stdlib.h>. This is however implicit: http://www.opengroup.org/onlinepubs/009695399/basedefs/unistd.h.html As you can see \<unistd.h> may or not define macros that trigger a specific API (POSIX.1-1988, SUSv1, SUSv2, SUSv3, etc.).

I'll investigate what happens in the case of this specific failure and let you know.

[d15ce025-cf0a-4385-9ce9-e8874168e077]

Logged In: YES user_id=52414

Aaargh! I've rebuilt a version of Python 2.4.2 with Sun's C compiler and GNU's C++ compiler but I'm unable to reproduce the problem:

$ cat > foo.cpp
#include <Python.h>
#include <cwchar>
$
$ g++ -I/usr/local/python-2.4.2/include/python2.4 -c foo.cpp
$ 

Same Solaris 8 workstation, no OS updates, same GCC and same Sun compilers. Oh well...

I think it's still a good idea to include \<unistd.h> before \<limits.h>, \<stdio.h>, \<string.h>, \<errno.h> and \<stdlib.h>. But that's your call, I don't mind as long as I'm able to build Python.

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Logged In: YES user_id=21627

Thanks for the update. I believe nothing in the POSIX spec mandates to include unistd.h before anything else (unlike sys/types.h, which often is prerequisite to other headers), so I'm closing this report.

[smontanaro]

Logged In: YES user_id=44345

I'm coming late to this party (it seems the bar is closed), however...

At work we just stumbled upon a similar problem when trying to build the latest release of matplotlib (0.87.5, avaialble at http://matplotlib.sourceforge.net/) on Solaris 10 using gcc 3.4.1. We get errors like the following:

In file included from

/opt/app/g++lib6/gcc-3.4/lib/gcc/i386-pc-solaris2.10/3.4.1/../../../../include/c++/3.4.1/bits/postypes.h:46, from /opt/app/g++lib6/gcc-3.4/lib/gcc/i386-pc-solaris2.10/3.4.1/../../../../include/c++/3.4.1/iosfwd:50, from /opt/app/g++lib6/gcc-3.4/lib/gcc/i386-pc-solaris2.10/3.4.1/../../../../include/c++/3.4.1/ios:44, from /opt/app/g++lib6/gcc-3.4/lib/gcc/i386-pc-solaris2.10/3.4.1/../../../../include/c++/3.4.1/ostream:45, from /opt/app/g++lib6/gcc-3.4/lib/gcc/i386-pc-solaris2.10/3.4.1/../../../../include/c++/3.4.1/iostream:45, from swig/agg_buffer.h:7, from src/agg.cxx:1584:

/opt/app/g++lib6/gcc-3.4/lib/gcc/i386-pc-solaris2.10/3.4.1/../../../../include/c++/3.4.1/cwchar:145: error: `::btowc' has not been declared

/opt/app/g++lib6/gcc-3.4/lib/gcc/i386-pc-solaris2.10/3.4.1/../../../../include/c++/3.4.1/cwchar:150: error: `::fwide' has not been declared

If I add

    #include <wchar.h>

at the top of Python.h it builds fine (modulo a couple warnings). One warning which might be significant is:

In file included from

/opt/app/g++lib6/python-2.4/include/python2.4/Python.h:10, from src/agg.cxx:97:

/opt/app/g++lib6/python-2.4/include/python2.4/pyconfig.h:835:1: warning: "_POSIX_C_SOURCE" redefined In file included from /usr/include/wchar.h:11, from /opt/app/g++lib6/python-2.4/include/python2.4/Python.h:7, from src/agg.cxx:97: /usr/include/sys/feature_tests.h:266:1: warning: this is the location of the previous definition

I don't have enough gcc smarts to understand what's happening, or even if it's the same problem Dimitri encountered on Solaris 8, but it kind of looks like the same sort of problem to me.

Skip

[61337411-43fc-4a9c-b8d5-4060aede66d0]

Logged In: YES user_id=21627

Skip, your problem is completely independent of the problem reported here. It rather has to do with the way Sun non-C89 API.