search

Attnam / ivan

Iter Vehemens ad Necem - a continuation of the graphical roguelike by members of http://attnam.com
GNU General Public License v2.0
301 stars 43 forks source link

Compiling: Can not find stddef.h #524

Open Jubatian opened 5 years ago

Jubatian commented 5 years ago

I tried to compile the game on Debian Stretch, and couldn't get it work either on the current master or version v055. When I go through the process as the INSTALL document suggests, for cmake, I get reports on not finding Threads:

-- Looking for pthread.h -- Looking for pthread.h - not found -- Could NOT find Threads (missing: Threads_FOUND)

I have pthread.h at /usr/include/pthread.h and at /usr/include/newlib/pthread.h

Since it generated a Makefile, I tried to give it a go (make -j), however then first on "Building CXX object FeLib/CMakeFiles/FeLib.dir/Source/rawbit.cpp.o" I get a fatal error: "'stddef.h' file not found".

I have this file at /usr/lib/gcc/x86_64-linux-gnu/6/include/stddef.h (at least this is which corresponds to the native compiler, I do some cross-compiling on this system for various targets).

The native GNU C/C++ toolchain appears functional for me, at least I can build for example Battle for Wesnoth from source.

ryfactor commented 5 years ago

Hi @Jubatian, I love your Flight of a Dragon game! https://github.com/Jubatian/foad

I got IVAN working on a fresh Debian Stretch in a virtual box (amd64 iso image). I only had to install git, cmake, g++ and libSDL2-dev, libSDL-mixer-dev and libpng-dev libraries. Maybe your system has some extra goodies because of the embedded c stuff you do? Our repo's cmake files might not be intelligent enough...

Where's your install of SDL2 located?

jakwings commented 5 years ago

What were the include paths when it failed? More detailed error log will help.

Jubatian commented 5 years ago

Hi Both,

Thanks :)

I was guessing the same for the error, maybe it simply tries to use one of my more esoteric compilers for some reason. avr-gcc for example certainly wouldn't work too well for this for sure.

The SDL2 includes are on /usr/include/SDL2 (nothing fancy, just as it came out of the package manager). The mentioned packages seem to be installed, at least if you refer libSDL2-mixer-dev (but I also have the SDL1 development packages).

Could you help me getting more error log out of this? I am not really knowledgeable with these make systems (such as cmake), and can't find how I could get it for example to display the commands issued for compiling (which could show it clear if it was using the wrong compiler). I tried digging around in the makefiles for this for a while without success.

I have barely any more meaningful reports as of now following the normal INSTALL procedure.

[ 30%] Building CXX object FeLib/CMakeFiles/FeLib.dir/Source/specialkeys.cpp.o
In file included from /home/pettyes/games_etc_stuff/Attnam/ivan/fantasyname/namegen.cc:10:
/home/pettyes/games_etc_stuff/Attnam/ivan/fantasyname/namegen.h:72:10: fatal error: 'stddef.h' file not found
#include <stddef.h>       // for size_t
         ^~~~~~~~~~
In file included from /home/pettyes/games_etc_stuff/Attnam/ivan/xbrzscale/libxbrzscale.cpp:19:
In file included from /home/pettyes/games_etc_stuff/Attnam/ivan/xbrzscale/libxbrzscale.h:22:
In file included from /usr/include/SDL2/SDL_stdinc.h:34:
/usr/include/x86_64-linux-gnu/sys/types.h:146:10: fatal error: 'stddef.h' file not found
#include <stddef.h>
         ^~~~~~~~~~
In file included from /home/pettyes/games_etc_stuff/Attnam/ivan/xbrzscale/xbrz/xbrz.cpp:7:
In file included from /home/pettyes/games_etc_stuff/Attnam/ivan/xbrzscale/xbrz/xbrz.h:10:
/usr/lib/gcc/x86_64-linux-gnu/6.3.0/../../../../include/c++/6.3.0/cstddef:50:10: fatal error: 'stddef.h' file not found
#include <stddef.h>
         ^~~~~~~~~~

So this would be an extract of how the stddef.h is not found. The only significant thing I see is that it tries to include files from the correct locations (/usr/lib/gcc/x86_64-linux-gnu/6.3.0), but somehow the paths don't work out. If I query the compiler itself for its include paths, I see that they look correct, however I don't know what the game's compile process might do in this regard (and whether it might use the wrong compiler).

jakwings commented 5 years ago

make VERBOSE=1 will show the commands.

cmake -DCMAKE_CXX_FLAGS='-v' will output more content.

You can also grep -rIisn PATTERN_HERE ./ in the build dir.

Jubatian commented 5 years ago

I attached both the output of CMake and Make called with adding the verbosity parameters. CMake itself doesn't seem to do anything regarding it, still doesn't show how it can not find pthreads.h (which is at /usr/include/pthread.h), however the Make output feels like revealing something, at least if I interpret all that right.

So I have a local LLVM installation, only having the Emscripten target (to build the web based version of the CUzeBox emulator), this is integrated in my user paths so it shows up like any globally installed compiler package in my local environment. For me it seems like this compiler was found and attempted to be used, if this was the case, it obviously wouldn't work too well.

dump_cmake.txt dump_make.txt

jakwings commented 5 years ago

-DCMAKE_CXX_FLAGS='-v' only affects the compiler. cmake --trace-expand shows more about cmake.

Passing -j1 (single thread) to "make" can prevent random overlaps for output of stdout and stderr.

If I understand correctly, you're not trying to compile IVAN with Emscripten? Neither /usr/lib/gcc/x86_64-linux-gnu/6/include nor /usr/lib/gcc/x86_64-linux-gnu/6.3.0/include seems to be used.

If it is impossible to remove unrelated paths from $PATH, then you can try:

export CC=/path/to/gcc CXX=/path/to/g++

If (unlikely) the correct compiler still does not use the right include path, you can:

export {C,CXX}FLAGS='-I /usr/lib/gcc/x86_64-linux-gnu/6/include -I /usr/include' \
       LDFLAGS='-L /usr/lib/gcc/x86_64-linux-gnu/6/lib'

and do not pass -DCMAKE_C_FLAGS=(anything) nor -DCMAKE_CXX_FLAGS=(anything) to cmake.

Jubatian commented 5 years ago

I am totally clueless what is going on here. I have Emscrpiten stuff added in my .bashrc, removed them, checked that they are all indeed removed by looking through my environment variables (newly started shell using the modified .bashrc), set up CC and CXX to point at my native compiler, cleaned the repo (git clean -f), yet CMake somehow still manages to fish out that Emscripten compiler by some dark magic and attempts to use it.

The only thing I could figure out is that the missing pthread.h is also due to that, since it tries to check for it with Emscripten, and can't find stddef.h during the check (which pthread.h includes).

I really would like to know how it even sees the Emscripten compiler this case which is on a local path, what information it uses to set up CMAKE_CXX_COMPILER?

dump_set.txt dump_env.txt dump_cmake.txt

Jubatian commented 5 years ago

Huh, nevermind. git clean -df. I can finally build and run it! It would still be nice to know why it picks Emscripten if it is available, or if it is intended to be buildable by Emscripten, why doesn't that work this case (I find it odd that seemingly it is intended to be built with the GNU tools, and CMake rather chooses an LLVM compiler).

jakwings commented 5 years ago

Congrats!

It must be due to CMakeCache.txt in the build dir. Having a clean build dir is important once the compile flags has changed. And make clean won't delete CMakeCache.txt for you.

Now, does it still works without manually setting CC and CXX?

Jubatian commented 5 years ago

It works! (no CC or CXX environment variable set up at all) Interestingly it doesn't try to use my RISC-V compiler while that's also similarly installed locally like Emscripten.

(I didn't try to use make clean, I see that with any automatic makefile generation system, be it automake, cmake or scons, the make clean should normally only clean up the build, and not the generated Makefiles themselves. I just didn't notice that git clean -f won't clean up my repo, first I tried git reset --hard, just noticing that it does nothing to untracked files did I go on attempting to find some quick solution. Of course a manual wipe + any command to checkout / reset the repo would have done it)

jakwings commented 5 years ago

It would still be nice to know why it picks Emscripten if it is available,

It provides clang++ whose name is so famous. It will have a smaller chance if the bin path is behind gcc's or if the name is prefixed like ems-clang++.

if it is intended to be buildable by Emscripten.

impossible without modification to IVAN's source code.

I tried git reset --hard, just noticing that it does nothing to untracked files.

Yeah, learning git is hard. ;) I just noticed that you run cmake in the root of the repo. And fantasyname/Makefile gets modified if so doing (not a big problem). I would recommend mkdir build && cd build && cmake ... Cleaning is simply rm -RI build.

My shell prompt is customized to show the git status. Other than git clean -df, often I also use git stash && git stash drop # or && git stash clear to discard all local changes to tracked files.

Jubatian commented 5 years ago

I changed the order in $PATH, and it indeed picks the proper compiler if Emscripten was behind! But it is odd that it wouldn't pick the RISC-V compiler which is the very first item on my path, somehow it can see that that particular compiler wouldn't do the job.

Yes, I run cmake on the root since INSTALL documents it that way. And I ran make with the -j parameter as well, since INSTALL documents it that way. I always try to follow what the particular application's documentation says, knowing things may just fail because of trying to do it different otherwise (I of course found it odd why -j was needed for make, but guessed let's go with it, the application's author sure knows better or if there was any weird issue without -j).

So maybe time to update INSTALL then! Battle for Wesnoth in particular describes that you can build out of the source tree, and I do it that way: it is always better if supported properly. But it may happen that it isn't, and you get errors with some application especially because of trying to do just that.

By the way how do you get that shell prompt modification? I know it is possible to modify the prompt, just curious about whether you knew something which would work well in the practice (I mean the added info. is practical).

jakwings commented 5 years ago

cmake always checks if the compiler works (the order is unknown), like this:

-- The CXX compiler identification is Clang 7.0.0
-- The C compiler identification is Clang 7.0.0
-- Check for working CXX compiler: /Users/mmmmmm/.local/opt/llvm-7.0.0/bin/clang++
-- Check for working CXX compiler: /Users/mmmmmm/.local/opt/llvm-7.0.0/bin/clang++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Check for working C compiler: /Users/mmmmmm/.local/opt/llvm-7.0.0/bin/clang
-- Check for working C compiler: /Users/mmmmmm/.local/opt/llvm-7.0.0/bin/clang -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done

Maybe you can find /usr/share/git/git-prompt.sh or /usr/local/share/git/git-prompt.sh Or https://github.com/git/git/blob/master/contrib/completion/git-prompt.sh It provides basically what I need. It is self-documented. The relevant settings are GIT_PS1_SHOWDIRTYSTATE, GIT_PS1_SHOWUNTRACKEDFILES, GIT_PS1_SHOWUPSTREAM.

Jubatian commented 5 years ago

Thanks, I could set up the Git prompt. Interestingly when using Midnight Commander, it only works properly with PROMPT_COMMAND, and not by setting PS1 (when entering a directory in mc, it wouldn't pick up the Git part of the prompt, only on the second navigation would it appear). Anyway, seems quite a nice thing to have!

The problem is getting weirder. I tried to restore my old .bashrc to try a few things now knowing that the Git repo should be cleaned proper with git clean -**d**f (and not by git clean -f as I assumed before), however now I can not reproduce the problem at all. CMake would select the proper native compiler now, always, desperate I even tried clone a fresh copy of the repo, to reboot, and nothing.

(Seriously I hate to report problems as this seems to happen just too often to me. Have a problem, spend several hours on it, call somebody to help, then at once poof, the problem disappears as if it was magic, and there is no way to reproduce it, I am staring at the screen as if I saw a ghost. At least it happens way more often on Windows)

It may still worth a look as to why it is even possible for CMake to select Clang for this game if the game is not possible to be compiled and linked by Clang (or is it?). There should be some way to tell it that the project needs the GNU tools, and it should error out instead of trying to go on if it only (?) finds Clang available.

jakwings commented 5 years ago

The Clang you used has been customized for Emscripten (for transpiling C/C++/etc to JavaScript/WebAssembly). I am not familiar with Emscripten, so not even sure if a C++ project can be so easily transpiled.

The Clang I'm using is just like GCC. It has support from many platforms: https://llvm.org/

The best practice is to clean up the build environment. Export CC and CXX only when necessary, and you don't have to put the bin path in $PATH since normally the built-in search paths should have been automatically set up when compiling the compiler. Then only the native compiler is available through $PATH. There are options to specify/ignore some paths but it is unclear in which order the compilers are selected when they are in the same directory.

Jubatian commented 5 years ago

So it should then compile fine with an appropriate native LLVM. I know about the compiler infrastructure, however I just didn't need it here so far, of course I might have tried it with for example the CUzeBox emulator (should work just fine if it works with the Emscripten Clang). Anyway then it is reasonable for CMake to pick up Clang if it sees one, however here the platform was definitely wrong. At least if Emscripten actually reports that properly.

I wouldn't only have the native compiler in my $PATH even if I removed all my local installations as I have a few cross-compilers installed as stock Debian packages too, and they are accessible. Not having the compiler on the path makes things awkward, at least when you don't use some automake solution (which I don't, I don't know how complex they are, but possibly it wouldn't be so convenient to set it up for every AVR project I am messing around with for example). Emscripten provided a very elaborate addition of environment variables on its install, and I really didn't want to mess around with it.

It would be nice if CMake at least would error out after getting into this situation, even it just saying something like "no suitable compiler was found" could be more helpful than trying to go on and fail with such obscure messages.

jakwings commented 5 years ago

The problem is not how many compilers are available. The compilers that can be auto-detected by cmake is limited. The issue is the clang from Emscripten and the environment you set up are not suitable for arbitrary projects. If you think it is a good default environment, then you may still be able to compile successfully by furthur manipulating the search paths. After all this is an anvanced topic so I don't have the urge to add this check until I play around with Emscripten. PR welcome.

Jubatian commented 5 years ago

I am aware of that it is not a good default environment for building stuff with Clang, my only problem here is that why CMake doesn't notice it in any manner that the platform is not right when normally it should be capable to select the correct compiler or indicate that it couldn't find one suitable (or maybe for some reason it assumes that Emscripten is okay, and possibly could mess up even if I had a native Clang).

Maybe this is rather a CMake issue that something related to this game.

I would say close it as a wontfix. No time, no energy, and I don't even have network bandwidth (traffic-limited plan) to experiment with various compiler installations and configurations to see what is actually happening here.

jakwings commented 5 years ago

Summary:

No time, no energy

then don't panic and please wisely use the native compiler before you're familiar with Emscripten.

At least you can play IVAN now.

Just my own thoughts.

Jubatian commented 5 years ago

My Emscripten installation works fine for stuff which is meant to be compiled with it (such as my own Uzebox emulator). It is not trivial to get something working within Emscripten, it only makes things easier, but if a particular project's architecture is just too poorly fitted for it, then it may even be nearly impossible. And it of course doesn't have all the Unix utilities, here with the missing stddef.h, the situation seems like that possibly CMake sees it a native compiler, and sets up paths into usr/include (which shouldn't be done with Emscripten at all), while the compiler specific paths are missing (even if those were set up, this likely wouldn't work). And-or maybe the game even uses some headers which don't exist in Emscripten for whatever reason.

So the situation is that however it happens, CMake makes a really wrong guess by choosing the Emscripten Clang for compiling, and it possibly even messes up the paths. With this compiler, the native include paths shouldn't be visible at all, so if CMake did the right job setting up things, there wouldn't be "stddef.h not found" type errors. Headers might be missing, but only because the game directly includes things which doesn't exist within Emscripten.

So what I would expect as possible reasonable, correct behaviors here:

My best guess that this case your game is either consciously or inconsciously set up to prioritize Clang, and something is preventing CMake to discover that the Clang it found is not suitable for the task. Most likely a problem on CMake's end, or a situation of a general lack of information if in this particular locally installed setup CMake has no means to extract the compiler's capabilities (however these mechanisms work).

I have this setup actually since years, and so far nothing I tried failed to build in it in any even remotely similar manner. However CMake use is still relatively rare, as far as I can recall, Battle for Wesnoth is the only significant project using CMake which I built from source here. Most of the stuff around uses Autotools, which never had any issue with the locally installed Emscripten and its paths.

jakwings commented 5 years ago

My best guess that this case your game is either consciously or inconsciously set up to prioritize Clang, and something is preventing CMake to discover that the Clang it found is not suitable for the task.

https://gitlab.kitware.com/cmake/community/wikis/FAQ#how-do-i-use-a-different-compiler https://gitlab.kitware.com/cmake/community/wikis/FAQ#how-do-i-use-a-different-compiler https://gitlab.kitware.com/cmake/community/wikis/FAQ#how-do-i-use-a-different-compiler (please submit an issue to the cmake project if you are so paranoid)

Enough guesses. Prove us:

Or you can wait for our proof. No ETA.

I have this setup actually since years, and so far nothing I tried failed to build in it in any even remotely similar manner. However CMake use is still relatively rare, as far as I can recall, Battle for Wesnoth is the only significant project using CMake which I built from source here. Most of the stuff around uses Autotools, which never had any issue with the locally installed Emscripten and its paths.

Please update your outdated workflow. How can you talk this proudly since you have little experience with cmake. It's a pity that you still know so little about Emscripten.

If you do not want to do anything more useful here, relax.

Summary:

ryfactor commented 5 years ago

I think @Jubatian has a point. Wesnoth compiles on his setup no sweat.

Further inspection of the Wesnoth CMakeLists.txt reveals it to have almost 10 times the number of lines IVAN's does. I think a better question is: what does Wesnoth's cmake setup do to avoid @Jubatian's build problem, that IVAN's cmake setup does not?