search

ladislav-zezula / StormLib

Official GitHub repository of the StormLib library created by Ladislav Zezula (author)
http://www.zezula.net/mpq.html
MIT License
557 stars 213 forks source link

Build failing on FreeBSD #73

Closed cen1 closed 8 years ago

cen1 commented 8 years ago

I've been testing my new project (http://cross.build) with StormLib and it seems that FreeBSD build is broken due to a missing define. Seems to be hidden behind a PLATFORM_MAC macro in StormPort.h.

Build commands:

sudo pkg install -y lang/gcc5
sudo pkg install -y lbzip2
sudo pkg install -y cmake
cmake -G "Unix Makefiles" -H./ -B./build && cd build && make

Result:

Started by user admin Building remotely on cfbbc2a1dff79d0543ccba8390946cfb6 in workspace /home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611

git rev-parse --is-inside-work-tree # timeout=10 Fetching changes from the remote Git repository git config remote.origin.url https://github.com/ladislav-zezula/StormLib.git # timeout=10 Fetching upstream changes from https://github.com/ladislav-zezula/StormLib.git git --version # timeout=10 git -c core.askpass=true fetch --tags --progress https://github.com/ladislav-zezula/StormLib.git +refs/heads/:refs/remotes/origin/ git rev-parse refs/remotes/origin/master^{commit} # timeout=10 git rev-parse refs/remotes/origin/origin/master^{commit} # timeout=10 Checking out Revision b86b4e7f6385ced4b34c1c84785ac5dc19be660f (refs/remotes/origin/master) git config core.sparsecheckout # timeout=10 git checkout -f b86b4e7f6385ced4b34c1c84785ac5dc19be660f git rev-list b86b4e7f6385ced4b34c1c84785ac5dc19be660f # timeout=10 [f0f78982-a45a-4acb-ac47-44516bed7611] $ /bin/sh -xe /tmp/hudson5082708045181654517.sh

  • sudo pkg install -y lang/gcc5 Updating FreeBSD repository catalogue... FreeBSD repository is up-to-date. All repositories are up-to-date. Checking integrity... done (0 conflicting) The most recent version of packages are already installed
  • sudo pkg install -y lbzip2 Updating FreeBSD repository catalogue... FreeBSD repository is up-to-date. All repositories are up-to-date. Checking integrity... done (0 conflicting) The most recent version of packages are already installed
  • sudo pkg install -y cmake Updating FreeBSD repository catalogue... FreeBSD repository is up-to-date. All repositories are up-to-date. The following 5 package(s) will be affected (of 0 checked):

New packages to be INSTALLED: cmake: 3.5.0 jsoncpp: 0.6.0.r2_2 libarchive: 3.2.0,1 lzo2: 2.09 cmake-modules: 3.5.0

The process will require 27 MiB more space. 5 MiB to be downloaded. [cfbbc2a1dff79d0543ccba8390946cfb6] Fetching cmake-3.5.0.txz: .......... done [cfbbc2a1dff79d0543ccba8390946cfb6] Fetching jsoncpp-0.6.0.r2_2.txz: .......... done [cfbbc2a1dff79d0543ccba8390946cfb6] Fetching libarchive-3.2.0,1.txz: .......... done [cfbbc2a1dff79d0543ccba8390946cfb6] Fetching lzo2-2.09.txz: .......... done [cfbbc2a1dff79d0543ccba8390946cfb6] Fetching cmake-modules-3.5.0.txz: .......... done Checking integrity... done (0 conflicting) [cfbbc2a1dff79d0543ccba8390946cfb6] [1/5] Installing lzo2-2.09... [cfbbc2a1dff79d0543ccba8390946cfb6] [1/5] Extracting lzo2-2.09: .......... done [cfbbc2a1dff79d0543ccba8390946cfb6] [2/5] Installing jsoncpp-0.6.0.r2_2... [cfbbc2a1dff79d0543ccba8390946cfb6] [2/5] Extracting jsoncpp-0.6.0.r2_2: .......... done [cfbbc2a1dff79d0543ccba8390946cfb6] [3/5] Installing libarchive-3.2.0,1... [cfbbc2a1dff79d0543ccba8390946cfb6] [3/5] Extracting libarchive-3.2.0,1: .......... done [cfbbc2a1dff79d0543ccba8390946cfb6] [4/5] Installing cmake-modules-3.5.0... [cfbbc2a1dff79d0543ccba8390946cfb6] [4/5] Extracting cmake-modules-3.5.0: .......... done [cfbbc2a1dff79d0543ccba8390946cfb6] [5/5] Installing cmake-3.5.0... [cfbbc2a1dff79d0543ccba8390946cfb6] [5/5] Extracting cmake-3.5.0: .......... done

  • cmake -G 'Unix Makefiles' -H./ -B./build -- The C compiler identification is Clang 3.4.1 -- The CXX compiler identification is Clang 3.4.1 -- Check for working C compiler: /usr/bin/cc -- Check for working C compiler: /usr/bin/cc -- works -- Detecting C compiler ABI info -- Detecting C compiler ABI info - done -- Detecting C compile features -- Detecting C compile features - done -- Check for working CXX compiler: /usr/bin/CC -- Check for working CXX compiler: /usr/bin/CC -- works -- Detecting CXX compiler ABI info -- Detecting CXX compiler ABI info - done -- Detecting CXX compile features -- Detecting CXX compile features - done -- Configuring done -- Generating done -- Build files have been written to: /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/build
  • cd build
  • make Scanning dependencies of target storm [ 3%] Building CXX object CMakeFiles/storm.dir/src/adpcm/adpcm.cpp.o [ 6%] Building CXX object CMakeFiles/storm.dir/src/huffman/huff.cpp.o [ 9%] Building C object CMakeFiles/storm.dir/src/jenkins/lookup3.c.o [ 12%] Building C object CMakeFiles/storm.dir/src/lzma/C/LzFind.c.o [ 16%] Building C object CMakeFiles/storm.dir/src/lzma/C/LzmaDec.c.o [ 19%] Building C object CMakeFiles/storm.dir/src/lzma/C/LzmaEnc.c.o [ 22%] Building C object CMakeFiles/storm.dir/src/pklib/explode.c.o [ 25%] Building C object CMakeFiles/storm.dir/src/pklib/implode.c.o [ 29%] Building CXX object CMakeFiles/storm.dir/src/sparse/sparse.cpp.o [ 32%] Building CXX object CMakeFiles/storm.dir/src/FileStream.cpp.o /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:96:73: error: use of undeclared identifier 'O_LARGEFILE' handle = open(pStream->szFileName, O_RDWR | O_CREAT | O_TRUNC | O_LARGEFILE, S_IRUSR | S_IWUSR | S_IRGRP | S_IROTH); ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:143:23: error: variable has incomplete type 'struct stat64' struct stat64 fileinfo; ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:143:16: note: forward declaration of 'stat64' struct stat64 fileinfo; ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:148:43: error: use of undeclared identifier 'O_LARGEFILE' handle = open(szFileName, oflag | O_LARGEFILE); ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:217:58: error: unknown type name 'off64_t'; did you mean 'off_t'? lseek64((intptr_t)pStream->Base.File.hFile, (off64_t)(ByteOffset), SEEK_SET); ^~~ off_t /usr/include/sys/types.h:173:18: note: 'off_t' declared here typedef off_t offt; /* file offset / ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:217:13: error: use of undeclared identifier 'lseek64'; did you mean 'lseek'? lseek64((intptr_t)pStream->Base.File.hFile, (off64_t)(ByteOffset), SEEK_SET); ^~~ lseek /usr/include/sys/types.h:376:8: note: 'lseek' declared here off_t lseek(int, off_t, int); ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:288:58: error: unknown type name 'off64_t'; did you mean 'off_t'? lseek64((intptr_t)pStream->Base.File.hFile, (off64_t)(ByteOffset), SEEK_SET); ^~~ off_t /usr/include/sys/types.h:173:18: note: 'off_t' declared here typedef __off_t offt; / file offset _/ ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:288:13: error: use of undeclared identifier 'lseek64'; did you mean 'lseek'? lseek64((intptr_t)pStream->Base.File.hFile, (off64_t)(ByteOffset), SEEK_SET); ^~~ lseek /usr/include/sys/types.h:376:8: note: 'lseek' declared here off_t lseek(int, off_t, int); ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:349:61: error: unknown type name 'off64_t'; did you mean 'off_t'? if(ftruncate64((intptr_t)pStream->Base.File.hFile, (off64_t)NewFileSize) == -1) ^~~ off_t /usr/include/sys/types.h:173:18: note: 'off_t' declared here typedef off_t offt; / file offset _/ ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:349:12: error: use of undeclared identifier 'ftruncate64'; did you mean 'ftruncate'? if(ftruncate64((intptr_t)pStream->Base.File.hFile, (off64_t)NewFileSize) == -1) ^~~ ftruncate /usr/include/sys/types.h:372:6: note: 'ftruncate' declared here int ftruncate(int, offt); ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:491:19: error: variable has incomplete type 'struct stat64' struct stat64 fileinfo; ^ /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/src/FileStream.cpp:491:12: note: forward declaration of 'stat64' struct stat64 fileinfo; ^ 10 errors generated. ** Error code 1

Stop. make[2]: stopped in /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/build *\ Error code 1

Stop. make[1]: stopped in /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/build *\ Error code 1

Stop. make: stopped in /usr/home/crossbuild/workspace/f0f78982-a45a-4acb-ac47-44516bed7611/build Build step 'Execute shell' marked build as failure Finished: FAILURE

Same using g++5 just to be sure: cmake -DCMAKE_CXX_COMPILER=/usr/local/bin/g++5 -DCMAKE_C_COMPILER=/usr/local/bin/gcc5 -G "Unix Makefiles" -H./ -B./build && cd build && make

ladislav-zezula commented 8 years ago

Well, "broken" is not the correct word. A build for FreeBSD was never available. And given that you are the first one in about 10 years concerned about it, it's not gonna be a big deal :-)

I am not a BSD user, not programmer, so if you want a BSD build, feel free to add the missing defines into CMakeLists.txt (you can also create your own makefile as well). I'll gladly add them into the main branch, but I am unable to verify the building process.

On the other hand, the code base is fairly stable (not much changes lately), as the MPQ format is dying anyway.

cen1 commented 8 years ago

I'll try to set the platform macro to Mac or Linux when FreeBSD is detected and see if that goes through. Hopefully that works because if it does not, BSD specific defines will need to be added to the source.

Btw, we are using StormLib on FreeBSD for years. :)

wheybags commented 8 years ago

Unrelated but your site doesn't load on chrome on android (just eternal spinning gear) On 24 May 2016 08:27, "cen1" notifications@github.com wrote:

I'll try to set the platform macro to Mac or Linux when FreeBSD is detected and see if that goes through. Hopefully that works because if it does not, BSD specific defines will need to be added to the source.

Btw, we are using StormLib on FreeBSD for years. :)

— You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub https://github.com/ladislav-zezula/StormLib/issues/73#issuecomment-221187825

ladislav-zezula commented 8 years ago

Btw, we are using StormLib on FreeBSD for years. :)

Ok. If you make changes to the makefiles for FreeBSD, let me know, I'll add them to the code base.

Unrelated but your site doesn't load on chrome on android (just eternal spinning gear)

Well, it does load on my Chrome on Android, pretty fast :-)

cen1 commented 8 years ago

Ok, apparently the only problem is FileStream.cpp and *64 stream functions. If I rename them to regular non-64 versions it compiles without problems as a Linux target. I am yet to find the info whether 64 bit headers are available on FreeBSD.

Edit: My impression is that *64 style functions should not be used at all. Instead, regular naming convention should be used together with _FILE_OFFSETBITS macro which uses 64bit behind the scenes. https://en.wikipedia.org/wiki/Stat(system_call)

Thoughts?

cen1 commented 8 years ago

@ladislav-zezula gentle poke

ladislav-zezula commented 8 years ago

(COMMENT DELETED)

ladislav-zezula commented 8 years ago

I am not a big fan of changing the current Linux code just because FreeBSD needs something else. The code there has been there for years and works. What about to add a new #ifdef blocks into FileStream.h and FileStream.cpp (and possibly StormPort.h):

ifdef PLATFORM_FREEBSD

// Add read/write/seek code for FreeBSD

endif

cen1 commented 8 years ago

Well, if my theory is correct, the changed code will work completely the same. Just needs to be tested. I found additional info for stat64: "The glibc stat() wrapper function hides these details from applications, invoking the most recent version of the system call provided by the kernel, and repacking the returned information if required for old binaries." So, if stat64() is available it will be used when stat() is used in the code.

Anyway, where do I find the test MPQ files to run the tests on (test/README.txt)? Are they available for download anywhere? I will prepare a pull request and you can let it bake for a while. I want to make sure all tests pass and I'll also use the patched version on our production ghost bots to see if anything was affected.

edit: I'll also make an #ifdef version just in case. edit2: specifically need to test on files>4GB

cen1 commented 8 years ago

Ok, I figured out the simplest solution for now: it is possible to override function names at compiler level, so a simple ifdef for freebsd in cmake should do it: set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -DO_LARGEFILE -Dstat64=stat -Dlstat64=lstat -Dlseek64=lseek -Doff64_t=off_t -Dfstat64=fstat -Dftruncate64=ftruncate

I'll prepare a pull.

ladislav-zezula commented 8 years ago

Merged.