Creating an Arch Linux PKGBUILD

[mxmilkiib]

Following on from stsaz/ff-3pt#1 in the correct project;

Might you have a programmatic method of reproducing the build?

I was looking to create an Arch Linux PKGBUILD for fmedia. From a general perspective, a vcs (rather than bin) PKGBUILD is easier because the package version can be taken automatically from the git repo (on install), where as bin PKGBUILDs need to be manually updated by the maintainer (or edited by the user) for each new release.

But if you're not building for the latest versions of libraries (targeted by rolling release distros), might you be using distro source packages? Arch might have source or bin versions (mostly in the AUR) available that sometimes relate to other distros or such.

Might the use of git submodules to retrieving library code be something you would be interested in? It could be done recursively from the fmedia repo.

Here's my initial PKGBUILD without moving/patching, just went through with the specific versions, I'll come back to that over the next few days. Already I noticed flac dep. matches the Arch Extra repos version so the source could be copied. I might make a binary version too, for kicks/experience.

[mxmilkiib]

The PKGBUILD is updated, part way through sorting the patching now.

I seen to be having issues with Monkey's Audio codec. I'm trying straight from MAC_SDK_433.zip, but get the following;

patching file Source/Console/Console.cpp
Hunk #1 FAILED at 13 (different line endings).
1 out of 1 hunk FAILED -- saving rejects to file Source/Console/Console.cpp.rej
patching file Source/Shared/NoWindows.h
Hunk #1 FAILED at 49 (different line endings).
1 out of 1 hunk FAILED -- saving rejects to file Source/Shared/NoWindows.h.rej
patching file Source/Shared/GlobalFunctions.cpp
Hunk #1 FAILED at 5 (different line endings).
Hunk #2 FAILED at 120 (different line endings).
2 out of 2 hunks FAILED -- saving rejects to file Source/Shared/GlobalFunctions.cpp.rej
patching file Source/MACLib/APEDecompress.cpp
Hunk #1 FAILED at 60 (different line endings).
Hunk #2 FAILED at 75 (different line endings).
Hunk #3 FAILED at 279 (different line endings).
Hunk #4 FAILED at 288 (different line endings).
Hunk #5 FAILED at 297 (different line endings).
Hunk #6 FAILED at 313 (different line endings).
Hunk #7 FAILED at 324 (different line endings).
Hunk #8 FAILED at 336 (different line endings).
Hunk #9 FAILED at 345 (different line endings).
Hunk #10 FAILED at 363 (different line endings).
Hunk #11 FAILED at 376 (different line endings).
11 out of 11 hunks FAILED -- saving rejects to file Source/MACLib/APEDecompress.cpp.rej
patching file Source/MACLib/APEDecompress.h
Hunk #1 FAILED at 17 (different line endings).
Hunk #2 FAILED at 25 (different line endings).
Hunk #3 FAILED at 68 (different line endings).
3 out of 3 hunks FAILED -- saving rejects to file Source/MACLib/APEDecompress.h.rej
patching file Source/MACLib/UnBitArray.cpp
Hunk #1 FAILED at 33 (different line endings).
1 out of 1 hunk FAILED -- saving rejects to file Source/MACLib/UnBitArray.cpp.rej
patching file Source/MACLib/UnBitArrayBase.cpp
Hunk #1 FAILED at 6 (different line endings).
Hunk #2 FAILED at 14 (different line endings).
Hunk #3 FAILED at 26 (different line endings).
Hunk #4 FAILED at 42 (different line endings).
Hunk #5 FAILED at 66 (different line endings).
Hunk #6 FAILED at 181 (different line endings).
6 out of 6 hunks FAILED -- saving rejects to file Source/MACLib/UnBitArrayBase.cpp.rej
patching file Source/MACLib/UnBitArrayBase.h
Hunk #1 FAILED at 41 (different line endings).
Hunk #2 FAILED at 55 (different line endings).
2 out of 2 hunks FAILED -- saving rejects to file Source/MACLib/UnBitArrayBase.h.rej
patching file Source/Shared/All.h
Hunk #1 FAILED at 60 (different line endings).
Hunk #2 FAILED at 130 (different line endings).
2 out of 2 hunks FAILED -- saving rejects to file Source/Shared/All.h.rej

[stsaz]

yeah, MAC uses windows line endings. Try this command to patch it:

unix2dos *.patch
cat *.patch | patch -p1 --ignore-whitespace --binary

First you convert .patch files to windows line endings, and then apply patches with --ignore-whitespace --binary.

From the quick look on your script I can say that several libs are not needed for fmedia (e.g. sqlite and lzma). Only these libs are the real dependencies: libALAC, libfdk-aac, libFLAC, libMAC, libmp3lame, libmpg123, libmpc, libogg, libopus, libsoxr, libvorbisenc, libvorbis, libwavpack, libDynamicAudioNormalizer. Unfortunately I don't have a script that can easily download, patch and build all of them at once.

[mxmilkiib]

dos2unix does the job, thanks :)

sqlite3.h isn't available separately, arch official is 3.22, and the version required is 3.16.

What about lzma/0001-x86-filter.patch etc.?

Which version of libopus?

Hmm, I'm getting stuck on the second mpg123 patch for mpg123.h. The output from makepkg, which constructs the package from the PKGBUILD, is

...
/home/milk/pkgs/fmedia-git/src/ff-3pt/mpg123
patching file src/libmpg123/libmpg123.c
patching file src/libmpg123/readers.c
can't find file to patch at input line 88
Perhaps you used the wrong -p or --strip option?
The text leading up to this was:
--------------------------
|-- 
|2.7.4
|
|--- src/libmpg123/mpg123.h 2017-07-18 10:18:46.000000000 +0300
|+++ src/libmpg123/mpg123.h 2017-07-25 22:27:38.322495340 +0300
--------------------------
File to patch: src/libmpg123/mpg123.h
patching file src/libmpg123/mpg123.h
Hunk #1 FAILED at 17.
Hunk #2 FAILED at 49.
2 out of 4 hunks FAILED -- saving rejects to file src/libmpg123/mpg123.h.rej
...

mpg123 svn revision 4297 is marked as version 1.25.3, the noted required version, so I'm confused.

[stsaz]

sqlite and lzma are not needed for fmedia, so there's no need to build them!

I used opus 1.1.3 but I think any version will work, because there are no patches for it.

mpg123.h file is a copy of mpg123.h.in and it's patched by mpg123/Makefile:

src/libmpg123/mpg123.h: src/libmpg123/mpg123.h.in
    cp -v $< $@
    patch -p0 < $(FF3PT)/mpg123/mpg123.h.patch 

How does makepkg script work? Did you configure it to compile my mpg123-ff.c and all other similar files (those that end with -ff) too? Those files are needed to be compiled into the libs in order to be used by fmedia.

[mxmilkiib]

I've not gotten as far as addressing make/the Makefiles. The Arch wiki has a Creating packages article. I edit the PKGBUILD, run makepkg -C in the same directory, see what kind of error comes up next.

After fmedia has been built, an install is done to a directory that gets compressed into a package for install on an Arch [based] system. After that, clone an empty fmedia-git from the AUR, drop in the PKGBUILD (and generate an .SRCINFO), git add/commit/push, and it's available in the AUR.

[mxmilkiib]

I'm addressing make/the Makefiles now :) What shell language are your README commands in? In zsh, from the root of ff-3pt, I tried;

make -rRf **/Makefile FF3PT=.

and

make -rRf **/Makefile FF3PT=`pwd`

but get

make: Nothing to be done for 'dynanorm/Makefile'.
make: Nothing to be done for 'fdk-aac/Makefile'.
make: Nothing to be done for 'flac/Makefile'.
make: Nothing to be done for 'jpeg/Makefile'.
make: Nothing to be done for 'lzma/Makefile'.
make: Nothing to be done for 'mac/Makefile'.
make: Nothing to be done for 'mad/Makefile'.
make: Nothing to be done for 'mp3lame/Makefile'.
make: Nothing to be done for 'mpg123/doc/examples/Makefile'.
make: Nothing to be done for 'mpg123/Makefile'.
make: Nothing to be done for 'musepack/Makefile'.
make: Nothing to be done for 'ogg/Makefile'.
make: Nothing to be done for 'opus/celt/dump_modes/Makefile'.
make: Nothing to be done for 'opus/Makefile'.
make: Nothing to be done for 'png/Makefile'.
make: Nothing to be done for 'soxr/Makefile'.
make: Nothing to be done for 'sqlite/Makefile'.
make: Nothing to be done for 'vorbis/Makefile'.
make: Nothing to be done for 'wavpack/Makefile'.
make: Nothing to be done for 'zlib/contrib/blast/Makefile'.
make: Nothing to be done for 'zlib/contrib/minizip/Makefile'.
make: Nothing to be done for 'zlib/contrib/puff/Makefile'.
make: Nothing to be done for 'zlib/contrib/untgz/Makefile'.
make: Nothing to be done for 'zlib/Makefile'.
make: Nothing to be done for 'zlib/nintendods/Makefile'.

[stsaz]

The Makefile files from ff-3pt repo can be executed like this:

cd /src/flac-1.3.1/
make -rRf /ff-3pt/flac/Makefile
ls -l *.so

First, you enter the directory with the library's original source code and apply patches from ff-3pt repo. Then you call the appropriate Makefile from ff-3pt repo for that library.

I started using my own makefiles to throw away all the unneeded stuff from library code and to compile in a clear and simple API. The downside is that an update to a new library version can be difficult.

[mxmilkiib]

So I'm now leaving the downloaded library source in the src directory where makepkg deposits it, cding in and calling make -rRf ../ff-3pt/fdk-aac/Makefile FF3PT=../ff-3pt but that gets me an error;

...
g++ -c -pipe -O3 -flto -fno-asynchronous-unwind-tables -fpic -fvisibility=hidden -m64 -march=x86-64 -fno-exceptions -fno-rtti -IlibAACdec/include -IlibMpegTPDec/include -IlibSBRdec/include -IlibAACenc/include -IlibMpegTPEnc/include -IlibSBRenc/include -IlibSYS/include -IlibFDK/include -IlibPCMutils/include -Wno-narrowing libFDK/src/qmf.cpp -olibFDK/src/qmf.o
g++ -c -pipe -O3 -flto -fno-asynchronous-unwind-tables -fpic -fvisibility=hidden -m64 -march=x86-64 -fno-exceptions -fno-rtti -IlibAACdec/include -IlibMpegTPDec/include -IlibSBRdec/include -IlibAACenc/include -IlibMpegTPEnc/include -IlibSBRenc/include -IlibSYS/include -IlibFDK/include -IlibPCMutils/include -Wno-narrowing libFDK/src/scale.cpp -olibFDK/src/scale.o
make: *** No rule to make target 'libPCMutils/src/limiter.o', needed by 'libfdk-aac-ff.so'.  Stop.
==> ERROR: A failure occurred in prepare().

[stsaz]

Are you trying to build fdk-aac-0.1.4 ? I've just checked it and it builds flawlessly on my machine. Newer versions (e.g. 0.1.5) can't be built with my Makefile.

[stsaz]

I've added a new makefile which can be used to automatically build all audio libs for fmedia, see ff-3pt/Makefile. I think you can copy those instructions to your script. ff-3pt/README file explains how to run it.

[mxmilkiib]

Awesome, thanks!

There's an error when compiling FLAC;

...
gcc -c -pipe -O3 -flto -fno-asynchronous-unwind-tables -fpic -fvisibility=hidden -m64 -march=x86-64 -funroll-loops -I. -Isrc/libFLAC/include -Iinclude -I/home/milk/pkgs/fmedia-git/src/ff-3pt/flac -DHAVE_CONFIG_H -DFLAC_EXP -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wstrict-prototypes -Waggregate-return -Wcast-align -Wnested-externs -Wshadow -Wundef -Wunreachable-code -Winline -Wdeclaration-after-statement -Wno-unused-function -Wno-unused-variable -Wno-unused-parameter -Wno-implicit-fallthrough src/libFLAC/window.c  -osrc/libFLAC/window.o
gcc FLAC-ff.o src/libFLAC/bitmath.o src/libFLAC/bitreader.o src/libFLAC/bitwriter.o src/libFLAC/cpu.o src/libFLAC/crc.o src/libFLAC/fixed.o src/libFLAC/fixed_intrin_sse2.o src/libFLAC/fixed_intrin_ssse3.o src/libFLAC/float.o src/libFLAC/format.o src/libFLAC/lpc.o src/libFLAC/lpc_intrin_sse.o src/libFLAC/lpc_intrin_sse2.o src/libFLAC/lpc_intrin_sse41.o src/libFLAC/lpc_intrin_avx2.o src/libFLAC/md5.o src/libFLAC/memory.o src/libFLAC/stream_decoder.o src/libFLAC/stream_encoder.o src/libFLAC/stream_encoder_intrin_sse2.o src/libFLAC/stream_encoder_intrin_ssse3.o src/libFLAC/stream_encoder_intrin_avx2.o src/libFLAC/stream_encoder_framing.o src/libFLAC/window.o -pipe -s -O3 -flto -fno-asynchronous-unwind-tables -fpic -Wl,-no-undefined -m64 -march=x86-64 -static-libgcc -lm -shared -olibFLAC-ff.so
/var/tmp/ccKHTR7A.ltrans0.ltrans.o: In function `flac_encode_init':
<artificial>:(.text+0x1e6bc): undefined reference to `FLAC__ogg_encoder_aspect_set_defaults'
/var/tmp/ccKHTR7A.ltrans0.ltrans.o: In function `flac_decode_init':
<artificial>:(.text+0x2887f): undefined reference to `FLAC__ogg_decoder_aspect_set_defaults'
<artificial>:(.text+0x28bb5): undefined reference to `FLAC__ogg_decoder_aspect_flush'
<artificial>:(.text+0x28be2): undefined reference to `FLAC__ogg_decoder_aspect_reset'
/var/tmp/ccKHTR7A.ltrans1.ltrans.o: In function `read_callback_.lto_priv.22':
<artificial>:(.text+0xcc6b): undefined reference to `FLAC__ogg_decoder_aspect_read_callback_wrapper'
/var/tmp/ccKHTR7A.ltrans1.ltrans.o: In function `FLAC__stream_decoder_delete':
<artificial>:(.text+0x135e5): undefined reference to `FLAC__ogg_decoder_aspect_set_defaults'
<artificial>:(.text+0x13629): undefined reference to `FLAC__ogg_decoder_aspect_finish'
collect2: error: ld returned 1 exit status
make[2]: *** [/home/milk/pkgs/fmedia-git/src/ff-3pt/flac/Makefile:59: libFLAC-ff.so] Error 1
make[2]: Leaving directory '/tmp/ff3pt-build/flac-1.3.2'
make[1]: *** [/home/milk/pkgs/fmedia-git/src/ff-3pt/flac/Makefile:56: all] Error 2
make[1]: Leaving directory '/tmp/ff3pt-build/flac-1.3.2'
make: *** [Makefile:61: flac] Error 2

[stsaz]

There comes the difference from running the original ./configure script. Try this patch to explicitly disable OGG support: fix-flac.txt

[mxmilkiib]

Patched, and there's undefined reference to `_flac_encode_info', etc. now;

...
gcc -c -pipe -O3 -flto -fno-asynchronous-unwind-tables -fpic -fvisibility=hidden -m64 -march=x86-64 -funroll-loops -I. -Isrc/libFLAC/include -Iinclude -I/home/milk/pkgs/fmedia-git/src/ff-3pt/flac -DHAVE_CONFIG_H -DFLAC_EXP -D_FORTIFY_SOURCE=2 -Wall -Wextra -Wstrict-prototypes -Waggregate-return -Wcast-align -Wnested-externs -Wshadow -Wundef -Wunreachable-code -Winline -Wdeclaration-after-statement -Wno-unused-function -Wno-unused-variable -Wno-unused-parameter -Wno-implicit-fallthrough src/libFLAC/window.c  -osrc/libFLAC/window.o
gcc FLAC-ff.o src/libFLAC/bitmath.o src/libFLAC/bitreader.o src/libFLAC/bitwriter.o src/libFLAC/cpu.o src/libFLAC/crc.o src/libFLAC/fixed.o src/libFLAC/fixed_intrin_sse2.o src/libFLAC/fixed_intrin_ssse3.o src/libFLAC/float.o src/libFLAC/format.o src/libFLAC/lpc.o src/libFLAC/lpc_intrin_sse.o src/libFLAC/lpc_intrin_sse2.o src/libFLAC/lpc_intrin_sse41.o src/libFLAC/lpc_intrin_avx2.o src/libFLAC/md5.o src/libFLAC/memory.o src/libFLAC/stream_decoder.o src/libFLAC/stream_encoder.o src/libFLAC/stream_encoder_intrin_sse2.o src/libFLAC/stream_encoder_intrin_ssse3.o src/libFLAC/stream_encoder_intrin_avx2.o src/libFLAC/stream_encoder_framing.o src/libFLAC/window.o -pipe -s -O3 -flto -fno-asynchronous-unwind-tables -fpic -Wl,-no-undefined -m64 -march=x86-64 -static-libgcc -lm -shared -olibFLAC-ff.so
/var/tmp/ccv1Ct5g.ltrans0.ltrans.o: In function `flac_encode_info':
<artificial>:(.text+0x13c18): undefined reference to `_flac_encode_info'
/var/tmp/ccv1Ct5g.ltrans0.ltrans.o: In function `flac_encode_init':
<artificial>:(.text+0x150d6): undefined reference to `_flac_encode_init'
/var/tmp/ccv1Ct5g.ltrans0.ltrans.o: In function `flac_decode_init':
<artificial>:(.text+0x15464): undefined reference to `_flac_decode_init'
<artificial>:(.text+0x154b9): undefined reference to `_flac_decode_info'
/var/tmp/ccv1Ct5g.ltrans0.ltrans.o: In function `flac_encode':
<artificial>:(.text+0x13c04): undefined reference to `_flac_encode'
/var/tmp/ccv1Ct5g.ltrans0.ltrans.o: In function `flac_decode':
<artificial>:(.text+0x15164): undefined reference to `_flac_decode'
collect2: error: ld returned 1 exit status
make[2]: *** [/home/milk/pkgs/fmedia-git/src/ff-3pt/flac/Makefile:59: libFLAC-ff.so] Error 1
make[2]: Leaving directory '/tmp/ff3pt-build/flac-1.3.2'
make[1]: *** [/home/milk/pkgs/fmedia-git/src/ff-3pt/flac/Makefile:56: all] Error 2
make[1]: Leaving directory '/tmp/ff3pt-build/flac-1.3.2'
make: *** [Makefile:61: flac] Error 2

[stsaz]

These functions are added by patches 0001-new-decoder-interface.patch and 0002-new-encoder-interface.patch. Please check whether they are applied correctly (it should be done automatically by flac/Makefile right after ./configure script is executed).

[mxmilkiib]

Getting there :+1:

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:../ff-3pt/_bin/linux-amd64
make install

gets

gcc -c -pipe -std=c99 -fno-strict-aliasing -Wno-deprecated-declarations -g -O3 -flto -fno-asynchronous-unwind-tables -fpic -fvisibility=hidden -m64 -march=x86-64 -pthread -Werror -Wall -Wextra -Wno-unused-parameter -I../ffos  ../ffos/FFOS/fflinux.c -off-obj/fflinux.o
In file included from ../ffos/FFOS/socket.h:30:0,
                 from ../ffos/FFOS/queue.h:8,
                 from ../ffos/FFOS/asyncio.h:7,
                 from ../ffos/FFOS/sig.h:7,
                 from ../ffos/FFOS/fflinux.c:9:
../ffos/FFOS/fflinux.c: In function ‘ffskt_sendfile’:
../ffos/FFOS/unix/skt.h:62:39: error: implicit declaration of function ‘writev’; did you mean ‘write’? [-Werror=implicit-function-declaration]
 #define ffskt_sendv(sk, iov, iovcnt)  writev(sk, iov, iovcnt)
                                       ^
../ffos/FFOS/fflinux.c:125:7: note: in expansion of macro ‘ffskt_sendv’
   r = ffskt_sendv(sk, hdtr->headers, hdtr->hdr_cnt);
       ^~~~~~~~~~~
cc1: all warnings being treated as errors
make: *** [../ffos/makerules:36: ff-obj/fflinux.o] Error 1

[stsaz]

Try this patch: fix-linux-writev.txt

It seems that your system's sys/socket.h doesn't include sys/uio.h internally. Anyway, it's good that you've found this. The patch includes it explicitly, should help.

[mxmilkiib]

Now the make install fails with;

...
gcc -c -pipe -std=c99 -fno-strict-aliasing -Wno-deprecated-declarations -g -O3 -flto -fno-asynchronous-unwind-tables -fpic -fvisibility=hidden -m64 -march=x86-64 -Werror -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-implicit-fallthrough -I../ff -I../ff-3pt -I../ffos  ../ff/FF/audio/ffaac.c -off-obj/ffaac.o
gcc -c -pipe -std=c99 -fno-strict-aliasing -Wno-deprecated-declarations -g -O3 -flto -fno-asynchronous-unwind-tables -fpic -fvisibility=hidden -m64 -march=x86-64 -Werror -Wall -Wextra -Wno-unused-parameter -Wno-missing-field-initializers -Wno-implicit-fallthrough -I../ff -I../ff-3pt -I../ffos  ../ff/FF/aformat/ffaac-adts.c -off-obj/ffaac-adts.o
gcc -shared ./aac.o ./aac-adts.o ./ff-obj/ffos.o ./ff-obj/fflinux.o ./ff-obj/ffunix.o ./ff-obj/ffarray.o ./ff-obj/ffstring.o ./ff-obj/ffnumber.o ./ff-obj/ffdbg.o ./ff-obj/ffutf8.o ./ff-obj/ffaac.o ./ff-obj/ffaac-adts.o ./ff-obj/ffpcm.o -pipe -g -O3 -flto -fno-asynchronous-unwind-tables -fpic -Wl,-no-undefined -m64 -march=x86-64 -static-libgcc -L../ff-3pt-bin/linux-amd64  '-Wl,-rpath,$ORIGIN' -lfdk-aac-ff  -oaac.so
/usr/bin/ld: cannot find -lfdk-aac-ff
collect2: error: ld returned 1 exit status
make: *** [../fmedia/makerules:297: aac.so] Error 1
==> ERROR: A failure occurred in build().

[stsaz]

-L../ff-3pt-bin/linux-amd64 - this means that it searches for libfdk-aac-ff.so in that directory. Check whether third-party libs are there. If not, you should copy them from ../ff-3pt/_bin/linux-amd64. Or you can specify -DFF3PTLIB=../ff-3pt/_bin/linux-amd64 flag to make.

[mxmilkiib]

Thanks for all the assistance, fmedia-git is now in the AUR.