64-bit builds for OSX using moduleset

totaam commented 9 years ago

Split from Xpra-org/xpra#533. See Xpra-org/xpra#533#comment:77

Minor fixes + tweaks already done: r9089, r9090.

Things that need fixing:

the moduleset needs updating, adding cups, etc as per the comments in Xpra-org/xpra#533.
fix the pyobjc build failure
detect when trying to run on a 32-bit system and provide a helpful error message
add new download links for it
nice-to-have: libvpx 1.4 (see failure in ticket/832#comment:1, still fails on 64-bit..)

That's it!

totaam commented 9 years ago

FWIW: I've uploaded a beta 0.15.0 build which works quite well, see https://xpra.org/beta/osx/x86_64/

@afarr: it would be worth playing with it a bit to see if there are any other issues - stability issues (or even improvements), or if there are any differences in performance, etc..

totaam commented 9 years ago

2015-09-24 23:06:04: smo commented

Updated in r10674 from upstream. We may still need to make further updates to this for our needs.

totaam commented 8 years ago

2015-10-27 21:16:05: smo commented

This is still on hold until we can get gstreamer up to date.

totaam commented 8 years ago

2015-10-29 18:41:09: smo commented

Can we reschedule this for a further release. Unless you think 64bit builds are not worth the bother and we can just close this.

totaam commented 8 years ago

2015-10-30 01:33:18: antoine commented

Re-scheduling. Worth doing, maybe after the next LTS so we have both 32-bit and 64-bit.

totaam commented 8 years ago

2015-11-05 23:50:40: smo commented

Tried this myself today on osx 10.9.5 didn't get as far as I hoped some weird issue here.

(cd /Users/user/gtk/inst/_jhbuild/root-tango-icon-theme/Users/user/gtk/inst/share/icons/Tango/16x16 && /Users/user/gtk/inst/libexec/icon-name-mapping -c actions)
Bizarre copy of UNKNOWN in scalar assignment at /Users/user/gtk/inst/lib/perl5/5.16.2/darwin-thread-multi-2level/XML/Parser/Expat.pm line 125.
make[3]: *** [install-data-local] Error 25
make[2]: *** [install-am] Error 2
make[1]: *** [install-recursive] Error 1
make: *** [install-recursive] Error 1
*** Error during phase install of tango-icon-theme: ########## Error running make install DESTDIR=/Users/user/gtk/inst/_jhbuild/root-tango-icon-theme *** [41/97]

If I run this utility I also get the same issue.

/Users/user/gtk/inst/libexec/icon-name-mapping -c actions
Bizarre copy of UNKNOWN in scalar assignment at /Users/user/gtk/inst/lib/perl5/5.16.2/darwin-thread-multi-2level/XML/Parser/Expat.pm line 125.

Doesn't appear to be a 64bit issue really just some sort of broken perl module?

totaam commented 8 years ago

2015-12-02 23:01:13: smo commented

Building on 10.11.1 with xcode 7.1 and these were the ones that failed
*** the following modules were not built *** [97/97]
libvpx sdl gst-plugins-bad-xpra pygtkglext meta-osx-xpra-deps
sdl was missing headers for X libvpx needs updating to 1.5.0 which builds fine need to update moduleset

gst-plugins-bad-xpra failed to build with an error about ssl header files we might be able to disable this
In file included from gstapexraop.c:33:
./gstapexraop.h:33:10: fatal error: 'openssl/rand.h' file not found
#include <openssl/rand.h>
         ^
In file included from gstapexplugin.c:29:
In file included from ./gstapexsink.h:32:
./gstapexraop.h:33:10: fatal error: 'openssl/rand.h' file not found
#include <openssl/rand.h>
pygtkglext failed with this error
gdkglext.override:36:10: fatal error: 'GL/gl.h' file not found
#include <GL/gl.h>

totaam commented 8 years ago

we don't need SDL
for libvpx, can we add a moduleset per version, or add one via the customrc file?
pygtkglext needs the opengl headers and libs to link against, which may have moved: Where are the OpenGL header files located on MacOSX?

totaam commented 8 years ago

2015-12-03 20:10:40: smo commented

gtk/gdkgl/gdkglext.c change to #include <OpenGL/gl.h> afterwards this builds no problem.

I'm not sure about having multiple modulesets at the moment I include it by url in customrc file but one of the files that gets included also has a url we include from svn trunk

totaam commented 8 years ago

2016-02-27 13:38:57: antoine commented

This should allow us to get libyuv on osx: Xpra-org/xpra#973#comment:3

totaam commented 8 years ago

Should be a priority for the next milestone.

totaam commented 8 years ago

2016-06-07 22:27:26: smo commented

Still having issues trying to do this now I think its an issue with jhbuild during the configure phase of x264 it produces this stacktrace

*** Configuring x264 *** [72/116]
Traceback (most recent call last):
  File "/Users/user/.local/bin/jhbuild", line 32, in <module>
    jhbuild.main.main(sys.argv[1:])
  File "/Users/user/Source/jhbuild/jhbuild/main.py", line 155, in main
    rc = jhbuild.commands.run(command, config, args, help=lambda: print_help(parser))
  File "/Users/user/Source/jhbuild/jhbuild/commands/__init__.py", line 183, in run
    return cmd.execute(config, args, help)
  File "/Users/user/Source/jhbuild/jhbuild/commands/__init__.py", line 52, in execute
    return self.run(config, options, args, help)
  File "/Users/user/Source/jhbuild/jhbuild/commands/base.py", line 268, in run
    return build.build()
  File "/Users/user/Source/jhbuild/jhbuild/frontends/buildscript.py", line 163, in build
    error, altphases = module.run_phase(self, phase)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/__init__.py", line 420, in run_phase
    method(buildscript)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/autotools.py", line 217, in do_configure
    cmd = self._get_configure_cmd(buildscript)
  File "/Users/user/Source/jhbuild/jhbuild/modtypes/autotools.py", line 130, in _get_configure_cmd
    cmd = self.static_analyzer_pre_cmd(buildscript) + template % vars
KeyError: u'libdir'

totaam commented 8 years ago

2016-06-16 16:18:20: antoine commented

Any progress? We can skip x264 and do it by hand if need be.

Blocker for Xpra-org/xpra#1231.

totaam commented 8 years ago

2016-06-16 23:17:20: smo commented

The problem is with libdir replacement in those strings maybe this isn't needed anymore since if I remove it I can compile this package through jhbuild.

I will commit fix for this shortly to trunk. Looks like a similar issue with ffmpeg as well.

totaam commented 8 years ago

2016-06-17 06:13:17: smo commented

my current .jhbuildrc-custom for el capitan machine

setup_sdk(target="10.9", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"
modules = [ "python", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python",
"gstreamer", "gst-plugins-base", "gst-plugins-good", "meta-gstreamer-1.0",
"meta-osx-xpra-deps"]

#skip iconv and add some switches to turn off stuff that was breaking
skip.append("libiconv")
#skip.append("expat")
append_autogenargs('gstreamer', '--enable-introspection=no')
append_autogenargs('gst-plugins-base', '--enable-introspection=no')
append_autogenargs('gst-plugins-good', '--enable-introspection=no --disable-deinterlace')
append_autogenargs('gst-plugins-bad-1.0', '--enable-opengl=no --disable-apple_media')

#change moduleset
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

totaam commented 8 years ago

2016-06-17 16:15:36: antoine commented

This does not look ready to me, or I need some more precise instructions.

Following, more or less: [https://winswitch.org/dev/macosx.html] but with the new customrc on a 10.10 system. Found some useful commands along the way:
$ sw_vers -productVersion
10.10.5
$ xcrun --show-sdk-version
10.11
$ xcrun --show-sdk-path
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
And now for the (long) list of problems:
during boostrap:
ERROR: no certificate subject alternative name matches
requested host name `cmake.org'.
more problematic during the build step:
*** Configuring itstool *** [5/116]
(..)
checking for python module libxml2... ./configure: line 2422: 16059 Done                    echo "import $py_module"
 16060 Abort trap: 6           | python - >&/dev/null
not found
configure: error: Python module libxml2 is needed to run this package
Probably the same cause as this other error:
Making all in doc
/Users/osx/gtk/inst/etc/xml/catalog:1: parser error : Document is empty
I/O error : Attempt to load network entity http://www.oasis-open.org/docbook/xml/4.1.2/docbookx.dtd
(..)
And another related one (as I found out much later..):
GISCAN   GLib-2.0.gir
Traceback (most recent call last):
File "./g-ir-scanner", line 53, in <module>
from giscanner.scannermain import scanner_main
File "./giscanner/scannermain.py", line 29, in <module>
import tempfile
File "/Users/osx/gtk/inst/lib/python2.7/tempfile.py", line 32, in <module>
import io as _io
File "/Users/osx/gtk/inst/lib/python2.7/io.py", line 51, in <module>
import _io
ImportError: No module named _io
make[2]: *** [GLib-2.0.gir] Error 1
make[2]: *** Waiting for unfinished jobs....
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of gobject-introspection: ########## Error running make -j 2  *** [27/116]
Same for gst-python and most python modules, pango (gtk-doc), gtk-mac-integration, librsvg, gtk-engines, gtk-quartz-engine .. It tries to build pycairo-python2.6 - that looks wrong. Looks like we want to force python to build earlier. I found that out later, after much time wasted. Most python modules then build OK. Maybe just a _gtk_osx_use_jhbuild_python = True line? Or maybe I was meant to build meta-osx-xpra-deps and not just a plain build?
the gi bindings end up missing, see Xpra-org/xpra#1204#comment:4

gst-plugins-bad errors out on openssl, we need to build openssl earlier

a lot of the sound codecs seem to get built after the gstreamer plugins, that looks wrong?

gtkglext errors out with the KeyError: u'libdir' problem.

gtk is still at 2.24.28?

gdk-pixbuf error: C compiler cannot create executables

librvg is missing gdk-pixbuf

gst-plugins-bad-1.0: errors on missing gir files

python-pyobjc-framework-cocoa errors on cannot find protocol declaration for 'NSProgressReporting', cannot find protocol declaration for 'NSURLSessionStreamDelegate'
spotted this in gst-libav, which looks wrong:
Configured with: --prefix=/Applications/Xcode.app/Contents/Developer/usr --with-gxx-include-dir=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk/usr/include/c++/4.2.1
lots of other modules need to be updated for https (and forced to use curl for download?):

hicolor-icon-theme now lives here: [https://icon-theme.freedesktop.org/releases/hicolor-icon-theme-0.13.tar.gz]

cairo: [https://www.cairographics.org/releases/cairo-1.14.0.tar.xz]

py2cairo: [https://www.cairographics.org/releases/py2cairo-1.10.0.tar.bz2]

liboil: [https://liboil.freedesktop.org/download/liboil-0.3.17.tar.gz]

all the gstreamer downloads, ie: orc got moved to [https://gstreamer.freedesktop.org/src/orc/orc-0.4.25.tar.xz]

Cython: [https://pypi.python.org/packages/b1/51/bd5ef7dff3ae02a2c6047aa18d3d06df2fb8a40b00e938e7ea2f75544cac/Cython-0.24.tar.gz]

pkg-config

pixman

it rebuilds openssl every time, which makes things painfully slow

subversion has even more problems

the opengl build error can be solved with ln -sf /System/Library/Frameworks/OpenGL.framework/Headers /Users/osx/gtk/inst/include/GL

totaam commented 8 years ago

2016-07-04 10:25:50: antoine commented

At some point we will have to switch the existing build environment to a locked version, and move forward with the trunk / 64-bit / newer build env. Until then, r12959 fixes building ffmpeg but this may also get in the way of supporting webcam capture in the future (we can't have both..).

totaam commented 8 years ago

2016-07-12 17:52:23: antoine commented

Milestone renamed

totaam commented 8 years ago

2016-08-27 05:18:08: antoine commented

FYI: bumped to openssl 1.1.0 in r13481, which requires a newer perl version than the one available on my 10.5.x VM, so then I had to figure out how to build the latest stable perl there:
tar -zxvf perl-5.24.0.tar.gz 
cd perl-5.24.0
./Configure -des -Dprefix={JHBUILD_PREFIX} \
    -Accflags="-arch i686 -I{JHBUILD_PREFIX}/include/ -L{JHBUILD_PREFIX}/lib/" \
    -Aldflags="-arch i686 -I{JHBUILD_PREFIX}/include/ -L{JHBUILD_PREFIX}/lib/"
make && make install
Bonus: this version seems to install cleanly without needing a chmod. Downside: python-cryptography needs updating to 1.5 to support openssl 1.1 : [https://cryptography.io/en/latest/changelog/]. (done in r13482)

@smo: if that works for you, please add perl to the moduleset.

totaam commented 8 years ago

2016-09-09 01:39:12: smo commented

Okay I ran into the perl issue as well.

I will add this to the moduleset but should we require it or leave it as an optional thing on older systems that need it?

totaam commented 8 years ago

2016-09-09 03:13:48: antoine commented

Hopefully we will be moving to a newer 64-bit build system soon, so this should be optional.

totaam commented 8 years ago

2016-09-13 20:33:57: smo commented

r13692:13695 adds perl

Was some issues with the configure line since it is non standard. Not sure if I need to add the cc and ld flags so I left them out for now.

If the new openssl seems to install properly i'll see if I can remove some of the hacks from the moduleset for installing it in the first place.

totaam commented 8 years ago

2016-09-20 23:48:37: smo commented

Tried with the latest changes to the moduleset on 64 bit. Seems to get further now. I got this error today
build/temp.macosx-10.9-x86_64-2.7/_openssl.c:415:10: fatal error: 'openssl/opensslv.h' file not found
#include <openssl/opensslv.h>
         ^
1 error generated.
error: command '/usr/bin/gcc' failed with exit status 1
*** Error during phase build of python-cryptography: ########## Error running python setup.py build *** [106/119]
Confirmed that openssl is installed.

totaam commented 8 years ago

2016-09-27 10:47:05: antoine changed priority from major to blocker

totaam commented 8 years ago

2016-09-27 10:47:05: antoine commented

Using 10.10+ as a build platform will also allow us to fix Xpra-org/xpra#404 and Xpra-org/xpra#1305

totaam commented 8 years ago

2016-10-13 17:49:21: antoine commented

Tried again as part of Xpra-org/xpra#1340:

got an error with librsvg not finding gdk-pixbuf gir

ffmpeg complained about missing libvpx

occasionally had to switch things around with code-select: jhbuild requires the developer path but compiling things require the command line tools.. PITA - how do get get both to work at the same time?

totaam commented 7 years ago

2016-10-28 11:02:42: antoine commented

Also a blocker for Xpra-org/xpra#1340.

totaam commented 7 years ago

2016-11-02 02:16:31: antoine commented

gstreamer 1.10: [https://gstreamer.freedesktop.org/releases/1.10/].

totaam commented 7 years ago

2016-12-16 19:45:05: antoine commented

After moving to a newer build host, we can start using a number of new APIs and do:

Xpra-org/xpra#776

webcam support etc

totaam commented 7 years ago

2016-12-28 08:42:03: antoine commented

I have pointed my 10.5.x build VM to the stable 1.0.x branch after merging the latest set of safe updates (14622). Now we can start "breaking" things again to make 64-bit builds work properly, I am building on 10.10.x with xcode 7.2.1:

r14623: libvpx 1.6.0 + ffmpeg webcam (+ r14625 to fix the same error as comment:15, fix hinted in comment:17 but without a link to the actual commit)

r14624: openssl 1.1.x now that we have a newer python 2.7.3 (r14619)

drop gstreamer 0.10: r14628 + r14629, r14630 + r14632

64bit package naming: r14633

After that:
jhbuild buildone -f openssl to make sure it is rebuilt before python (this should probably be in the moduleset instead)

jhbuild build -f - and wait a while (see minor fixes below)
make a signed build in one swoop (based on the instructions in Xpra-org/xpra#1340#comment:19):
security unlock-keychain -p THEACCOUNTPASSWORD
export CODESIGN_KEYNAME="Antoine Martin"
./make-all.sh
the resulting DMG / PKG are not much smaller than before: [http://xpra.org/beta/osx]
Things left TODO:
openssl is still a pain to build + install, still requires
chmod +x ${JHBUILD_PREFIX}/lib/libssl.* ${JHBUILD_PREFIX}/lib/libcrypto.*
the gi bindings are missing and we really need those for gstreamer.. but [ https://github.com/anholt/libepoxy libepoxy] won't build (could be related to Adds support for OS X without X11), and it is required by GTK3, which is needed by the gobject-introspection module... I still managed to build it with jhbuild buildone -f pygobject3

gstreamer 1.10.x (version bumped in r14627, fix for gst-plugins-bad in r14631): downloads fail because they redirect to https and we get a certificate issue, again (as found in comment:19)

speex codec is missing, needs adding to moduleset?
pyobjc-core moaned:
error: SDK root '/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk /Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk' does not exist
Not sure why it has the same path 3 times as the command it runs to get that path only returns it once, but running like this allows me to move on:
export SDKROOT=/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.11.sdk
python ./setup.py build
python ./setup.py install
IIRC, some versions of glib / gi were held back for compatibility reasons (with what?) - can we move forward again?
Note: there is a weird setup issue on my 10.10.x system which forces me to switch to command line tools to ensure the build finds gcc / clang but switch back to "developer" mode to launch jhbuild:

sudo xcode-select -switch /Applications/Xcode.app/Contents/Developer

sudo xcode-select -switch /Library/Developer/CommandLineTools

totaam commented 7 years ago

2016-12-30 03:15:18: antoine commented

Updates:

r14648 removes libdir from gtkglext

r14647 fixes mpfr download link, r14649 uses a reliable link (one that will not expire when the next release comes out), r14650 (undoes an unwanted change)

r14646 opus update to 1.1.3

r14645 fixes openssl install (no longer installed every time or needing chmod)

r14651 + r14652 + r14653: use "1.0" suffix for gstreamer modules, enable gobject-introspection

totaam commented 7 years ago

2016-12-30 04:08:58: antoine commented

r14656 updates to libpng 1.6.27 following fix a potential "NULL dereference" bug

r14660 adds speex

Some of these updates have been applied to the v1.0.x branch: 14657 (libpng), 14658 (mpfr), 14659 (opus)

TODO:

pygobject3 dependency? (build gtk3, surely that can be fixed?)

gi / glib updates?

Will ignore: pyobjc and other build env weirdness for now.

totaam commented 7 years ago

2017-01-14 05:56:47: smo commented

Everything builds for me on this machine

sw_vers && xcodebuild -version
ProductName:  Mac OS X
ProductVersion:   10.12.1
BuildVersion: 16B2555
Xcode 8.2.1
Build version 8C1002

totaam commented 7 years ago

2017-01-14 06:02:41: smo commented

Also works on this system

sw_vers && xcodebuild -version
ProductName:    Mac OS X
ProductVersion: 10.11.1
BuildVersion:   15B42
Xcode 7.1
Build version 7B91b

I put nasm in the modules part of the config because libjpeg-turbo will need it. using this config ~/.jhbuildrc-custom

_gtk_osx_use_jhbuild_python = True
setup_sdk(target="10.9", sdk_version="10.11", architectures=["x86_64"])
os.environ["CC"] = "/usr/bin/gcc"
os.environ["DYLD_LIBRARY_PATH"] = ""
build_policy = "updated-deps"

modules = [ "openssl", "python", "yasm", "nasm", "libxml2", "itstool", "gtk-doc",
"meta-gtk-osx-bootstrap", "meta-gtk-osx-core",
"libcroco", "librsvg","meta-gtk-osx-python", "meta-gtk-osx-themes",
"gtk-quartz-engine","gtk-mac-integration-python", "meta-osx-xpra-deps"]

#change moduleset  
moduleset="http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"

totaam commented 7 years ago

2017-01-14 06:21:36: antoine changed status from new to closed

totaam commented 7 years ago

2017-01-14 06:21:36: antoine set resolution to fixed

totaam commented 7 years ago

2017-01-14 06:21:36: antoine commented

Works for me too (doing another full rebuild from scratch to check), minus the weirdness of having to run xcodeselect on this build system. (PITA but not a showstopper)

We can follow up in Xpra-org/xpra#1404. The next release will be 64-bit only for OSX! (and targetting 10.9+ only)

totaam commented 5 years ago

2019-10-21 03:37:33: antoine commented

webcam: see Xpra-org/xpra#1231

Xpra-org / gtk-osx-build