osx moduleset changes tracker ticket

totaam commented 7 years ago

Tracking the latest changes and recording why / how things work the way they do.

Related to Xpra-org/xpra#533 (original moduleset ticket), Xpra-org/gtk-osx-build#13 (64-bit), Xpra-org/xpra#641 (osx packaging) Recent osx tickets: Xpra-org/xpra#1271 (websocket client changes) Moduleset updates and build errors: Xpra-org/xpra#1007 and Xpra-org/xpra#544 (misc lib updates), Xpra-org/xpra#971 (build failures), r9116, .. Sound changes: Xpra-org/xpra#970 (switch to gstreamer 1.x) Xpra-org/xpra#1069 (gstreamer 1.6), Xpra-org/xpra#1168 (gstreamer 1.8)

Recent changes: r13715, r13714, r13713, r13712, r13711, r13710, r13709, r13708, r13707, r13706, r13705, r13704, r13703, r13702, r13701, r13700, r13699, r13698, r13697, r13696, r13695, r13694, r13693, r13692

Got some errors updating today:

python tries to build the outdated 2.7.10 version and fails:

xzcat -d "/Users/osx/gtk/source/pkgs/Python-2.7.10.tar.xz" | tar xf -
*** Error during phase checkout of python: could not download patch (error: 404) *** [3/119]

(skipped)

libjpeg turbo is using an older version - why? (I have 1.5 build on 10.5.x so I've skipped this change too)

gtk+ fails (patch not needed?):

*** Applying patch http://git.gnome.org/browse/gtk-osx/plain/patches/0001-Bug-707945-GTK2-Quartz-typeahead-find-in-GtkTreeView.patch *** [31/119]
patch -p1 < "/Users/osx/.cache/jhbuild/0001-Bug-707945-GTK2-Quartz-typeahead-find-in-GtkTreeView.patch"
patching file modules/input/imquartz.c
Reversed (or previously applied) patch detected!  Assume -R? [n]

are there any other modules we are downgrading because of this upstream merge?

libffi fails to build:

libtool: compile:  /usr/bin/gcc-4.2 -DHAVE_CONFIG_H -I. -I. -I./include -Iinclude -I./src -I/Users/osx/gtk/inst/include -arch i386 -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -I/Users/osx/gtk/inst/include -arch i386 -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -isysroot /Developer/SDKs/MacOSX10.5.sdk -isysroot /Developer/SDKs/MacOSX10.5.sdk -I. -I./include -Iinclude -I./src -arch i386 -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -arch i386 -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -MT src/x86/win32.lo -MD -MP -MF src/x86/.deps/win32.Tpo -c src/x86/win32.S  -fno-common -DPIC -o src/x86/.libs/win32.o
src/x86/win32.S:1325:section difference relocatable subtraction expression, ".LFE5" minus ".LFB5" using a symbol at the end of section will not produce an assembly time constant
src/x86/win32.S:1325:use a symbol with a constant value created with an assignment instead of the expression, L_const_sym = .LFE5 - .LFB5
src/x86/win32.S:1317:section difference relocatable subtraction expression, ".LEFDE5" minus ".LASFDE5" using a symbol at the end of section will not produce an assembly time constant
src/x86/win32.S:1317:use a symbol with a constant value created with an assignment instead of the expression, L_const_sym = .LEFDE5 - .LASFDE5
src/x86/win32.S:unknown:missing indirect symbols for section (__IMPORT,__jump_table)
make[2]: *** [src/x86/win32.lo] Error 1
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of libffi: ########## Error running make -j 2 *** [16/119]

pixman fails to build:

CCLD     infinite-loop
Undefined symbols:
"_blend_soft_light", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_darken", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_hard_light", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_difference", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_exclusion", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_overlay", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_color_dodge", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_screen", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_multiply", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_color_burn", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
"_blend_lighten", referenced from:
  _blend_funcs.10173 in libutils.a(utils.o)
ld: symbol(s) not found
collect2: ld returned 1 exit status
make[2]: *** [infinite-loop] Error 1
make[2]: *** Waiting for unfinished jobs....
CCLD     trap-crasher
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of pixman: ########## Error running make -j 2 *** [18/119]

gtk-mac-integration-python fails to build:

libtool: compile:  /usr/bin/gcc-4.2 -DHAVE_CONFIG_H -I. -I.. -I.. -I/Users/osx/gtk/inst/include -arch i386 -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -I/Users/osx/gtk/inst/include -arch i386 -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -isysroot /Developer/SDKs/MacOSX10.5.sdk -isysroot /Developer/SDKs/MacOSX10.5.sdk -D_REENTRANT -I/Users/osx/gtk/inst/include/gtk-3.0 -I/Users/osx/gtk/inst/include/gio-unix-2.0/ -I/Users/osx/gtk/inst/include/cairo -I/Users/osx/gtk/inst/include/pango-1.0 -I/Users/osx/gtk/inst/include/atk-1.0 -I/Users/osx/gtk/inst/include/cairo -I/Users/osx/gtk/inst/include/pixman-1 -I/Users/osx/gtk/inst/include/freetype2 -I/Users/osx/gtk/inst/include/libpng16 -I/Users/osx/gtk/inst/include/freetype2 -I/Users/osx/gtk/inst/include/libpng16 -I/Users/osx/gtk/inst/include/gdk-pixbuf-2.0 -I/Users/osx/gtk/inst/include/libpng16 -I/Users/osx/gtk/inst/include/glib-2.0 -I/Users/osx/gtk/inst/lib/glib-2.0/include -xobjective-c -arch i386 -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -arch i386 -I/Developer/SDKs/MacOSX10.5.sdk/usr/include -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -isysroot /Developer/SDKs/MacOSX10.5.sdk -mmacosx-version-min=10.5 -MT libgtkmacintegration_gtk3_la-GtkApplicationNotify.lo -MD -MP -MF .deps/libgtkmacintegration_gtk3_la-GtkApplicationNotify.Tpo -c GtkApplicationNotify.c  -fno-common -DPIC -o .libs/libgtkmacintegration_gtk3_la-GtkApplicationNotify.o
In file included from GtkApplicationDelegate.c:23:
GtkApplicationDelegate.h:25: error: cannot find protocol declaration for 'NSApplicationDelegate'
make[2]: *** [libgtkmacintegration_gtk3_la-GtkApplicationDelegate.lo] Error 1
make[2]: *** Waiting for unfinished jobs....
mv -f .deps/libgtkmacintegration_gtk3_la-GtkApplicationNotify.Tpo .deps/libgtkmacintegration_gtk3_la-GtkApplicationNotify.Plo
make[1]: *** [all-recursive] Error 1
make: *** [all] Error 2
*** Error during phase build of gtk-mac-integration-python: ########## Error running make -j 2 *** [41/119]

many more packages were not built because of earlier failures or package downgrades I decided to skip

Until we move to a clean 64-bit build env, we need this one to remain stable. (and we'll keep updating this one with minor stuff like ffmpeg and friends after then 1.0 release, and use trunk for the more bleeding edge stuff)

totaam commented 7 years ago

2016-09-14 20:59:53: smo commented

There are many changes and I didn't have a chance to test them all. Some will have to be reverted i'm sure and some patches tweaked.

Right now I'm trying to solve the issue why it is ignoring my overriding of moduleset with
moduleset = "http://xpra.org/svn/Xpra/trunk/osx/jhbuild/modulesets-stable/gtk-osx.modules"
No matter what I do it seems to be ignoring it. I can't seem to figure this one out I will continue to try.

in .jhbuildrc-custom

totaam commented 7 years ago

2016-09-14 22:10:50: smo commented

libffi should be reverted to old version for 10.5 even left myself warning in there and missed it

r13725 reverts this to older version once again

totaam commented 7 years ago

2016-09-14 22:29:04: smo commented

pixman had same build error reverting it for now

done in r13728

totaam commented 7 years ago

2016-09-14 23:28:56: smo commented

I believe I've discovered another issue. Openssl 1.1 doesn't seem to work properly with Python 2.7.12. Sometimes we can see errors like this
AttributeError: 'module' object has no attribute 'sha1'
because the hashlib module is broken

I am going to try and compile python before I install openssl and use the system openssl but this is probably not a good fix.

totaam commented 7 years ago

2016-09-14 23:34:47: smo commented

r13729 removes the patch that is not needed for gtk+

totaam commented 7 years ago

2016-09-15 00:10:36: smo commented

I've built python before doing openssl which seems to fix the issue for now. Not sure if python is built with openssl support or not at this point.

Maybe it never did build with openssl support in the first place?

totaam commented 7 years ago

2016-09-15 00:49:19: smo commented

new glib doesn't build on old system with this error
checking OSX version >= 10.9.0... no
configure: error: OSX version is too old!
Will have to revert this change as well

totaam commented 7 years ago

2016-09-15 00:51:58: smo commented

glib reverted to 2.44.1 in r13730

totaam commented 7 years ago

2016-09-15 01:57:41: smo commented

problem with gobject-introspection

configure: error: Package requirements (glib-2.0 >= 2.48.0) were not met:

Requested 'glib-2.0 >= 2.48.0' but version of GLib is 2.44.1

totaam commented 7 years ago

2016-09-15 02:52:00: smo commented

r13732 reverts gobject-introspection and all associated packages

totaam commented 7 years ago

The python 2.7.x vs openssl 1.1 issue is tracked here: https://bugs.python.org/issue26470 This includes a fairly large patch which we may be able to use until python 2.7.13 comes out.

totaam commented 7 years ago

2016-09-15 07:12:47: smo commented

Everything builds once again for me.
*** success *** [119/119]
Still might be some issues with patches but these are probably from sources that redirect to https.

The process was something like this
jhbuild bootstrap
jhbuild build python 

jhbuild build perl
jhbuild -m ~/Source/jhbuild/modulesets/bootstrap.modules build perl-xml-simple --force

jhbuild build curl

export SSL_CERT_FILE="$HOME/gtk/inst/etc/ssl/cacert.pem" 
curl -o ${SSL_CERT_FILE} -k https://curl.haxx.se/ca/cacert.pem

add to .jhbuildrc-custom so curl works for downloads
os.environ["SSL_CERT_FILE"] = "/Users/osx/gtk/inst/etc/ssl/cacert.pem"

jhbuild build
On older systems here where we can't update jhbuild anymore we probably need to overwrite ~/Sources/jhbuild/modulesets/bootstrap.modules with the one we have.

An update would do this normally its the only file you can't override in .jhbuildrc-custom

Maybe it would be simpler to hold on off new openssl for a bit is there some reason we need the latest?

totaam commented 7 years ago

Links to changesets: r13725 (libffi downgrade), r13726 (syntax typo), r13727 (removed python patch), r13728 (pixman revert), r13729 (gtk patch removed), r13730 (glib revert), r13731 (patch no longer applies?), r13732 (revert gobject-introspection)

Looks like comment:12 needs editing, shouldn't it be "jhbuild build python"?

We can go with openssl 1.0.2 until python 2.7.13 comes out.

totaam commented 7 years ago

2016-09-20 20:38:10: smo commented

r13786 changes openssl back to 1.0.2h until new python comes out.

Probably change this in the next release.

Going to do a full rebuild to make sure all is well then close this ticket.

totaam commented 7 years ago

2016-09-21 19:09:26: smo commented

Full rebuild worked no problem now I need to get pkg and dmg out to testing to see how it works.

totaam commented 7 years ago

2016-09-23 00:11:12: smo commented

r13828 adds patch for bomutils to fix the prefix it installs to

totaam commented 7 years ago

2016-09-23 00:44:26: smo commented

Here are some changes beteween the stable-moduleset we use and the one from gtk-osx project

Some of these I don't remember so I will have to look into them. Others are because our build host is osx 10.5.x ||# xpra version||# gtk-osx version||# reason|| ||libjpeg-turbo-1.4.2.tar.gz||jpegsrc.v9a.tar.gz||better|| ||libxml2-2.9.2.tar.gz||libxml2-2.9.4.tar.gz||??|| ||orc-0.4.25.tar.xz||orc-0.4.22.tar.xz||newer|| ||gstreamer-1.8.3.tar.xz||gstreamer-1.4.3.tar.xz||newer|| ||gst-plugins-base-1.8.3.tar.xz||gst-plugins-base-1.4.3.tar.xz||newer|| ||gst-plugins-good-1.8.3.tar.xz||gst-plugins-good-1.4.3.tar.xz||newer|| ||gst-plugins-ugly-1.8.3.tar.xz||gst-plugins-ugly-1.4.3.tar.xz||newer|| ||gst-plugins-bad-1.8.3.tar.xz||gst-plugins-bad-1.4.3.tar.xz||newer|| ||glib-2.44.1.tar.xz||glib-2.48.tar.xz||compatibility|| ||libffi-3.0.13.tar.gz||libffi-3.2.1.tar.gz||compatibility|| ||gobject-introspection-1.44.0.tar.xz||gobject-introspection-1.48.0.tar.xz||compatibility|| ||pango-1.36.8.tar.xz||pango-1.38.1.tar.xz||??|| ||gdk-pixbuf-2.30.8.tar.xz||gdk-pixbuf-2.34.0.tar.xz||??|| ||gtk+-3.16.7.tar.xz||gtk+-3.20.3.tar.xz||newer|| ||pixman-0.32.6.tar.gz||pixman-0.34.0.tar.gz||??|| ||gtk-mac-integration-2.0.5.tar.xz||gtk-mac-integration-2.0.8.tar.xz||compatibility|| ||openssl-1.0.2h.tar.gz||openssl-1.0.2f.tar.gz||newer|| ||pygobject-3.16.2.tar.xz||pygobject-3.16.1.tar.xz||newer|| ||gmp-6.0.0.tar.bz2||gmp-5.0.5.tar.bz2||newer||

totaam commented 7 years ago

Please bump openssl: OpenSSL Fixes Over a Dozen Exploits, Including a High-Severity Vulnerability

totaam commented 7 years ago

Saw the openssl update: r13851 + r13852, added cryptography 1.5.1 in r13853.

For reference, I wanted up update and found some failures:

*** the following modules were not built *** [119/119]
pycairo-python2.6 pygtk gtk-mac-integration-python meta-gtk-osx-python

Why does it even try to build pycairo-python2.6?? It also rebuilt a lot of things I didn't think needed doing. Could this have been caused by a transient network failure to download our modules overrides? I've switched to https for the base moduleset URL since, and the problem seems to have gone away.

totaam commented 7 years ago

BTW, why are we still on 2.4.30? I see this bug has been fixed and looks relevant: improved fullscreen behaviouur: The attached patch improves the behaviour of fullscreen windows on El Capitan and perhaps other somewhat recent versions of OS X.

totaam commented 7 years ago

And again: openssl critical bug.

totaam commented 7 years ago

2016-10-12 01:07:52: smo commented

openssl updated in r14112 disabled libyuv building for libvpx in r14113 build libvpx 1.6.0 again for ffmpeg in r14114

totaam commented 7 years ago

r13900: x264 20160906
r13898: enable libx264 encoder for shadow servers support
r13853 + r13888: cryptography 1.5.1 then 1.5.2
r13887: enable more encoders so we can use enc_ffmpeg on osx (#1107)
r14087: pillow 3.4.1, ffmpeg 3.1.4

totaam commented 6 years ago

For 2.4 see Xpra-org/gtk-osx-build#2

Xpra-org / gtk-osx-build