Steps towards an official windows build for gstreamer

[GoogleCodeExporter]

As we have been already talking, moving to JHBuild would be the best direction 
towards the integration with the GStreamer community. 
This weekend I had the change to discuss with Julien Moutte and Michael Smith 
and we kind of agree on the followings step to have a windows build officially 
support by gstreamer:
  1) Use JHBuild to build the external dependencies
  2) Have a repository from which to pull down all the dependencies (binaries, headers, libs, etc...) in a blob, like we currently provide, so that's easy to compile gstreamer with mingw for example.
  3) Have a schedule to update the dependencies each 6 months, for example, to reduce the risk of introducing new bugs and only upgrade in the middle if changes in the API requires it.
  4) Create a gst-common-win32 package for all the configuration files, crt's, etc... like we have in shared/SDKS and shared/MSVC
  5) Move the Visual Studio projects to the sources under the win32/vs folder.
  6) Automatize the build with a buildbot for MSVC and GCC.
  7) Get into the release schedule of gstreamer.

The main idea is the replicate as much as possible the Unix way of doing things 
using the same tree (/bin, /lib, /include etc), which makes things easier for 
1), 2) 5) and 6). So using a tree in which we only have to specify the prefix 
to deduce from it the location of the headers, dependencies and binaries as 
it's done in unix will make everything easier.

Regarding JHBuild, the first thing we should work on is the bootsrap module, 
which downloads and install all the utilities needed for the build system like 
mingw stuff, autotools, pkg-config, etc.... This is pretty much like david's 
script, but can be integrated with jhbuild too. Here is an example of bootsrap 
module(http://www.gitorious.org/jhbuild/jhbuild/blobs/windows/modulesets/bootstr
ap-msys.modules).
After that we should be including all the dependencies one by one.
I don't know if it's better for the moment to use a separate git repo for 
that...

Original issue reported on code.google.com by ylatuya on 28 Oct 2010 at 12:48

[GoogleCodeExporter]

JHBuild has module set for many of the dependencies we need:
http://git.gnome.org/browse/jhbuild/tree/modulesets
We just need to modify them to apply the patches and config/make args needed to 
make them compile with in mingw.
JHBuild offers a big flexibility and you can either use the default autotools 
steps or customized command line steps.

Original comment by ylatuya on 28 Oct 2010 at 12:59

[GoogleCodeExporter]

Original comment by david.g.hoyt on 28 Oct 2010 at 8:33

• Changed title: _Steps towards an official windows build for gstreamer _

[GoogleCodeExporter]

Can I take it from 5) that MSVC projects will continue to be maintained under 
this plan? I think it's important to do so for mainstream Windows developers to 
really take to GStreamer. For a time I was working on CMake build files so that 
I could generate MSVC files, though others did more work than I 
(http://code.google.com/p/gstreamer-cmake/).

Original comment by joshthep...@gmail.com on 16 Nov 2010 at 4:34

[GoogleCodeExporter]

The goal is to keep maintaining the visual studio project, but as part of the 
gstreamer project, as it was done some years ago. I think too that it's 
important to have support for visual studio as many developers doesn't come 
from the UNIX world and don't care/ don't have time / don't really need to 
learn about mingw/msys and autotools.

Original comment by ylatuya on 17 Nov 2010 at 12:46

[GoogleCodeExporter]

@David: You will probably like that... I can create the whole msys environment 
with jhbuild :P

$jhbuild bootstrap msys-bootstrap

I have created a branch in gitorius but I haven't committed the bootstrap 
moduleset. I'll also create a new wiki page with the build instruction... no 
more than 4 lines ;)
Next step is starting to build the dependencies with jhbuild :D

Original comment by ylatuya on 11 Dec 2010 at 2:43

[GoogleCodeExporter]

I am using the following repository:
http://gitorious.org/~ylatuya/jhbuild/ylatuyas-jhbuild/commits/windows

Original comment by ylatuya on 11 Dec 2010 at 10:15

[GoogleCodeExporter]

I have started know with the dependencies modules and right now the following 
deps  are built with jhbuild:
libtool
libxml2
intltool
pixman
expat
freetype
fontconfig
libiconv
gettext-runtime
libtasn1
libgpg-error
libgcrypt
gnutls
glib

Original comment by ylatuya on 14 Dec 2010 at 8:12

[GoogleCodeExporter]

Are you doing this all from Windows?

What are you using as your compiler? Please be sure and use mingw-w64 (NOT 
mingw.org). I have my own build of GCC I've been using that incorporates all 
the directx headers. You can download it from the downloads page.

What's jhbuild's 64-bit support on Windows looking like?

I'd like to have several subprojects like how gstreamer has gstreamer, 
gst-plugins-base, gst-plugins-good, etc. Ones like: 

  ossbuild-env:          creates installer w/ all the development tools for windows: gcc, bash, libtool, perl, autoconf, jhbuild, etc. and scripts for getting all the build dependencies for other OSes
  ossbuild-samples:      contains sample media used for testing (similar to samples.mplayerhq.hu)
  ossbuild-msvc:         plugins/templates/samples for visual studio integration
  ossbuild-source:       contains all the source code for the packages we support
  ossbuild-main:         where all the real work goes on (ossbuild-get, similar to apt-get but for windows packages - works on linux/windows/osx; ossbuild-dpkg for creating ossbuild packages; jhbuild scripts for doing builds; etc.)

What do you think? It's not 100% thought out yet, but that's the direction I'd 
like to be taking things.

Original comment by david.g.hoyt on 14 Dec 2010 at 10:02

[GoogleCodeExporter]

I'm doing everything from windows.
For the moment I'm only looking to replicate our build system from scratch with 
jhbuild and defining the moduleset for the external dependencies. Organizing it 
with a different layout is very easy. You define the main moduleset 'ossbuild', 
and include the different submodules "ossbuild-external-dependencies", 
"gstreamer-windows", "vlc-windows" and "ffmpeg-windows" for example.
jhbuild has a bootstrap module, which is what you call "ossbuild-env"[1]. This 
module downloads and install all the dev tools in the msys/mingw root. You 
would usually do the following steps to setup the build environent:

* Install Python27
* Install  MSYS-1.0.11
* Clone jhbuild repo
* Install jhbuild (make -f Makefile.windows install)
* Run "jhbuild bootstrap msys-bootsrap" 

At this point you have an msys/mingw environment ready to build everything. For 
the moment I'm using tdm's gcc compiler, the 32bits version, but using 
mingw-w64 is just a matter of changing the download url's :)

Then you define in the deps submodule all the external libraries and their 
dependencies. For instance, for glib we have:

<autotools id="glib" autogen-sh="configure">
    <branch module="glib/2.26/glib-2.26.1.tar.bz2" version="2.26.1"
            repo="gnome.org">
    <patch file="glib-autogen-libtool-check.patch" strip="1"/>
    </branch>
    <dependencies>
      <dep package="libiconv"/>
      <dep package="gettext-runtime"/> 
    </dependencies>
  </autotools>

'jhbuild build glib' will download glib's tarball and extract it in 
$PREFIX/src/. It will run configure, make and make install (the parameters can 
be configured in the module description too). If libiconv and gettext-runtime 
haven't been built before it will built them too, otherwise they are skipped.

We can either use jhbuild directly, or write some python scripts to wrap 
jhbuild. For example ossbuild-env will download and install Python27, Msys and 
jhbuild and run 'jhbuild bootstrap install module'

As you saw in the glib's examples, jhbuild already handles everything around 
building stuff (fetching the source code, compiling it and installing, 
dependencies and reverse dependencies).

$ jhbuild info glib
Name: glib
Module Set: gstreamer-external-deps
Type: autogen
Install date: 2010-12-14 21:50:47
URL: http://download.gnome.org/sources/glib/2.26/glib-2.26.1.tar.bz2
Version: 2.26.1
Tree-ID: 2.26.1-fa649bd7bb5f01623eb631b42fd3360f
Sourcedir: c:/build/src\glib-2.26.1
Requires: libiconv, gettext-runtime
Required-by: shared-mime-info, clutter, desktop-file-utils, dbus-glib, 
gobject-introspection, libsoup, libgdata, libgda, dbus

[1]http://gitorious.org/~ylatuya/jhbuild/ylatuyas-jhbuild/blobs/windows/modulese
ts/bootstrap-msys.modules

Original comment by ylatuya on 14 Dec 2010 at 11:01

[GoogleCodeExporter]

ossbuild-env is actually all about creating a single windows installer for 
*other* devs to use. Think of it eventually being like the visual studio 
command prompt -- all the tools you would want are available if you need them 
-- no downloading necessary (I have to use this on systems that don't have an 
internet connection). We can use jhbuild to get the initial environment going 
and for doing updates, but after the first release you could just use our own 
installer to set things up. It's a lot less steps than install python, install 
msys, clone the repo, etc. Remember that the idea is to ease the barrier of 
entry for non-open source, non-linux developers. You tell them to do all of 
that and they'll stare at you funny. I want this to be as easy and painless as 
possible for Windows devs. And from following the mingw mailing list, I know 
that a proper msys env has not been easy to setup for a lot of people (think 
back on the first time you attempted to set one up by yourself). Most Windows 
devs aren't going to care about what each dependency is and why they need it -- 
they just want to get started coding. We can certainly use a python script to 
ease the burden of downloading and setting up the environment from which we'll 
generate the installer. In fact, I'd highly encourage that. (c:

Getting the compiler/environment right is non-trivial, especially in the case 
of having multiple target platforms. TDM gcc by default will compile to 64-bit. 
You have to use -m32 to get it to compile for x86. Compilation seems to be much 
smoother when using configure with a single target gcc. It's a pain getting 
winres and related tools to compile to 32-bit. This is partially why I chose to 
compile my own gcc. Switching out gcc compilations nontrivially affects your 
builds. I've been going through that pain now for the past couple of years (one 
version of gcc compiles a lib cleanly, another introduces a bug -- e.g. the 
liba52 issue from this release). Take a look at all the changes to 
Build-All-Windows-x86.sh every time I changed compilers and you'll see how much 
was required to get it to work right. So I'd like us to standardize around one 
compiler version and stick with it (at least per release). The easiest way for 
us to do that and ensure its ubiquity and that it satisfies all our 
requirements is to compile it ourselves. That also provides an exact audit 
trail for any bugs and issues that arise (e.g., library xyz was miscompiled due 
to a bug in gcc revision xyz, here are workarounds or fixes...). I compile in 
the exact gcc revision number that my builds use. It's important for me to have 
that trail in there and it might be for others as well. But yeah, it's a lot of 
extra work. But I think its worth it in order to maintain a stable, sane 
compilation environment. I don't imagine we'd be switching around compilers 
very often. A lot of the work these past couple of years and the reason for all 
the different lib builds has been to try and identify a good, known, stable gcc 
version that will work across all our supported platforms. And it hasn't been 
easy.

At any rate, I'd propose an installer that creates a typical msys setup and 
installs gcc in 2 different paths and then we have a batch file to launch an 
msys prompt which sets up the paths correctly to target the correct platform. 
e.g. an msys command prompt and an msys x64 cross tools command prompt. In the 
former, gcc always builds 32-bit, x86-compatible binaries and in the latter, 
gcc by default always builds 64-bit, x64-compatible binaries. How this 
environment is setup on Linux and OSX is TBD. But again, I'd prefer to retain 
the ubiquity so that devs don't have to worry about wildly different build 
environments from one platform to the next. IOW, they can concentrate on the 
things that are truly unique about OSX vs Linux vs Windows and not have to 
worry about the differences in their compiler and available tools.

ossbuild-source is important because I can't and don't want to rely on some of 
these packages being available for download from the Internet. Libraries like 
liba52 haven't been updated in years and who knows when the website hosting the 
files may suddenly disappear. Also, keeping the packages in-tree provides a 
further audit trail. We'll always know exactly which version we're using for 
which library because we have the full source for it plus our own 
documentation. For any release we make, we can point exactly to the source we 
used to make it. It seems that jhbuild can be readily modified to use this 
(which is great!). There's nothing stopping us from using jhbuild to pull down 
updated libraries if we want to maintain something like that. But the actual 
build (e.g. in the main ossbuild module) should be using ossbuild-source 
instead of going out to the websites/ftp to pull down the source.

Part of my ultimate goal here is that I could take release 1.0.0 (let's say) 
from all the modules, put them on a usb stick, copy it to an offline machine, 
and compile everything cleanly in the first pass with no problems. I think it's 
high time that these platforms had a ubiquitous build environment and open 
source software is in a keen position to provide that. Then you'll start to see 
wider adoption and usage. At least that's the idea.

Original comment by david.g.hoyt on 15 Dec 2010 at 4:47

[GoogleCodeExporter]

Also forgot to mention that if you are going to use tdm gcc, please, please use 
the mingw-w64-based version. The mingw.org headers/libs are vastly different 
and pretty much outdated from the mingw-w64 ones. Things that compile with the 
mingw.org-based one might not compile at all with mingw-w64. And that will mean 
problems when we get to a 64-bit windows build. Now's the best time to 
reconcile the problems and push fixes upstream to the respective projects using 
the compiler and headers/libs that'll be needed for all the platforms we 
support. I've had to do that on both the mingw-w64 side as well as on the lib 
side. Most of the time the bugs/problems are trivial.

Original comment by david.g.hoyt on 15 Dec 2010 at 5:03

[GoogleCodeExporter]

I think we should crearly separate the two targets that concern us, the first 
one consisting on the tools that will help us maintaining the build system and 
the second one providing the easiest solution possible for developers so that 
they can focus in the project they are hacking on.

jhbuild is defenitively the tool we are looking for to maintaining our build 
system. It can easily re-create from scratch the mysys/mingw environment and 
help us defining the set of external libraries and their dependencies as well 
as having track of versions beeing compiled and the location of the original 
sources.

I agree that setting-up a good msys/mingw environment is very hard, but once 
you have it (as we currently do), replicating it is as easy as downloading the 
correct files and jhbuild will help us automating this task.

In jhbuild you can configure the location where the sources will be downloaded, 
so just by modifing the module version number and the download url, you will be 
pulling the new tarball to the download folder and then you could update your 
repository if you need or want to have the tarballs in your local repo, which 
by far easier to maintain than with the current workflow.

Regarding the support of 32-bit and 64-bit and setting up the right variables 
for each environemt, it can also be handled with jhbuild. jhbuild runs 
everything in a shell, where the environemt is configured using ~/.jhbuildrc 
[1]. By default, jhbuild will use .jhbuilrc, but you can force a different 
configuration file, so we will end up having the default one for 32-bit and a 
second one for 64-bit.

The msys bootstrap module replicates currently all the downloads done by the 
script Tools/msys-env.bat. I just wanted to have something ready as fast as 
possible to start defining the different modules sets to familiarize my self 
with jhbuild so I didn't pay too much attentio wheter I was using the right 
compiler. To be honest I have lost track of the changes in the compiler we are 
using, so I would really appreciate if you could explain how to set up this 
properly. As far I understand is just a matter of downloading 
mingw-w32-1.0-bin_i686-mingw_20101011.zip instead of all the mingw packages, 
right?

Keeping in our repository all the tarballs is not that important to me. For me 
gnu.org, gnome.org or freedektop.org are reliable enough and you can will 
always now which version your compiling and you even check the checksum and the 
size of the tarball because it's defined in the the jhbuild's module. I don't 
see the difference between pulling down the tarball from the upstream server or 
from our own server. jhbuild keeps all the tarball in the folder specified in 
the config and they are downloaded only when you don't have them localy. 

[1]http://gitorious.org/~ylatuya/jhbuild/ylatuyas-jhbuild/blobs/windows/win32.jh
buildrc

Original comment by ylatuya on 16 Dec 2010 at 1:05

[GoogleCodeExporter]

Having the packages ourselves provides a number of benefits -- e.g. the one I 
described earlier where there are plenty of packages whose servers I don't 
trust (some yes, but a lot of them, no) -- but what it really comes down to is 
control and traceability. I need to have reasonable confidence that for a 
particular release, I can trace and know exactly what the source code was that 
went into compiling the binaries. Also consider a tool like "ossbuild-get" (or 
whatever name we come up with in the future) -- you issue a command 
"ossbuild-get a52-source", you'd want it to be able to download and get exactly 
the version we used to build the binaries so you can hack on it and make 
changes. And in future releases the same command would then download the 
appropriate version for that release. An easy way to track it all is to have 
ossbuild-get download it from a git release tag. So release 1.0.0 could get its 
source from 
http://cgit.freedesktop.org/ossbuild/ossbuild-source/tree/packages/manual/liba52
.tar.gz?id=RELEASE-1.0.0 (or some other URL like that). It also provides us an 
easy way to update packages (just submit via git). The tool would obviously be 
configurable if you didn't like or want to use git for that. Hope all that 
makes sense.

Overall, I think it's a fine idea to use jhbuild to build and maintain those 
packages. But I do want them collected in one place. It would still mean 
modifying a jhbuild script in ossbuild-source to change the URL. Then you would 
commit the new package (after testing that it doesn't affect the build). The 
jhbuild scripts in ossbuild-main would never have to change b/c they'd always 
be pulling from our own repo. This separates concerns and isolates potential 
errors to their respective modules.

It might do some good to visit every site where the packages were downloaded 
from to give you a sense that not all the websites are as reliable as gnu.org, 
gnome.org, and company. It might scare you a bit in fact. (c:

~ My point in ossbuild-env is to make it one step easier for devs. You have to 
remember that a lot of them have never heard of jhbuild or ever downloaded and 
setup python. I'm talking about trying to make it as idiot-proof (no offense 
intended to anyone reading this) as possible. In one fell swoop, our installer 
can install python, jhbuild, msys, git tools, mingw tools, and gcc (surprise 
surprise, it just *happens* to contain all the tools we need to build every 
libary/tool in the project). We should use jhbuild to download and organize the 
files that would be needed for subsequently creating an installer containing 
all of it. WiX's bootstrapper (heat.exe) makes it relatively easy to chain 
installers and I've already written WiX extensions that examines the file 
system and generates WiX files for creating installers. At this point it's 
straight forward enough. I think I could manage doing this portion pretty 
easily.

~ Regarding ~/.jhbuildrc for 32-bit vs. 64-bit builds -- that's fine for our 
use, but it's not going to fly with the average Windows developer. I'm thinking 
of the end product here, not necessarily what it is we'll be doing as project 
maintainers/developers (which we're obviously free to do however we like). From 
experience (I include myself here), the typical Windows dev will not want to 
edit that config. file and have to learn yet another build package. They won't 
care about which build system is intellectually/actually superior or what. 
msbuild does what they need to do 99% of the time, so why learn some 
python-based system just to get a 64-bit build? They're more likely to give up 
then try and figure it out. There's not time to twiddle around with 
configurations when you have a product to ship. So that means we need to 
organize the tools/install so that it'll just work out-of-the-box. But for own 
use, whatever we want goes. I think we definitely should define, override, and 
use our own jhbuild configuration to ensure that the build is the same across 
developers' machines.

~ Right now, the ReadMe.txt instructs you to run Tools/msys-env-x86.bat which 
downloads 
http://ossbuild.googlecode.com/files/mingw-w64-x86-ossbuild-bin-r164692.tar.lzma
 and 
http://ossbuild.googlecode.com/files/mingw-w64-x86_64-ossbuild-bin-r164692.tar.l
zma which is my own gcc build that I worked with the mingw-w64 folks to produce 
and it's configured in such a way as to provide (what I think is) maximum 
compatibility between msvc and gcc and to avoid buggy portions of gcc that 
result in bad binaries (it's not 100% perfect as you can tell -- e.g. the 
liba52 build...). I use a lot of the same options that TDM gcc uses, but it 
differs in others. My gcc build scripts are modified mingw-w64 ones forked 
directly from their repo. Check out Tools/gccbuild-w64-x86.mk for details. I 
helped them resolve several issues with a native Windows gcc build while 
getting it prepared and working.

What do you think?

Original comment by david.g.hoyt on 16 Dec 2010 at 11:09

[GoogleCodeExporter]

I agree a 100% with you, and that's why I want to make clear the separation 
between what we are going to use internally and what we want to provide to the 
developers to make their live easier :)
As you say we must have a wrapper like ossbuild-env to provide a build system 
ready to usedwith a single donwload, but at the same time I'd like to be able 
to re-create the whole build system from scratch, like jhbuild build does. Many 
devs won't even care about jhbuild, but it's going to be very useful for the 
gstreamer comunity for example, as it's the daily tool being used in many big 
projects, such as gtk+ or gstreamer.

And the same for the 2 jhbuild configs (32 and 64 bits)... it's a clean way to 
handle it although we will need a wrapper around to hide it to the devs.
'ossbuild-pkg build glib' will translate to 'jhbuild --config jhbuildrc.win64 
build glib' 

What I don't like about JHbuild modules set is that you usally end up with a 
big file describing all the modules (eg: 
ossbuild-external-dependencies.moduleset). So what I proposed is to define one 
module set for each project. So for glib we will have:
  * glib-release (uses ossbuild's repository to fetch the release package)
  * glib-upstream (the same but downloading it from the upstream repo)
  * glib-git (Uses the git repository to donwload the atest code)

repository type="tarball" name="ossbuild" default="yes" 
href="http://ossbuild.freedesktop.org/releases/"/>  
<repository type="tarball" name="gnome.org" 
ref="http://download.gnome.org/sources/"/>
<repository type="git" name="git.gnome.org" href="git://git.gnome.org/"/>

 <autotools id="glib-release" autogen-sh="configure"/>
    <branch module="glib.tar.bz2" version="1.0.0" repo=""ossbuild"/>
 <autotools id="glib" autogen-sh="configure">
    <branch module="glib/2.26/glib-2.26.1.tar.bz2" version="2.26.1" repo="gnome.org">
 <autotools id="glib-git" autogen-sh="configure">
    <branch/>

Original comment by ylatuya on 20 Dec 2010 at 2:04

[GoogleCodeExporter]

Looks like we're on the same page. (c:

The only thing w/ 2 configs -- I just want to have an environment as well that 
will work for 32 and 64-bit builds even if you're not using jhbuild at all (and 
I think you already understand that, I'm just being overly verbose for the sake 
of clarity). Beyond that, I'm silent because it wouldn't affect me. (c: I think 
what you're proposing wouldn't cause problems at all since those configuration 
files would only kick in when building with jhbuild anyway. Using 
configure/make from the command line would never make use of the jhbuild 
configurations anyway (right?).

I've still not dived deep into jhbuild, so you're the lead there. If what you 
say is the best way to proceed, then let's do it that way. (c: Sounds like a 
good idea to me to maintain the 3 separate options since it doesn't hurt that 
much to track them.

Original comment by david.g.hoyt on 20 Dec 2010 at 7:15

[GoogleCodeExporter]

I believe it's a good way to proceed, but not 100% sure if it's the best :P
In the end jhbuild build is just a helper around wget, tar, configure and make 
:) You can still use everything in the same way, the only difference is that it 
helps you setting the environment and managing the dependencies in a very clean 
way. 

Original comment by ylatuya on 20 Dec 2010 at 10:10

[GoogleCodeExporter]

ylatuya: I have the initial steps towards a build environment done and I've 
gotten to the jhbuild part. When I use your repo. and do the install, it seems 
like it leaves everything in my home directory (where I did the git clone) and 
adds the jhbuild script, but the script itself points to python code in my home 
directory. I'd like to do a system-wide (i.e. mingw-installation-wide) jhbuild 
install. Is that possible?

Original comment by david.g.hoyt on 11 Feb 2011 at 9:22

[GoogleCodeExporter]

ylatuya: figured it out. I'm using your jhbuild, btw. But I filed this patch 
for the jhbuild folks: https://bugzilla.gnome.org/show_bug.cgi?id=643922

my stuff's available on github, btw: https://github.com/ossbuild

The env is nearly complete. Perhaps sometime soon I can post the installer for 
download/evaluation. It's rather large, though...And I don't want to make it 
official until it's been evaluated/used/proven. At this point, it's all about 
getting the module sets defined, added, and tested.

The build env installer should have everything you need to get going and 
building on Windows. It creates shortcuts in your start menu to get you going 
(similar to Visual Studio command prompts). You start up the prompt and 
jhbuild's already there. It even passes jhbuild sanitycheck. (c: I haven't 
actually *built* anything with it yet, though. :/

Anyway, it's there if you're interested.

Original comment by david.g.hoyt on 5 Mar 2011 at 1:53

[GoogleCodeExporter]

Cool! I am on vacations but I'll be back tomorrow. I'll start adding the
modules I already have in separate modules sets like we agreed, 1 per
library to make easier contributions.

El 05/03/2011 03:54, <ossbuild@googlecode.com> escribi�:

it's

Original comment by ylatuya on 5 Mar 2011 at 8:38

[GoogleCodeExporter]

ylatuya: attached is a patch against your jhbuild env for better msys/windows 
compatibility. it's been submitted upstream as well: 
https://bugzilla.gnome.org/show_bug.cgi?id=643922

Original comment by david.g.hoyt on 8 Mar 2011 at 9:04

Attachments:

• 0001-Fixes-for-compatibility-with-msys-environments.patch

[GoogleCodeExporter]

I'm back from holidays and I'll start adding all the dependencies this week. I 
really want to have this ready ASAP because I don't have as much time as I had 
before to work in the VS stuff :(

Original comment by ylatuya on 8 Mar 2011 at 9:07

[GoogleCodeExporter]

Do you need write access to the github repo? If so, send me your public ssh key 
so I can add you. And I'll need to upload a copy of the env so you can see 
where it's at. If you use it, a lot of the jhbuild defaults are already set.

You'll want to git clone ossbuild-packages and then edit your .jhbuildrc file 
and set the following values:

modulesets_dir='C:/<path-to-ossbuild-packages>/packages/'
moduleset = modulesets_dir + 'ossbuild.modules'

jhbuild sanitycheck should pass and jhbuild list will print out:

gnome-common
ossbuild-core
ossbuild-media
ossbuild-lang
ossbuild-testing
ossbuild

As you can see, I'm only to gnome-common and nowhere past it. It's been a lot 
of work to get that far, though.

Original comment by david.g.hoyt on 8 Mar 2011 at 10:15

[GoogleCodeExporter]

[deleted comment]

[GoogleCodeExporter]

if a patch is needed, where should it be stored in the tree?

Original comment by ylatuya on 8 Mar 2011 at 10:31

[GoogleCodeExporter]

Patches should be stored in the same directory where the source is. The jhbuild 
module should be sure to apply it (as I'm sure you already know).

github said your ssh key was already in use. I looked you up and added you to 
the team. Consequently, I'm deleting your pub key from this google code issue. 
Please let me know if you're able to access and work on things.

Original comment by david.g.hoyt on 8 Mar 2011 at 11:08

[GoogleCodeExporter]

Will you be working on this from the linux side, cross-compiling, or from 
windows using the build env? Having both sides work would be nice -- you're 
probably far better at the linux side than I so it'd make sense for you to head 
that up. However, I'd like the focus (for now) to be a native build. Thoughts?

Original comment by david.g.hoyt on 8 Mar 2011 at 11:24

[GoogleCodeExporter]

Got it thanks :)

I have seen that you store for each module the sources package, I think we will 
never agree on that :P, but specially with git it's not a good idea at all.

Git is based on changesets and not in snapshots like svn[1][2], so when you 
checkout a repository, you are not downloading the snapshot of the repository 
but all the deltas from the the first commit. So if your are handling large 
binary files, cloning a repository of binary files is like downloading all the 
files that have been committed in the history of the repository.

So instead of using local tarballs committed in the repository, I would 
download them (which is also handled by jhbuild) either from upstream or from 
an ftp server of ours with a copy the tarballs, but I would never commit a lot 
of binary files that are going to change very frequently in a a git repository. 
With this solutions you still use a reliable source while keeping the repo free 
of big changesets.

And in case you don't like this idea, we should keep the tarballs in a 
submodule, but as you will read in the comments of this thread in 
stackoverflow[3] it's not a good idea (one of them says: "At the end, I had 
only ~6Gb of mp3's and ~83Gb in the .git dir").

[1]http://stackoverflow.com/questions/2188749/svn-and-git-versioning-models-diff
erence
[2]http://sourcefrog.net/weblog/software/vc/derivatives.html
[3]http://stackoverflow.com/questions/540535/managing-large-binary-files-with-gi
t

Original comment by ylatuya on 8 Mar 2011 at 11:31

[GoogleCodeExporter]

I'll upload the windows build env tonight asap or tomorrow.

Original comment by david.g.hoyt on 8 Mar 2011 at 11:32

[GoogleCodeExporter]

I don't expect people to be git cloning ossbuild-packages. It's in a git repo. 
simply because that makes it easier to manage and push to a server where it can 
be centrally managed. The jhbuild stuff can be pulling the files directly via 
http (e.g. 
https://github.com/ossbuild/ossbuild-packages/raw/master/packages/core/atk/atk.t
ar.bz2)

If you think the jhbuild stuff (.modules) should be elsewhere, let's discuss 
that. I personally like having it inline w/ the source, but I wanna be flexible.

Original comment by david.g.hoyt on 8 Mar 2011 at 11:44

[GoogleCodeExporter]

Forgot to mention that I'm only recommending us git clone ossbuild-packages for 
now  b/c the jhbuild scripts aren't completed/finalized. But once they're 
complete, we'll want to modify them to pull via http. I just haven't personally 
gotten to that stage yet.

I'm well aware of the diff. b/t svn and git, though.

Original comment by david.g.hoyt on 8 Mar 2011 at 11:47

[GoogleCodeExporter]

> So if your are handling large binary files, cloning a repository of binary 
files is like downloading all the files that have been committed in the history 
of the repository.

According to 
http://stackoverflow.com/questions/2188749/svn-and-git-versioning-models-differe
nce, 2nd answer (and according to your comment), git stores binary deltas. So 
it's not like you're downloading every file that number of times. Granted from 
one gzipped tar to the next there's a significant enough difference that 
statistically it might as well be what you're saying, but the comment remains 
factually incomplete/misleading.

Besides you can always do: 
https://git.wiki.kernel.org/index.php/GitFaq#How_do_I_do_a_quick_clone_without_h
istory_revisions.3F

Does jhbuild have a command that will download packages only (and then cache 
them) without doing a build?

Original comment by david.g.hoyt on 8 Mar 2011 at 11:57

[GoogleCodeExporter]

Some of the recommendations given in 
http://stackoverflow.com/questions/540535/managing-large-binary-files-with-git 
are to put it in a separate repo. (in fact, the comment you quoted, "At the 
end, I had only ~6Gb of mp3's and ~83Gb in the .git dir", starts out by saying 
"I would use submodules ... or two distinct repositories."), which is exactly 
what's been done. I'll read up on submodules, but if that's better, then it'd 
make sense to move to that. If the packages make better sense inside an svn 
repo., that's fine. But the snapshotting/tagging is important to me and to 
others (for security reasons).

Sorry for all the comments -- just real time uploading my thoughts as I think 
them. (c:

Original comment by david.g.hoyt on 9 Mar 2011 at 12:13

[GoogleCodeExporter]

In the future, sparse clones can also solve the problem: 
http://article.gmane.org/gmane.comp.version-control.git/155389

Although for now, it seems, depth of history hasn't been implemented.

Original comment by david.g.hoyt on 9 Mar 2011 at 12:21

[GoogleCodeExporter]

Ok, so ossbuild-packages is not what devs will clone, but a different project 
that stores the tarballs to make them accessible in an easy way right?

I think they should be in separate project, but not it in ossbuild-main, 
because it's something that it's going to be in continuous evolution and you 
can't really set a release point when you are managing a big bunch of projects.

In ossbuild-main I would have the packaging utils and a wrappers around jhbuild 
and a submodule with jhbuild, so that we can use it directly, to make the 
distribution package or to create a binary with py2exe for distribution in the 
release.

I would expect people to contribute new packages like it's done in homebrew [1] 
and updating them only by changing the target URL[2].

Regarding the different configurations, how would you like to manage them?
Using subfolders for each project like
 * libvpx/ossbuild/libvpx.modules
 * libvpx/release/libvpx.modules
 * libvpx/git/libvpx.modules
Or defining different modules names in the same file
 libvpx-ossbuild, libvpx-release and libvpx-git?

I think the last option is better because we can easily wrap "ossbuild make 
libvpx" to "jhbuild build libvpx-$env".

[1]https://github.com/mxcl/homebrew/commit/ae01d7743c3c61151ad18501ffb27c7995a4a
05d
[2]https://github.com/mxcl/homebrew/commit/8a08355729db8295db1e53af02904c36e0135
217

Original comment by ylatuya on 9 Mar 2011 at 12:27

[GoogleCodeExporter]

To download the modules and cache them you can use the "update" option:
http://library.gnome.org/devel/jhbuild/2.32/command-reference.html.es#command-re
ference-update

What I fear is that a repository of 500MB (I hope that in the end we will be 
managing a lot project ;)) might end in a clone of 10GB after several 
iterations, because deltas of binaries are not like deltas of changesets (a 
delta of 2 compressed tarballs is almost like the tarball itself and that's why 
I said it's like downloading all the files committed in the history).

Original comment by ylatuya on 9 Mar 2011 at 12:35

[GoogleCodeExporter]

> Ok, so ossbuild-packages is not what devs will clone, but a different project 
that stores the tarballs to make them accessible in an easy way right?

Right.

> I think they should be in separate project, but not it in ossbuild-main, 
because it's something that it's going to be in continuous evolution and you 
can't really set a release point when you are managing a big bunch of projects.

Not sure if I entirely follow. I think I agree. :p

> In ossbuild-main I would have the packaging utils and a wrappers around 
jhbuild and a submodule with jhbuild, so that we can use it directly, to make 
the distribution package or to create a binary with py2exe for distribution in 
the release.

Run with it -- I'll follow your lead here.

> I would expect people to contribute new packages like it's done in homebrew 
[1] and updating them only by changing the target URL[2].

Essentially yes (however I'm not familiar with homebrew at all). In my 
thinking, a release would target a specific tag of the source (packages). So in 
that sense, with each full release we'd be changing the target URL b/c the URL 
for that release's packages tag will have changed. Perhaps this is undesirable 
if you'd rather see ossbuild as more of a rolling release distro. :/ I prefer 
to have definable releases to provide users with a clear, distinct upgrade 
path. I can see package updates as bugs/fixes arise, but we'd want to maintain 
overall project integrity via carefully selecting fixes that won't adversely 
change behavior. In essence, I'd view our project as a distro. for Windows. :D

> I think the last option is better because we can easily wrap "ossbuild make 
libvpx" to "jhbuild build libvpx-$env".

Agreed.

Original comment by david.g.hoyt on 9 Mar 2011 at 12:42

[GoogleCodeExporter]

A submodule in git is like the equivalent to svn externals, but more powerful.
For instance in gstreamer, gstreamer-common is added as a submodule to all the 
repositories.
In gst-plugins-base you only store a commit hash for the 'common' submodule 
defined in '.gitmodules', so when you update the submodule using 'git submodule 
update', you are checking out the commit of gstreamer-common in the folder 
'common'.

Original comment by ylatuya on 9 Mar 2011 at 12:51

[GoogleCodeExporter]

>Essentially yes (however I'm not familiar with homebrew at all).
It's just an example to show how easy is to add a new project (adding a file), 
and upgrading a project (changing the url and the md5).

>I prefer to have definable releases to provide users with a clear, distinct 
upgrade path. I can see package updates as bugs/fixes arise, but we'd want to 
maintain overall project integrity via carefully selecting fixes that won't 
adversely change behavior. In essence, I'd view our project as a distro. for 
Windows. :D

Agreed! For me the stability is very important and over all these years playing 
around with the windows builds it's a lesson we have learned pretty well ;)

Original comment by ylatuya on 9 Mar 2011 at 12:55

[GoogleCodeExporter]

I haven't been able to find any info. about this (probably b/c it doesn't exist 
in the way I'm thinking about it) -- can I define a variable in jhbuild and 
then use it in different module sets?

e.g. I'm thinking of a variable like "OSSBUILD_BASE_URL" (e.g. with a value of 
"https://github.com/ossbuild/ossbuild-packages/raw/master/packages/") and then 
using it for URLs so we would only have one place to update.

I may not be thinking about this right. And you may have a better/alternative 
solution (using the python "repos" dictionary? something else?).

Original comment by david.g.hoyt on 9 Mar 2011 at 12:58

[GoogleCodeExporter]

IOW, a way of centrally defining a value that's accessible from the module set 
XML.

Original comment by david.g.hoyt on 9 Mar 2011 at 12:59

[GoogleCodeExporter]

One thing we should define are the requirements/commands of ossbuild-main. Some 
ones are clear, more or less doing what jhbuild is doing (build, update, list, 
blablabla), but others aren't, like being able to target different environments 
(release, git, etc..), being able to package a project (like running "ossbuild 
package gstreamer"), and getting in distribution tree that only contains the 
built dependencies of gstreamer.

Why don't we open a wiki page for that? 

Original comment by ylatuya on 9 Mar 2011 at 1:02

[GoogleCodeExporter]

A wiki page would be a great idea -- I wish google code didn't add a commit to 
the repo., though, for each edit. It'd be fine if it was a separate repo., 
though.

Original comment by david.g.hoyt on 9 Mar 2011 at 1:06

[GoogleCodeExporter]

How about the forum that nobody uses? :D

Original comment by david.g.hoyt on 9 Mar 2011 at 1:08

[GoogleCodeExporter]

If you want to use it: 
http://ossbuild.hoytsoft.org/forums/yaf_postst27_Next-gen-ossbuild-ossbuildmain.
aspx

Original comment by david.g.hoyt on 9 Mar 2011 at 1:11

[GoogleCodeExporter]

I don't think that's possible and it's very useful, so we could probably patch 
jhbuild for that.
But this use case you define a repository like:

<repository type="tarball" name="ossbuild"
    href="https://github.com/ossbuild/ossbuild-packages/raw/master/packages/"/>

And use it like:

<autotools id="fribidi">
    <branch module="fribidi-0.19.2.tar.gz" version="0.19.2"
            repo="ossbuild">
    </branch>
</autotools>

Original comment by ylatuya on 9 Mar 2011 at 1:15

[GoogleCodeExporter]

Will that work if you have multiple subdirectories? e.g. fribidi would be in 
http://.../master/packages/core/fribidi/fribidi-0.19.2.tar.gz and not in 
http://.../master/packages/fribidi-0.19.2.tar.gz

> I don't think that's possible and it's very useful, so we could probably 
patch jhbuild for that.

I think the variables idea wouldn't be very difficult to add to jhbuild. But if 
it's not needed, great. (c:

Original comment by david.g.hoyt on 9 Mar 2011 at 1:28

[GoogleCodeExporter]

windows build env (ossbuild-build-env-v1.0.0-beta00-preview-00.exe) is 
available at: http://www.yourfilelink.com/get.php?fid=622586

pardon the cheesy file hosting, but it's a large file. and please pardon the 
cheesy installer UI -- WiX' bootstrapper is still in the works. In the future, 
we'll have something that looks better.

Original comment by david.g.hoyt on 9 Mar 2011 at 1:50

[GoogleCodeExporter]

>Will that work if you have multiple subdirectories? e.g. fribidi would be in
you can use: module="fribidi/fribidi-0.19.2.tar.gz"
the 'module' property sets a relative path using as base url the 'href' 
property in the repository.

NP with the installer :)

Original comment by ylatuya on 9 Mar 2011 at 3:25

[GoogleCodeExporter]

I have the wrapper working and the setuptools enabled for the project :)
I have pushed a new branch for the moment.

You just have to clone the repo and run (under linux):
$git submodule init
$python setup.py build
$python setup.py install --prefix /home/user/.local
$ossbuild -h
$ossbuild list
$ossbuild help build

Time to sleep :D

Original comment by ylatuya on 9 Mar 2011 at 4:13