Linux: very slow fetching of mainline kernel - migrate from git to tar.xz?

[KristofRobot]

I am encountering very slow fetch times of the mainline kernel from git://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable.git (see https://github.com/linux-sunxi/meta-sunxi/blob/master/recipes-kernel/linux/linux_4.1.bb)

Is there any reason we don't just fetch the compressed tar.xz package from https://www.kernel.org/pub/linux/kernel/v4.x/linux-4.1.2.tar.xz? That seems to be a lot faster in my case.

[jlucius]

There was a discussion about a similar issue for raspberry pi on the yocto mailinglist (which has not been concluded as far as I know).

Fetching the kernel via git is slow and used a lot of space but makes it possible to target specific commits and makes it possible to easy change the git url to a forked copy or other repository.

Fetching it via package is faster, uses less space in the build system but makes it only possible to use released versions.

For my development I like the advantages of the git fetch, but maybe for the general public the package version is better?

[ebutera]

I agree that for development the git version is better, but maybe it makes sense to switch to tars for the general use case. I don't have a strong opinion against or in favour of it really.

[KristofRobot]

There was a discussion about a similar issue for raspberry pi on the yocto mailinglist (which has not been concluded as far as I know).

Interesting! Are you referring to this thread: https://lists.yoctoproject.org/pipermail/yocto/2015-July/025403.html ?

In general, I am assuming that someone who would benefit from the git approach has the knowledge to quickly change the recipe to use git - while on the other hand the "general public" might not realize they could change to a package version to speed things up. For that reason I am leaning towards the package version.

OTOH, if there is a need to target specific kernel commits, git would be the best approach (over package + patches) - but that scenario seems unlikely?

Couldnt we have the best of both worlds by using an approach as explained at http://www.yoctoproject.org/docs/1.8/bitbake-user-manual/bitbake-user-manual.html#bb-bitbake-preferences?

 It is common for a recipe to provide two versions -- a stable, numbered (and preferred) version, and a version that is automatically checked out from a source code repository that is considered more "bleeding edge" but can be selected only explicitly.

For example, in the OpenEmbedded codebase, there is a standard, versioned recipe file for BusyBox, busybox_1.22.1.bb, but there is also a Git-based version, busybox_git.bb, which explicitly contains the line

    DEFAULT_PREFERENCE = "-1"

to ensure that the numbered, stable version is always preferred unless the developer selects otherwise. 

[jlucius]

Yes that´s what i meant. In this case the advantage is that we have a stable repository for the package.

Doing both worlds is a good idea.The more common approach I have seen is to set a

PREFERRED_VERSION_linux ?= "4.1%"

in sunxi.inc

Change the recipes for 4.1 to use tar.xz and do a new one for bleeding edge 4.2. It would also be possible with DEFAULT_PREFERENCE but I have seen the PREFERRED_VERSION in most layers.

I can do a patch/pull request later this day or do you want to do one? (if everybody is fine with this idea?)

[KristofRobot]

If you are able to do the patch later today, please go ahead.

I have no preference for DEFAULT_PREFERENCE vs PREFERRED_VERSION - let's do what is most commonly done.

Thanks!

On Thu, Jul 23, 2015 at 8:25 AM, Noxenious notifications@github.com wrote:

Yes that´s what i meant. In this case the advantage is that we have a stable repository for the package.

Doing both worlds is a good idea.The more common approach I have seen is to set a

PREFERRED_VERSION_linux ?= "4.1%"

in sunxi.inc

Change the recipes for 4.1 to use tar.xz and do a new one for bleeding edge 4.2. It would also be possible with DEFAULT_PREFERENCE but I have seen the PREFERRED_VERSION in most layers.

I can do a patch/pull request later this day or do you want to do one? (if everybody is fine with this idea?)

— Reply to this email directly or view it on GitHub https://github.com/linux-sunxi/meta-sunxi/issues/99#issuecomment-123992661 .

[ebutera]

I think it's better to set the DEFAULT_PREFERENCE in the kernel recipe, if i'm not wrong if you set it in a machine .conf include and then modify it (for example to switch from 4.1 to 4.2) all the layers will be parsed again. See also @naguirre commit f4ebd29a5075e94d78b5c31dcf9d856962430e67

[ebutera]

thinking more about it, using PREFERRED_VERSION looks cleaner, just i don't like to have it in a machine .conf file...

[jlucius]

Suggesting PREFERRED_VERSION was just following what I have seen in other layers. I have no preference for either.

FSL, TI: none Raspberypi: http://git.yoctoproject.org/cgit/cgit.cgi/meta-raspberrypi/tree/conf/machine/include/rpi-default-versions.inc Intel: http://git.yoctoproject.org/cgit/cgit.cgi/meta-intel/tree/conf/machine/intel-core2-32.conf

Just tell me your preference and I will do the patch accordingly.

[ebutera]

let's hear from @naguirre too, i don't like it in machine conf but i see that many do so i'm ok for both ways...

[naguirre]

I don't have any issues with git fetching, but i have a very good bandwidth :-)

I like what KristofRobot proposed :

• linux-sunxi_3.4.bb : keep this one as is, it was using git for a while and i don't think there is another way to get the latest version of source code
• linux-4.1.bb : use tar.gz here.
• linux_git.bb (git protocol) : Use DEFAULT_PREFERENCE = "-1" to choose the 4.1 as the best version.

I don't like PREFERRED_VERSION too much. I think that it should be overridden in you local meta or conf if you want to change for another version or package.

[jlucius]

Ok, I have the patches ready, but have not tested them yet. Will do so today and make a push request.

[naguirre]

Thanks for your work !

2015-07-29 9:27 GMT+02:00 Noxenious notifications@github.com:

Ok, I have the patches ready, but have not tested them yet. Will do so today and make a push request.

— Reply to this email directly or view it on GitHub https://github.com/linux-sunxi/meta-sunxi/issues/99#issuecomment-125868903 .

Nicolas Aguirre Mail: aguirre.nicolas@gmail.com Web: http://www.calaos.fr Blog: http://dev.enlightenment.fr/~captainigloo/

[jlucius]

Ok tested it and everything works. New Pull request.

[ebutera]

merged