post-fetch command

[linacs]

Hi,

a customer of mine is using Cooker to retrieve different layers including one from his hardware vendor. This vendor made some technical choices that are very intrusive and that we don't want. Sadly, they used strong expressions such as "=" inside layer.conf, or rogue includes. There are also some inherits that are no longer compatible with our Yocto's version. This is why there is no other choice than to apply a little patch after fetching the layer sources (otherwise, we would have to fork their layer, but that could be annoying later, when the layer will evolve, to maintain it).

We were thinking of a "postfetch" key inside the menu, which would be executed with a simple "system()" in Python, just after fetching the sources. In our case, it would be a "patch" command call. What do you think of it? That could prevent us to break the Cooker workflow into two commands (with this patch application between the two). The customer is ready to pay me to do it and mainstream it if you're alright with the idea.

Thanks, Gilles

[pboettch]

a customer of mine is using Cooker to retrieve different layers including one from his hardware vendor. This vendor made some technical choices that are very intrusive and that we don't want. Sadly, they used strong expressions such as "=" inside layer.conf, or rogue includes. There are also some inherits that are no longer compatible with our Yocto's version. This is why there is no other choice than to apply a little patch after fetching the layer sources (otherwise, we would have to fork their layer, but that could be annoying later, when the layer will evolve, to maintain it).

In what way is it harder to maintain a (floating) patch rather than a fork of the their layer?

I see your point and it might seem a good idea to add this hook. But then we would need to add other hooks at different stages as well.

The hardest problem I think is that this hook would modify layer-code. How do you want cooker to check whether the changes done by the hook are already applied by a previous call of update?

I strongly think forking the layer and maintaining it is much easier for you and your client in the end.

I think policy for cooker should be to not change sources (layers) during its commands.

[linacs]

Thank you for your answer Patrick. The problem with forking is that the hw vendor makes evolutions from time to time (e.g. kernel patches), including Yocto version changes. We would have to reinject new modifications, instead of just changing the commit revision in the menu. And of course, both my customer and its vendor use separate custom Git repositories... Our patch is only 3 or 4 chunks, very tiny. It's just that without it, it breaks (one of the things is that the hw vendor forces a dependency to meta-imx, which is so bad...). It's no big deal to maintain.

Right now, we apply manually the patch, and everything works fine with Cooker (it's like other working modifications, committed or not, that we do when we are working on our layer). So sometimes, the simpler the better. I mean, it would be our responsibility to use this postfetch command as we want (it could be a patch like ours or anything else! We can imagine tar or detar, creating other directories, signaling to a server that we have fetched, whatever). We could consider adding hooks for success or failure of this postfetch command (in our specific case to detect if the patch is already applied? Well anyway it can be ignored if the modifications are already here!), if you're thinking of a "luxury" version as some kind of a state machine, but done is better than perfect. :)

[boucman]

hmm,

on the one hand, I had that problem once or twice with vendor layers, so there is a real need to patch layers and not simply "blindly pull" from upstream

on the other hand... just fork it and maintain your own git repo. If the layer is outdated, the chance of upstream disapearing are probably higher than the chances of upstream updating the layer...

Anyway... keeping a fork with just your patch already applied and rebasing your patch if/when upstream updates stuff would definitely be simpler and more robust than working with patches.

[pboettch]

totally agree @boucman as integrators we should not rely on (vendors) upstream repositories. Even Poky and OpenEmbedded should be locally mirrored.

@linacs I strongly recommend to not patch but to maintain a mirror for your situation. Ask yourself the question, will this layer still be there in 5 years?

However, I'm evolving regarding the hook-implementation. At first I was really against it, but hearing ideas from others (off-github) I think this might be a good idea.

[linacs]

I hope the repositories will still be here in 5 years, but in 10 years, it's less sure (however we buy the hw with this guarantee). What I do with my customers (or what they usually do), is a local archive for big deliveries (plus the download directory and patched sources from archiver bbclass). But this mirror idea is interesting.

It's just that the question is also about workflow during development. We use Cooker as some kind of specialized repo, to set up a new organization quickly for development and make it evolve frequently (which is what is intended with the different and heterogeneous sources in the menu, anyway) (at first, the customer used git links, so that git clone was also pulling the other repos, but it was much less powerful and harder to maintain. I also put Cooker for another customer who uses it with Gitlab CI). I'm thinking that some kind of tool to automate the mirroring of different remote sources and push them up-to-date in another repository (which is behind a VPN for this particular customer) (I have another one that pushes to its SVN, no kidding... HUGE pain), could be very interesting (cooker mirror ? :) ). I miss Git expertise here to be more relevant. But we always have the same problem of balance between practical and "solid" (and there might be more than one solution that could fit to all problems).

I'm glad to contribute to making your position evolve on this topic! :)

[cpb-]

I think that if you implement the post-fetch hook to run a script that applies a patch, you will need to implement pre-fetch to run a script that executes patch -R on the same patch, and that doesn't fail if the patch wasn't applied previously (for first run).

[cpb-]

BTW cooker mirror looks interesting to me too.

[pboettch]

I suggest to discuss hook-related questions here #123 .