Git plugin

[echo-bravo-yahoo]

Hey! I'll probably work around this for my current use-case by using the copyDir and runatboot plugins, but it would be really helpful to be able to provide an HTTPS or SSH git destination to clone so that the host system doesn't have to download software it won't use while building a rpi image. Also important is some way to run any relevant build or link steps.

Maybe something like:

--plugin git:"repo=git@github.com:gitbls/sdm.git|cert=./github-cert|runafterclone=npm install && npm link"

[gitbls]

Thx. This was suggested by @1stcall over here: https://github.com/gitbls/sdm/discussions/147

I don't understand your comment about " ..so that the host system doesn't have to download software it won't use while building a rpi image.". Are you referring to git itself?

Also note that you can easily do this yourself with a personal plugin. In what way would a git plugin be more desirable/better than rolling your own in a personal plugin?

[echo-bravo-yahoo]

Hey - I was not referring to git so much as the repository being downloaded. For instance, my raspi that burns images with sdm (the "parent") does not need to have the software that the burnt images ("children") need, but right now I'm downloading the repositories to the parent, then using the copyDir plugin to move the repo into the child and runatboot plugin to do a few build tasks that have to happen on the child (node-gyp doesn't differentiate between armv6l and armv7l, so builds done on a raspi 4 don't produce functional binaries for raspi 0, yay...).

This results in me having some wrapping scripts around sdm, partially defeating the point of using sdm. I will probably roll a personal plugin in the future, but this seems like a very common use-case with fairly little deviation. Interestingly enough, @1stcall encountered the same two use cases that are going to lead me to a custom plugin - cloning a git repo and adding apt sources.

I think that developing primarily for raspi 0 is leading me to more edge cases than raspi 3/4/5 development because of the spotty support for armv6l.

In short:

• I have a work-around for my use case
• But I think a generic, config-driven way to add git repos to a child image would be really valuable
• I'm happy to close this issue in favor of #147 or edit one issue to represent cloning repos and one adding apt sources

[gitbls]

I think maybe you misunderstood my question. I'm not questioning the value of a git plugin, rather I'm wondering whether it's better to have a prescriptive method implemented by an sdm-provided plugin or letting you do it the way you prefer in your own personal plugin. (Side note: I am nost definitely not a git/github expert, so if I'm missing a generalized way to download a repo, that would be why 🙄)

I can think of two methods for a git plugin to clone a repo into an IMG. Each has issues 😲:

1. git is installed on the host (where sdm is running), and the git clone is done in phase 0, into a directory that you specify to the git plugin.
   • Advantages: Host doesn't need to have git installed
   • Disadvantages: git needs to be installed on the host; File ownership of the cloned files could likely be problematic since the IMG's accounts and /etc/passwd are not yet set up and not accesible in Phase 0
2. git is installed into the IMG, and the git clone is done in Phase 1 or post-install. In this case, the git plugin would need to install git if it's not installed, and also copy the Cert into the IMG during Phase 0 so it's available for the git clone (when the host system is not accessible).
   • Advantages: Removes git requirement from the host; file ownership can be set using /etc/passwd in the IMG
   • Disadvantages: Requires git in the guest; requires copying Cert into the guest

Which scenario are you suggesting?

As far as adding apt sources, I haven't looked into it yet, but I think it should be relatively straightforward so likely to land it.

[echo-bravo-yahoo]

I see! Sorry for the misunderstanding.

I think I'm coming at this as a user of these tools which allow specifying git repositories as dependencies:

• npm
• vim-plug

As a user, it's really convenient to be able to enumerate dependencies this way.

However, these tools require git, their own toolchain, and have the ability to be opinionated about what sorts of things you're pulling down in a repository. I can't see a way around needing git if the goal is for the result to be a git repository; Github allows downloading zips of branches/tags, but they do not include the git history, and thus can't be easily pulled/updated later, which I think is a large part of the appeal of this way of managing dependencies.

Most repositories you'd pull down have some additional build steps you'd need to run afterwards, which leads me to believe the most general solution is running everything in phase 1. You'll have to provide a lot of things before-hand (see example below), but I think the standard clone -> build workflows will be a lot cleaner if they run in phase 1. Additionally, since I think it's important that they remain git repos (e.g., have a .git directory), I think it's fine to install git in the IMG; it'll be needed if the user wants to update their dependency from its upstream. This strategy closes the door for some cross-compilation setups where your host is substantially beefier than where the IMG will run, but I think that use case is much rarer (and probably a better fit for a personal plugin anyway).

Example customization to clone, install dependencies, and add to PATH a nodeJS project through git:

sudo sdm --customize
  --plugin copyFile:"from=/.ssh/github|to=/.ssh/github"
  --plugin copyFile:"from=/.ssh/conf|to=/.ssh/conf"
  --plugin apps:nodejs
  --plugin mkdir:"~/workspace/wastewater"
  --plugin git:"repo=git@github.com:echo-bravo-yahoo/wastewater.git|cert=./github|to=~/workspace/wastewater|runafterclone=npm install && npm link"

[gitbls]

Thanks. I prefer the Phase 1/post-install approach over the Phase 0 approach, but since there is sensitive security material (ssh key) involved, I wanted to be clear on it and check your concerns with it.

While most of the work would be done in Phase 1 or post-install phases, some plugins do some work in Phase 0 as well (e.g., copyfile). Also, it would be safter to process the git plugin in post-install, after all the Phase 1 stuff is done, eliminating the potential user's plugin ordering challenge.

Another point about your example worth mentioning is that using ~/something as an argument to a plugin won't work. For two reasons. First, what user would you expect that to be for? sdm has no clue, so best to use /home/username/workspace/... Secondly, due to a shortage of unused non-alphanumeric ascii characters, sdm uses "~" internally to separate the list of plugins specified. So, even if the first wasn't an issue, the second will get ya. 😲 (Just mentioning it to save you a few head-scratching minutes!)

That said, the general issue here is that there's no automated way to get the complete list of dependencies for a repo in a way that's directly consumable by sdm. Most githubs seem to have a textual description of the steps you need to follow to install. Very few have a complete and reliable installer, but even those don't really describe the installation in terms of dependencies.

Too bad github doesn't provide some structure around dependency visibility for tools such as sdm to use.

Sigh!

I've got a git plugin on my wishlist, and will have a good look at it after the holidays.

[1stcall]

Personally I would prefer the git plug-in to run in phase 1. I / The user would be responsible for including any dependencies within the apps plug-in listed before the github plug-in on the command line.

The runafterclone idea would also be very useful. It would also need to run in phase 1 from within the directory just cloned. My use case would probably just run a ǹpm installthen ǹpm run build as I am mainly using node or svelte apps that I am working on.

[gitbls]

I'm curious why Phase 1? Pedantically, phase 1 is when stuff gets installed. post-install is for doing things that depend on the stuff that got installed in Phase 1.

In my mind it's worth having zero or very few order dependencies, aka hidden breakage, among plugins within a phase. There's only one minor sequencing requirement in sdm at the moment (user plugin and $myuser).

Does anyone want a user argument? 🤔

[1stcall]

post-install works fine. I was confusing post-install with 1st boot. I would like to minimise anything going into 1st boot.

[gitbls]

That's exactly why I built sdm in the first place. When I boot a Pi I want it to be as immediately useful as possible, but also end up with a properly-configured system.

[1stcall]

I don't personally don't see the need for a user argument. I can specify /home/[myuser]/[mysubdir]/ for the destination in post-install as the home directory has been created by that point.

[gitbls]

You will certainly be able to use it without a user argument. But, if a user argument is specified the plugin should try to make it work seamlessly for folks who want it to just work in a more prescriptive manner, just like the other plugins.

The beauty of Linux 👍

[1stcall]

Thinking about it. There could be a file ownership issue. Maybe run the the git clone command with su [user] or sudo -u [user] could work.

[gitbls]

@1stcall are you suggesting that there could be a file ownership issue when the git clone is done, or afterwards when using the cloned copy? I've been thinking that a chown=group:user argument for the git plugin would be important for using the clone, but don't think there's any file ownership issue on the git clone, so not planning to do anything for that.

[1stcall]

Yes. As nspawn runs as root, all of the files cloned would be owned by root:root. If a user argument is passed. I would suggest you use sudo - u git clone. chown would work, but that seems to be fixing a problem you didn't need to have.

sudo -u would also use the credentials of the user stored in the users $HOME/.ssh directory & $HOME/.git for github.com.

[1stcall]

I don't know @echo-bravo-yahoo use case for the runafterclone argument, but if only for security reasons, this should not be run as root unless absolutely necessary.

[echo-bravo-yahoo]

runafterclone was just a thought, given that most of the repos I would clone need a build step of some sort.

You can already get the desired behavior using the runatboot plugin, but I thought I'd mention the possibility of runafterclone for very short commands (e.g., npm install / make). I do not understand when this would run relative to other parts of the sdm configuration, so it may be unreasonable to provide an in-line command to run there.

[gitbls]

I like the idea of runafterclone (although maybe not the name 😂), since it eliminates a potential hidden ordering dependency between the git plugin and runatboot plugin.

But, haven't thought about this in detail b/c I've been busy with a few other things.

[gitbls]

Here's what the git-clone plugin looks like at the moment. Appreciate your thoughts/questions of course!

Arguments:

• repo — Where's the repo?
• gitdir — Path to provide to git
• gitsw — Additional git clone switches you might want
• user — Username. User must be specified, and must already exist. root is a valid username
• preclone — Command or @script to run before the clone (will be run as specified user)
• postclone — Command or @script to run after the clone (will be run as specified user)
• gitphase — 1 or not specified to run during Phase 1. post-install to run there instead
• cert — Path to cert on host. Will be copied into /etc/sdm/assets/git-clone for later use
• logspace — If specified, plugin will print start/finish log space information

One other question: I don't use certs with git clone. What should this plugin do with the cert? Any good explanation or doc pointers on this?

@1stcall @echo-bravo-yahoo

[gitbls]

The above description is implemented in V11. Except for cert, which is NYI. Appreciate your feedback on it.

[gitbls]

Closing due to lack of activity.