local build?

[stsp]

Hi.

Have you considered introducing the local build option? That is, all the archives are downloaded from your git with git pull and then built locally. I want to integrate your project with the launchpad build farm and create an ubuntu PPA. I don't think the build farm allows to DL an arbitrary files, and even if it does, I'd prefer a local build as a start. What do you think?

[jwt27]

That would increase the size of the repository immensely. A git clone would download all versions of all supported packages. That's a severe penalty for most users who only need one version. Doesn't seem like a good idea to me.

[stsp]

Yes, all versions are not needed. Maybe providing only latest version in a separate repo, and link it to the main repo with the git submodule? Then users won't need to D/L it. Unfortunately launchpad does not support this trick yet, but it soon will: https://bugs.launchpad.net/git-build-recipe/+bug/1733603

[jwt27]

That still requires a large git repo with all possible packages and versions. Deleting old files won't help, git remembers all commits.

Maybe an easier option would be doing a shallow clone of the original source repos at a specific version tag. Possible issues there: djgpp isn't on git, and gcc must be unpacked using unpack-gcc.sh from djcross to apply target-specific patches.

I'm not familiar with Ubuntu's PPA system, but do they even allow running custom bash scripts on a build farm? If so, you'd think they would provide basic utilities like curl on their build systems.

[stsp]

That still requires a large git repo with all possible packages and versions. Deleting old files won't help, git remembers all commits.

The size will accumulate, but initially there would be only one version. Launchpad will do sequential pulls that will bring only the newer version to it. It doesn't need to re-clone the repo entirely before every build. And even if it did, it seem to be using shallow clones: https://bugs.launchpad.net/snapcraft/+bug/1617783 Users will never have to pull from that repo at all. So IMHO this may be acceptable.

Maybe an easier option would be doing a shallow clone of the original source repos at a specific version tag

I think it is possible only if you link them as a submodules.

I'm not familiar with Ubuntu's PPA system, but do they even allow running custom bash scripts on a build farm? If so, you'd think they would provide basic utilities like curl on their build systems.

You can do anything during the build, and use any tools. If your build scripts run bitcoin miner, it will run. :) But I am not sure if it gives the internet access to the build, as this would be totally insecure wrt bitcoin miners. And even if it does, this will trouble the integration work. Every time I need to test the recipe (which I will do initially locally), I will have to D/L things again and again. Local build will help a lot.

[andrewbird]

@stsp I'm not sure if it's possible with a launchpad git based recipe, but with the bazaar version I believe you can nest other content before building. See the nest and merge keywords at the following link http://doc.bazaar.canonical.com/plugins/en/builder-plugin.html

[stsp]

But it still seems to have be in the VCS. I.e. it doesn't look like it allows to just D/L and unpack some tgz, or does it? And as for being in the repos - this "merge" stuff resembles the git submodule technique to me in practice. Both allow you to get things in from another repo, just with slightly different techniques.

[andrewbird]

I imagined that nest allowed import from a different repo (in this case just containing the tgzs) whilst under the control of the recipe, then the build scripts in the master repo could be modified to test for existence of each tgz before failing back to wget if missing for the normal case. Just a thought...

[stsp]

How is that different from importing another repo via git submodule?

[andrewbird]

Doesn't git submodule require its configuration in the main git repo, which means all normal users of the main repo end up pulling down the submodule to build? If I'm correct about nest, only your recipe would be aware of the second repo containing the .tgzs. I never tried either nest or git submodule, so this is all supposition on my part.

[stsp]

Doesn't git submodule require its configuration in the main git repo

Yes.

which means all normal users of the main repo end up pulling down the submodule to build?

No. https://git-scm.com/book/en/v2/Git-Tools-Submodules

Here we’ll clone a project with a submodule in it. When you clone such a project, by default
you get the directories that contain submodules, but none of the files within them yet:
You must run two commands: git submodule init to initialize your local configuration file, and git
submodule update to fetch all the data from that project and check out the appropriate commit
listed in your superproject:

So submodules do not get fetched automatically. This is why users won't ever need to D/L them, and this is why I think this is the same thing as the nest that you've found.

[andrewbird]

Okay, it sounds like you have it all figured out. Will you publish the debs in the dosemu2 PPA, or somewhere else? I'm looking forward to getting my local install removed and having a proper packaged djgpp install, thanks!

[stsp]

Not me, its a proposal for @jwt27 :) I can only do the LP integration work. Maybe you can add the local install possibility yourself? :) Then we'll get things started.

[jwt27]

Looking into this asap. Right now I'm having a horrible reaction to some new medication and cant think straight. Please be patient...

[stsp]

Thanks, please get well soon. There is no rush anyway, as LP currently doesn't yet support submodules anyway.

[stsp]

Hmm, it seems for launchpad build we'd also need DESTDIR support.

[stsp]

@jwt27 how are you doing btw?

[stsp]

October 2018
jwt27 has no activity yet for this period.

... says github. Hope this is not the only possible answer.

[jwt27]

I had some kind of mental breakdown a while ago. Was put on various drugs, they all just made it worse. I'm really confused now and I get severe panic attacks when I try to concentrate on anything, so I don't think I'll be coding for a while.

Plus, I now have this to look forward to: https://en.wikipedia.org/wiki/Benzodiazepine_withdrawal_syndrome

[stsp]

Ah, thanks for an update! So you have not what I thought you have, which is a very good news. Sometimes the programmers have to solve the health care tasks, and I am glad yours is not from the unsolvable category, even if it looks tough. :)

[jwt27]

moving discussion to #10