api mode

[udeved]

@fhdk

Can you make it so that in api mode setbranch auto sync can be deactivated so it only set branch in conf and mirrorlist? Perhaps additional arg -N?

[udeved]

The api stuff replaces spaghetti code in couple of scripts. :) I just need to make sure, the chroot also has the pm.conf set to the branch, after pm package was installed. But this should not be a problem now with api mode.

[udeved]

We can now detect, get and set branch at package level too, so we coould generate a proper mirrorlist for the right branch the system is set. Previously, we either deactivated it, or it wrote default stable branch.

[udeved]

@fhdk

Do you think we probably try to trigger the mirrorlist update at package level with a json update trigger?

[fhdk]

your primary concerns are:

creating a mirrorlist on an empty chroot - Check

pacman-mirrors -a -p $path -S $branch -U $url

changing the default branch - Check

pacman-mirrors -anS $branch -p $path

getting the branch

pacman-mirrors -p $path -G

[udeved]

The are package level concerns, how we handle mirrorlist on install or update. This needs fine tuning. ;)

[fhdk]

Which file are you referring to

/etc/pacman.d/mirrorlist
/usr/share/pacman-mirrors/mirrors.json

[udeved]

/usr/share/pacman-mirrors/mirrors.json

This one.

We could trigger mirrorlist generation if this file gets updated. File based alpm hook.

[udeved]

I need to prevent the package from generating a mirrorlist in chroot, which it does currently, and it overwrites the custom mirrorlist with the buildmirror.

[fhdk]

pm must not generate a new mirrorlist then use -n

pacman-mirrors -n

this will download a new mirror file but not create a new mirror list

[udeved]

Yes, I mean our dev package, not the code itself. We need to fine tune the package with the new api options available to make it work in all cases.

[fhdk]

you will you pacman-mirrors-dev package and then in the dev package it will never generate a mirrorlist and overwrite the existing.

[fhdk]

So we should have in unstable a pm-pkg package solely for using in building iso's?

[udeved]

No, its not really realted specifically to buildiso. Its more general to properly address the shortcomings we had before api mode with the package.

We start out with no mirrorlist, and thats kind of problem, because our package needs to write one on first install.

[fhdk]

If the buildiso thingy creates the mirror list in the above described manner

pacman -ap $path -S $branch -U $url

The mirrorlist is now existing. Subsequently the buildiso installs pacman-mirrors but since the mirrorlist is now existing there is no need to have pacman-mirrors generating the mirrorlist at first run.

A mirror file will be delivered with the package and installed in /usr/share/pacman-mirrors. That mirror file (mirrors.json) will at some point be outdated but calling

pacman-mirrors -n

upon succesful installation will fetch the latest available mirrors.json file from Github and the latest available status.json from repo.manjaro.org.

I have just pushed a new pacman-mirrors-dev pkg with some modified hooks and scripts which only calls pacman-mirrors with -n which will download up-to-date data files but not touch the mirror list.

[fhdk]

Actually I find that this approach should be expanded to the the regular install as well - so an update to pacman-mirrors does not trigger a regeneration of the users mirror list since that regeneration is the mother of all kinds of possible disasters regarding the users updates.

[udeved]

Yes, our dev package is just test bed, I started with hooks there. I had to alter pkglists for build stuff to pull in dev package. Hence I'd love to see better branching with packages.

[udeved]

@fhdk

Cann you push the pkgbuild changes?

[fhdk]

Yes - I have been granted boxit access

[fhdk]

I cant change the github repo

[fhdk]

And likewise Jonathon and Matti

[udeved]

I see, too bad, can youpost your changes here? I can add them.

[udeved]

I don't have admin control to add you on github. You need to ask @philmmanjaro

[fhdk]

That is not an issue - if this works as you expect I will make PR on the core repo

[philmmanjaro]

@fhdk: I added you to the group. Please accept the invite. I like were this all goes.

[udeved]

@fhdk

Nice, but one little problem...

$ buildpkg -p openrc -doxw
 --> Loading compiler settings: x86_64
==> Removing chroot copy [/var/build/manjaro-tools/buildpkg/unstable/x86_64/root]...done
==> Creating install root at /var/build/manjaro-tools/buildpkg/unstable/x86_64/root
 --> Configuring pacman-mirrors: unstable
.: Info Writing mirror list
   pkgbuild        : http://mirror.netzspielplatz.de/manjaro/packages/unstable
.: Info Mirror list generated and saved to: /var/build/manjaro-tools/buildpkg/unstable/x86_64/root//etc/pacman.d/mirrorlist
  -> Installing packages to /var/build/manjaro-tools/buildpkg/unstable/x86_64/root
:: Synchronizing package databases...

All fine.

But...

: Running post-transaction hooks...
(1/3) Configuring pacman-mirrors ...
.: Warning Internet connection appears to be down
.: Info Mirror ranking is not available
.: Info Mirror list is generated using random method
hint: use `pacman-mirrors` to generate and update your pacman mirrorlist.

We need to figure out a way to prevent the package from writing a mirrorlist, if it already exists.

[udeved]

Once the base-devel group is installed in chroot/root, this chroot will be cloned to chroot/$USER. While the original chroot/root still has proper mirrorlist set, the user copy chroot has a ranndom mirrorlist, sinnce the package writes a new random list before the rsync chroot copy.

chroot/root

##
## Manjaro Linux mirrorlist
## Generated on 2017-06-04 21:55
##
## Use pacman-mirrors to modify
##

## Country : pkgbuild
Server = http://mirror.netzspielplatz.de/manjaro/packages/unstable/$repo/$arch

but chroot/$USER

##
## Manjaro Linux mirrorlist
## Generated on 2017-06-03 21:35
##
## Use pacman-mirrors to modify
##

## Country : Romania
Server = http://ftp.lug.ro/manjaro/unstable/$repo/$arch

## Country : China
Server = http://mirrors.tuna.tsinghua.edu.cn/manjaro/unstable/$repo/$arch

## Country : Hungary
Server = http://mirror.infotronik.hu/mirrors/pub/manjaro/unstable/$repo/$arch

snip

[udeved]

For buildiso, it looks different. There we might want a random list generated.

[fhdk]

I have committed some changes to the install scripts for pacman-mirrrors-dev https://github.com/manjaro/packages-core/commit/e1ca56ef521d3a08ac7d6c74acb6842e189deeef

Have look at them - I am not experienced bash scripter - but I could figure the script checked for the presence of a pacsave and that check I adopted to check for a mirror list and act accordingly.

I build a pacman-mirrors-dev pkg from the pkgbuild

[udeved]

Look good, but, it doesn't work. Something seems to happen earlier. Is the random list generation triggered if no network connection?

What I haven't found out yet, how I can set the branch corresponding to the mirrorlist in api mode. Or is the Branch var in pm.conf not used at all any longer?

[udeved]

.: Info Mirror list generated and saved to: /var/build/manjaro-tools/buildpkg/unstable/x86_64/root//etc/pacman.d/mirrorlist

Note the root//etc. Perhaps best to sanitize it to avoid path problems?

:: Running post-transaction hooks...
(1/3) Configuring pacman-mirrors ...
.: Warning Internet connection appears to be down
.: Info Mirror ranking is not available
.: Info Mirror list is generated using random method
hint: use `pacman-mirrors` to generate and update your pacman mirrorlist.

[fhdk]

Is the random list generation triggered if no network connection?

It is - but you are not using the package I build to today for pacman-mirrors-dev

pacman-mirrors.install - post_install() section

if [ -f /etc/pacman.d/mirrorlist ]; then
    /usr/bin/pacman-mirrors --no-mirrorlist
else
    /usr/bin/pacman-mirrors -yf5
fi

pacman-mirrors-install.script

if [ -f /etc/pacman.d/mirrorlist ]; then
    # fetch new data files
    pacman-mirrors --no-mirrorlist
else
    # create mirrorlist
    pacman-mirrors -yf5
fi

This should trigger the --no-mirrorlist if a mirror list exist - right?

pacman-mirrors-upgrade.script

pacman-mirrors --no-mirrorlist

Note the root//etc. Perhaps best to sanitize it to avoid path problems?

I will have a look but I have made some sanitizing in todays package

[udeved]

if [  ! -f /etc/pacman.d/mirrorlist ]; then
    # create mirrorlist
    pacman-mirrors -yf5
fi

I think this might be better.

[fhdk]

And the not operator shouldn't it be outside the []

if ! [  -f /etc/pacman.d/mirrorlist ]; then
    # create mirrorlist
    pacman-mirrors -yf5
fi

And why is this better than if exist only update mirror file

-- The sanitizer - I got it backwards :) - in 5-10 a new package is available

[fhdk]

oh no - I messed that sanitizing - I was using the same one in two places - have to split them :)

[udeved]

The not operator is also accepted outside, but it give a different logic afaik. sometime, bash is strange thing. https://www.tutorialspoint.com/unix/unix-basic-operators.htm http://tldp.org/LDP/abs/html/testconstructs.html#DBLBRACKETS http://tldp.org/LDP/abs/html/fto.html ;)

What about pm.conf? Is it used any longer to store the branch, or is branch only stored in mirrorlist? Ideally, I can set also the branch in pm.conf, ad if it doesn't exist, it will be created. The package would later honour the backup array, ie it would create a pm.conf.pacnew

[fhdk]

It depends on your use of the api.

If you do not use -U then pacman-mirrors.conf is used to store the branch.

This is the only situation where the pacman-mirrors.conf is not used is with -U.

[udeved]

I see, I use almost only -U, but it would be nice to reuse the chroot, if the branch was set in pm.conf. It may be unreliable with mirrorlist only? On buildiso, I would need the branch set in pm.conf. No problem to do it in scripts, but its a nice to have pm feature I think.

[fhdk]

It is not unreliable with the mirror list - as long as the pm installation does not touch an existing mirror list.

With the changes to the install script and hook scripts this shouldn't be an issue looking forward.

I have been around long enough to realize that the pm installation shouldn't touch the mirror list at all unless it does not exist. Seen to many forum issues relating to exactly that which is also why I have added the -y argument to the situations where pm should create the mirror list.

[udeved]

Hmm, I could call pm twice, one time before packages get installed to set branch and build mirror, and second time after all packaged have been installed, to set branch permanently annd to set new mirrorlist. This would work around not having the branch set in pm.conf for the iso.

[udeved]

Perhaps something like this would be nice?

$ sudo pacman-mirrors -a -p /mnt/work -U http://mirror.netzspielplatz.de/manjaro/packages/ -S unstable -W -W -- write branch to pm conf

[fhdk]

I don't see why - the only program caring about pm.conf is pm and pm is not installed in the chroot.

So why bother writing a pm.conf with a branch for an app that is not installed?

The other approach where you call pm a second time seem more logical as the installation of pm not touches an existing mirror list.

When you're just about ready to close and create the iso

pacman-mirrors - a -p $prefix -S $branch -g -m random

which will set the final branch and create a random mirror list with all mirrors

[udeved]

Well, we distribute the chroots for iso. So I need to set the branch in pm.conf in the iso chroot. Otherwise, as soon as user runs pm on the new install, it will set branch to stable.

[fhdk]

When you know that pacman-mirrors is installed you can manipulate that pacman-mirrors.conf almost anyway you like.

It makes no sense to install a configuration file for an app not installed - lets say if pm could install a conf then this conf would have to be maintained two places in code for the api and ex code for the installer. That is prone to go wrong at some point and it makes no sense.

[udeved]

Yes, true, I can sed the file, but, we would rely on the installer to set the branch permanently, the instllaer would have to call pm. Thats a cool thing, but it should not be a depend, eg to select the branch for the install, currently not possible is on todo. The installer basically accesses the same chroot again, similar procedure in general to to create the iso and connfigure some chroot stuff.

[fhdk]

I thought that all releases with exception of the testing stuff was preset with stable branch.

Are you telling me your'e are trying to build an option for the end user to select which branch the user wants?

[fhdk]

So you just want a pm.conf to put on the iso where the only content is a chosen branch?

pacman-mirrors.conf

Branch=unstable

[udeved]

Yes, I have to make it permanent on iso. Not so important in pkg chroot, but iso chroot I need to set it. Would be totally sufficient it this happened on -U too, and it writes a pm.conf with branch.

[fhdk]

I can do that :) But this specific has the potential to screw a users config and it cannot be copied to the final installation as the only content will be the branch.

I will need to check for the existence of a conf at the given target and if target file exists refuse to write the branch.

After thought - maybe a little rewrite and reordering.

It is a little confusing working with you - first you don't want to copy a pm.conf then you want pm to write one.

[udeved]

Your work is already great help, the build scripts already only rely on pm, and ugly code got removed. :)

[fhdk]

Thanks The pm.conf we are talking about - it will not be copied to the end users install?