I can help host a mirror etc.

[marekm72]

Hello, I'm the one who asked about the unofficial forum on the official one (thread hidden and locked now - not that I didn't expect that). It would be good if all this useful info lives in more places than just one, then it won't be too easy to take down etc. I'm not brave enough to set up a public Jenkins, as I'm not sure if/how it can be secured properly. Running a small local ISP in the north of Poland, have have some bandwidth available, some IPv4 and IPv6 address space, and a small XCP-NG server where I can run a few VMs. Feel free to email me at marekm at amelek dot net

[dd010101]

There is option to make for example script with git and GitWeb to make self-hosted public mirror as backup of this and vyos-build/vyos-missing repository for complete backup.

Do your have capacity to run Jenkins/build VM? The rough requirements are specified in the readme. This VM doesn't necessarily need to be publicly facing or to have high availability - it can also automatically push packages to another VM if that would benefit your infrastructure. Jenkins periodically checks for source code changes and it triggers build and push of specific package in question. That's how the official package repository works as well, it's fairy automatic once the setup is completed but some maintenance is required.

If you do then you can try to follow the instructions and see how far you get. In theory you should be easily able to reproduce the package repository with the current instructions. If you find something non-functional, unclear or something you don't like then just post it here or as another issue so we can improve the instructions.

[dd010101]

It would be good if all this useful info lives in more places than just one, then it won't be too easy to take down etc.

There is option to make for example script with git and GitWeb to make self-hosted public mirror as backup of this and vyos-build/vyos-missing repository for complete backup.

Just follow up on the self-hosted git mirror part - if you want some ideas how it can be done here is simple example.

[dd010101]

There was error in the git mirror example where nginx location for git daemon conflicted with gitweb and this caused the issue with missing styling for gitweb. See corrected nginx example for git-http-backend.

[marekm72]

OK, should work now, please test - https://git.amelek.net/

[dd010101]

All works - gitweb works, can be used to view and download source code. Git clone/fetch/pull via the URL in the gitweb also works, git push is correctly forbidden. IPv4 and IPv6 both work. 👍

[marekm72]

Project descriptions updated manually (copy-paste from GH), as the script left them as default "edit this description file".

[GurliGebis]

Would it be possible to mirror all the repos from the vyos github org? Just in case they decide to make them private (so we at least have a snapshot)

[dd010101]

Mirror for this project is for take-down purposes. That's something else than having mirror as backup against evil. Evil use-case isn't as easy as to just have mirror - if they go evil then it's likely they wound purge contents of the repositories first, thus your mirror would sync empty repos and thus everything would be deleted. You would need to have versioned backups and stop syncing if they go full evil.

The size isn't too bad (~500MB) thus you could...

Should you? I don't see much benefit but you can if you want to...

#!/bin/bash
set -e

dataDir="data"
reposDir="repos"

mkdir -p "$dataDir"
mkdir -p "$reposDir"

page=1
while [ $page -le 1000 ]
do
    echo "Processing page $page"

    path="$datadir/repos-$page.json"
    curl -sS --fail-with-body "https://api.github.com/orgs/vyos/repos?per_page=50&page=$page" -o "$path"

    emptyPage=true
    while read gitUrl
    do
        directory=$(echo "$gitUrl" | grep -oP '([^/]+).git')
        fullPath="$reposDir/$directory"
        if [ -d "$fullPath" ]; then
            echo "Updating $gitUrl in $fullPath"
            git -C "$fullPath" remote update
        else
            echo "Cloning $gitUrl as $fullPath"
            mkdir -p "$fullPath"
            git -C "$fullPath" clone --mirror "$gitUrl" .
        fi

        emptyPage=false
    done < <(cat "$path" | jq -c -r '.[].clone_url')

    if [ $emptyPage = true ]; then
        echo "All done"
        break
    fi

    page=$((page+1))
done

You can call this script periodically, like daily, and it will sync all VyOS repositories - you need versioned backups as well, like duplicity/duply to have chance to fight with purged repositories... You can use same gitweb/nginx for viewing and cloning - as for generic git mirror - just use this script in place the original one and choose different directory/vhost since naming conflicts.

You have always option to ask via GPL but that isn't a good choice since they would give you the least useful form they can - like without history - maybe even stripped. Also they said on forum something like they know, that GPL would force them to give source code only for those who received binaries from them - thus you would need subscription to even qualify. Having git mirror isn't bad idea per say but the question is - how likely this is? Would you even want to continue to use VyOS if they did go so far?

[GurliGebis]

Something like that yes :) What I'm fearing is, that they really don't want people to have >1.5 without paying for it - so by saying "you can build it yourself" (knowing that doing that is not something people could), they were giving the impression of allowing it. Now, when we are giving people a somewhat easy way to get back to building the images again, they might think of something else (like moving the github repos into private) - that is why I'm thinking of, if we need to have a mirror of them as well - to be guarded against that.

I'm not saying they will do that, but time will tell 🙂

[marekm72]

I should be able to look into it next week, one important question is how long these versioned backups need to be kept. My XCP-NG boxes don't have much storage (SSD, small and fast), but I have some internal TrueNAS boxes (HDD, large and slow) meant for backups (not accessible from outside, but I can manually restore something if ever needed).

Continue to use - why not, last free version could be forked like Vyatta was... under a different name of course. And only built from sources, where some bad change with delayed effects can be found and reverted even after a long time. Only trust the Debian APT repo and build the other .deb packages ourselves from sources, not only for 1.4 but 1.5 too even if they are still available.

[dd010101]

@GurliGebis

What I'm fearing is, that they really don't want people to have >1.5 without paying for it

I believe the closure of repository was about the reasons they said. At the end of day it's business. There are more benefits for them not only revenue but I see the reasons they said are true.

You may think that some of the reasons are nonsense like bandwidth cost but there are dumb ways how you could host the packages that would cost thousands of dollars so maybe they use the least sensible hosting for egress heavy data, who knows. That's why the next in line is 1.5 since that will eat the bandwidth now...

There is no reason to close the github unless they find revenue as reward and I don't see that happining since open source is plastered all over their brand. The users who use github won't pay the subscription even if their github is closed - there is no revenue to be bad. If this changes and they start to target small businesses and home users with their subscription then that gives them some, yet far fetched, reason to close their github.

@marekm72

Continue to use - why not, last free version could be forked like Vyatta was...

The fork-after-evil use-case would benefit from complete mirror for sure and since it doesn't look like big deal to host then why not.

I should be able to look into it next week, one important question is how long these versioned backups need to be kept.

Long enough that it will give you enough time to notice they did the nuke so you can stop the backups before they all fill with deleted contents.

If you want long time then look at duply, it's easy way to setup duplicity backups. These are delta file level backups - you can put them on local fs or over variety of remote protocols anywhere. You can backup just directory (or the whole fs on file level) and the backups are efficient for example you can have setup where you keep 2 full backups and increments between in way you have daily backups that take on average 3x the space of the data for months of backups. In this case few GB for all GIT repositories for months of daily backups.

If you would have Jenkins as build host then for sure don't backup the whole OS long term since there is a lot of temporary data - that would take a lot of space to store long backups and there is no reason anyway. That's why I would store long term backups only for pieces used to assemble everything not whole system - that's all the GIT repositories. Snapshot of whole OS is much easier to restore but you pay for it with storage - choose your poison type of thing - but here the more complicate restore is much sweeter poison I think.

[Crushable1278]

like moving the github repos into private

https://github.com/vyos/vyos-build/blob/current/.github/workflows/repo-sync.yml https://github.com/vyos/vyos-1x/blob/current/.github/workflows/repo-sync.yml

I can't help but wonder what this is for. It seems to have shown up in the last few weeks.

---

I'm not sure they can close the source of vyos-build and other vyos-* packages/repos as one of their "selling points" to being a contributor is that you don't have to assign the rights to your code to VyOS. This could also be why they closed development of the commands and other changes that were in and subsequently removed from vyos-1x to support VPP.

Also of note is the comment that future package repositories will be S3 bucket based rather than HTTP/S.

With the Sagitta build figured out, extending to Circinus is trivial. I personally have my doubts that when Circinus is released to LTS in 3-4 years, the code will be as available as it is today.

[dd010101]

like moving the github repos into private

I can't help but wonder what this is for. It seems to have shown up in the last few weeks.

Having current branch public is benefit for them - users report bugs, sometimes even fix the bugs, nobody will pay for current branch anyway and it does allow them to use the open-source facade. That's why locking current behind paywall doesn't make sense.

What does make sense though is to remove the stable branches from github and leave only current/development public. That's the most likely move if they ever will turn anti open-source rather than making repositories private or taking them offline.

...why they closed development of the commands and other changes that were in and subsequently removed from vyos-1x to support VPP.

Direction of VPP has clear incentive - turn something that would be free to paid addon. I do expect we will see more paid addons in the future. Let's hope they don't gut the VyOS of existing functionality to make the addons...

[marekm72]

Another former VyOS(tm) forum user (probably the first one to figure out that Jenkins magic, then banned and all posts removed) has mirrors on their own Forgejo - https://git.skysolutions.fi/explore/repos

As for VPP, I don't have that much traffic yet and I actually think it may be a good idea to offer that as a paid addon (if only the VyOS core was free) - if you are big enough to have so much traffic your CPU can't keep up without VPP then you probably can afford a subscription (or a Cisco). But reading more about VPP now makes me curious and want to try it someday on plain Debian with BIRD - all free (though quite complex, will take some time to learn).

[dd010101]

Another former VyOS(tm) forum user (probably the first one to figure out that Jenkins magic, then banned and all posts removed) has mirrors on their own Forgejo - https://git.skysolutions.fi/explore/repos

Kryptonian has also fork of vyos-build hosted there, named vyos-build-custom (and couple more) with changes to make it buildable, only for sagitta/current though. Similar but more invasive changes as found here. How did you find it?

Side note: the mirror via Forgejo (or Gitea, or Gitlab) isn't as universal, as the script and the method of backup outlined above. Since if you setup manual mirror - it won't auto-discover new repositories. It also won't do anything if stable branches are purged - unless you setup the same long-term versioned backup for whatever forge you use.

[Neboer]

Hi I'm Neboer and I'm very appreciate for all your works. I'm trying to build an "LTS" vyos image with the help of your site. Please keep this magic happen and make building an LTS iso much easier.

[GurliGebis]

Well, it won't be the specific LTS files they release, since that is not possible (all packages are not locked to a specific version), so it will be the latest LTS + all the fixes since then (which one could argue is a good thing) 🙂

[dd010101]

Closing in favor of #19