search

lima-vm / lima

Linux virtual machines, with a focus on running containers
https://lima-vm.io/
Apache License 2.0
14.54k stars 570 forks source link

Automation for updating templates #1347

Open AkihiroSuda opened 1 year ago

AkihiroSuda commented 1 year ago

It is really hard for me to create a PR like https://github.com/lima-vm/lima/pull/1236 to update the template image digests.

We have to have a tool for updating these templates automatically. The tool must retain comment lines and indentation styles in the YAMLs.

afbjorklund commented 1 year ago

Maybe also some means of sharing them, with some kind of FROM system ?

AkihiroSuda commented 1 year ago

Maybe also some means of sharing them, with some kind of FROM system ?

Yes, but that is a separate issue

jlm0x017 commented 1 year ago

Do you already have the link to the updated image? Or is that step needed as well? I expect you grab the checksum provided at the source, rather than generating it yourself, but please do confirm.

AkihiroSuda commented 1 year ago

Do you already have the link to the updated image?

No, e.g., we have to detect the latest version 20230124-1270 from https://cloud.debian.org/images/cloud/bullseye/ , but I'm not sure what is the robust way to do this.

w3m | grep might be enough, but seriously we should also consider adopting some machine learning stuff.

afbjorklund commented 1 year ago

If I remember correctly, there was some standard metadata for scraping upstream for new releases to package...

Like https://wiki.debian.org/debian/watch

Maybe something like that can be used here, to "describe" the various vendors and where they put their binaries ?

Some thing simple, with place holders for date strings and checksums.

afbjorklund commented 1 year ago

Example output:

https://qa.debian.org/cgi-bin/watch?pkg=containerd

Unfortunately, the others are not available as packages. But maybe something similar to this, but for images:

https://repology.org/project/nerdctl/versions

lobshunter commented 1 year ago

What about a naive bash/python script to replace placeholder strings in yaml? Something like:

ubuntu_image = `w3m | grep -e "xxx"`
sed -i "s/UBUNTU_IMAGE/$ubuntu_image/g" ubuntu.yaml

The robustness solely relies on consistent file naming of upstreams(counting on hyrum's law). Sadly I didn't find any public tool to retrive latest released images.

AkihiroSuda commented 1 year ago

I'm now planning to use yq

balajiv113 commented 1 year ago

How about using libosinfo (osinfo-db os) ??

I could see their db has info of different os variants https://gitlab.com/libosinfo/osinfo-db/-/tree/main/data/os

We might need to write a python wrapper on top of this library (the tool osinfo-db is not giving out info on image download URL's by architecture that is present in the xml file)

afbjorklund commented 1 year ago

yq can read xml too

yq -p xml -P
afbjorklund commented 1 year ago

The library and database are licensed under the terms of the GNU LGPL version 2 or later.

https://libosinfo.org/

AkihiroSuda commented 1 year ago

osinfo-db

Doesn't seem to contain permalinks: https://gitlab.com/libosinfo/osinfo-db/-/blob/ea8a7974a1f7189953c80fa9b1478b1ff8a75f8e/data/os/ubuntu.com/ubuntu-23.04.xml.in

    <image arch="x86_64" format="qcow2" cloud-init="true">
      <url>https://cloud-images.ubuntu.com/lunar/current/lunar-server-cloudimg-amd64.img</url>
    </image>
balajiv113 commented 1 year ago

yq can read xml too

True, but if we can use API it would be great. Else with yq we might need to read all xml files under each folder that we are interested in. With API i think it will be more managable

GNU LGPL version 2 or later

I thought since we are going to use this more of a build tool (Mostly a github actions workflow) this should not be a problem.

afbjorklund commented 1 year ago

It was mostly referring to the "and database", most of the tools actually seem to be GPL v2 (and to require glib)

balajiv113 commented 1 year ago

Doesn't seem to contain permalinks

True :( Supported examples are as below

AkihiroSuda commented 11 months ago

I guess we can consider using GPT

image

afbjorklund commented 3 days ago

For Ubuntu, this is implemented in the ironically named "simple streams" (it's 14M):

sudo apt install simplestreams ubuntu-keyring

sstream-query --json --max=1 --keyring=/usr/share/keyrings/ubuntu-cloudimage-keyring.gpg http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson release='noble' ftype='disk1.img' | jq -r '.[] | [.item_url,.arch,.sha256]'

https://philroche.net/2018/02/12/ubuntu-cloud-images-and-how-to-find-the-most-recent-cloud-image-part-1-of-3/

The JSON+GPG file is: http://cloud-images.ubuntu.com/releases/streams/v1/com.ubuntu.cloud:released:download.sjson

There is also a highlevel command:

sudo snap install image-status

$ image-status cloud-release
focal    amd64  20240626  disk1.img
jammy    amd64  20240627  disk1.img
mantic   amd64  20240619  disk1.img
noble    amd64  20240622  disk1.img

Where "disk1.img" is the old spelling of QCOW.