search

openSUSE / mentoring

The openSUSE Developer Mentoring Program
http://101.opensuse.org
60 stars 49 forks source link

Develop a lazy reposync service #206

Open agraul opened 9 months ago

agraul commented 9 months ago

Project Title:

Develop a lazy reposync service

Description:

Uyuni is an open source systems management solution that can be used to manage various Linux distributions using a powerful web UI and API.

Uyuni uses a component, reposync, to mirror linux package repositories for deb and rpm packages. reposync in its current form first downloads all packages, stores them in the filesystem, extracts metadata (e.g. name, version, dependencies, etc.) from packages, and then populates the database with the extracted data.

This project idea is to create a new, lazy, reposync. Instead of downloading all packages first and using them as the metadata-source, the new reposync uses the repository metadata (Packages for deb, primary.xml for rpm) to populate the database. At a later time, the actual mirroring/downloading of the packages takes place.

Deliverable:

There are two parts to this project. It would also be possible to only do the first part for now.

  1. A new component that reads repository metadata, parses it, and populates the Uyuni database
  2. A daemon that downloads packages and stores them in the filesystem

Mentor:

@agraul, @cbosdo

Skills:

Skill Level:

Project Size:

Get started:

rxbryan commented 8 months ago

Hi @agraul,

I'm Bryan Elee. I believe you're mentoring this project.

I would like to know if there are any tests or issues that need to be addressed before someone can start working on this project, aside from the "Get Started" instructions.

cbosdo commented 8 months ago

I would like to know if there are any tests or issues that need to be addressed before someone can start working on this project, aside from the "Get Started" instructions.

Getting a development Uyuni server is a must as you will need it later to test your work and experiment. Other than that we would like to see some code of yours with the application. This can be a fix for a simple issue like those listed in https://github.com/uyuni-project/uyuni/issues?q=is%3Aissue+is%3Aopen+label%3A%22good+first+issue%22 but if you feel like getting started with a prototype for your project that is fine too.

cbosdo commented 8 months ago

Note that we are moving the server to a big container: I just documented how to setup sumaform for this in https://github.com/uyuni-project/sumaform/pull/1518 This is the recommended way to go.

waterflow80 commented 8 months ago

@cbosdo, Seems like the link for creating the default storage pool is no longer valid.

I've tried to create a storage pool this way:

$ mkdir -p /uyuni/pools/default
$ virsh pool-define-as --name default --type dir --target /uyuni/pools/default
$ virsh pool-autostart default
$ virsh pool-start default
$ virsh pool-list

virsh-pool-list

However, when I run terraform apply (after init and validate), I still get this error message:

error-trraform-apply

I assume there's another way of defining the storage pool for libvirt and uyuni. Operating system: openSUSE Tumbleweed

cbosdo commented 8 months ago

@cbosdo, Seems like the link for creating the default storage pool is no longer valid.

Nice catch. I just updated this link and the one for the network.

I've tried to create a storage pool this way:

$ mkdir -p /uyuni/pools/default
$ virsh pool-define-as --name default --type dir --target /uyuni/pools/default
$ virsh pool-autostart default
$ virsh pool-start default
$ virsh pool-list

This looks good to me.

However, when I run terraform apply (after init and validate), I still get this error message:

error-trraform-apply

I wonder if that could be an issue with the libvirt connection. I am exporting LIBVIRT_DEFAULT_URI=qemu:///system in my shell configuration and added my user to libvirt and kvm groups. This way all the virsh commands I run with my user are targeting the system instance and not the user one.

You can verify this by running virsh -c qemu:///system pool-list. If the pool doesn't show up in that output then you only created it for your user libvirtd connection and you need to run the commands again with -c qemu:///system parameter.

waterflow80 commented 8 months ago

Thanks @cbosdo, that solved the error, I appreciate it.

But now I got another error, after running terraform apply again, I kept getting these logs:

module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Creating...
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [10s elapsed]
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [20s elapsed]
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [30s elapsed]
...

and then received this error message:

Error: Error while uploading source https://download.opensuse.org/distribution/leap/15.5/appliances/openSUSE-Leap-15.5-Minimal-VM.x86_64-Cloud.qcow2: Error while copying source to volume stream error: stream ID 5; PROTOCOL_ERROR; received from peer
     with module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"],
     on backend_modules/libvirt/base/main.tf line 57, in resource "libvirt_volume" "volumes":
     57: resource "libvirt_volume" "volumes" {

I appreciate your help.

cbosdo commented 8 months ago

But now I got another error, after running terraform apply again, I kept getting these logs:

module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Creating...
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [10s elapsed]
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [20s elapsed]
module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"]: Still creating... [30s elapsed]
...

These are indicating the image is being downloaded, so they are good.

and then received this error message:

Error: Error while uploading source https://download.opensuse.org/distribution/leap/15.5/appliances/openSUSE-Leap-15.5-Minimal-VM.x86_64-Cloud.qcow2: Error while copying source to volume stream error: stream ID 5; PROTOCOL_ERROR; received from peer
     with module.base.module.base_backend.libvirt_volume.volumes["opensuse155o"],
     on backend_modules/libvirt/base/main.tf line 57, in resource "libvirt_volume" "volumes":
     57: resource "libvirt_volume" "volumes" {

Could that be a permission issue on the pool folder? Here are the permissions I have on my pool:

  <target>
    <path>/public/vms</path>
    <permissions>
      <mode>0755</mode>
      <owner>1000</owner>
      <group>100</group>
    </permissions>
  </target>

Debugging this would probably be easier on gitter.

johnathankahn commented 8 months ago

Hey @agraul @cbosdo what will be the selection criteria ? what weightage will contributions hold ?

cbosdo commented 8 months ago

Hey @agraul @cbosdo what will be the selection criteria ? what weightage will contributions hold ?

This is more a gut feeling or rule of thumb than anything quantifiable. A solid and researched application is always the key. But the quality of the contributions and the communications surrounding them will be modifiers for us. No application is a no-go since it breaks rules, and we clearly prefer one nice contribution with good team work rather than a dozen of poor quality ones.