Integrate Oela Importer v2 with Oela Box

[maxbyte9p]

• Could be placed into a tools directory or stay a separate component designed to be used with Oela Box
• Should be ran from the host or from inside the Koji server?
• If designed as a separate component how would that work?
  • Would a plugin or API type system be suitable?
    • The way Vagrant handles plugins is interesting. Vagrant utilizes GRPC for communication which theoretically allows plugins to be written in any language
    • I could design a general API type system.
    • Not quite sure how any of this would work from the host system.
    • Could just mount a tools directory from the Oela Box directory to the virtual machines utilizing SSHFS
    • Any tools like Oela Importer would be easily accessible from inside of the virtual machines and could be ran in an SSH shell.

[maxbyte9p]

This is likely going to be very time consuming which is not a bad thing.

I think the best course of action could be to design tools like Oela Importer as separate components which can be added to a tools directory in the Oela Box root directory.

I do fear however that this design choice could make it harder for users to comprehend commonly used tools that are not part of stock Oela Box. I've seen how messy Fedora development can be due to having a bunch of tools that interact with Koji, but seemingly no clear indication what tools are in use or how they come together.

A naming scheme I have been using is prefixing the names of tools meant for Oela Box with "Oela". Although with enough tools available I can see this being hard to keep track of.

Something else I don't want is to make Oela Box too large and hard to work on effectively. I guess to take some notes from Vagrant's plugin implementation they include builtins with stock Vagrant which are required or commonly used.

[maxbyte9p]

I'm working on refactoring and redesigning Oela Importer. It may be of interest to make it a fairly simple tool which can be included in Bash scripts.

[maxbyte9p]

I've got a basic redesign going. It takes command line arguments and it does some filtering like I want it. I'm wondering whether or not to redesign how it filters. If given a list of branches it checks against the left most and if the left most exists then the filtering returns with True. So it can't check if a repository has all the branches in the list. The script just goes "Good enough for me" then returns a boolean.

[maxbyte9p]

Not sure whether or not if I want to go with the design of "acceptable branches" or "required branches"

[maxbyte9p]

I could maybe have the ability to do both. "acceptable branches" being sloppy and just having Oela Importer check if a repository has at least one of the branches then pull the first matching branch. "required branches" being precise and checking if a repository has all the branches listed.

[maxbyte9p]

Doing required branches is really easy to implement. Something along the lines of this works great.

set(acceptable_branches).issubset([i.name for i in repos[0].get_branches()])

[maxbyte9p]

I have now figured out how I want to handle branches. Make the user explicitly say which branch they want to import from. I may also add regex support in the future for this, so any branch matching the regex can be imported.

I've got the prototype working in the Koji vm and it successfully imported 10 OpenELA repositories to Koji.

[maxbyte9p]

I'm implementing regex support. So far it seems to be going smoothly and it opens up a lot of opportunities.

Prototype branch_match

def branch_match(b,pb):
 return re.match(pb, b.name)

Prototype has_branch

def has_branch(r, pb):
 return tuple(filter(lambda x: branch_match(x, pb), r.get_branches()))

Simple regex can be used to get specific branch types or a general range.

>>> has_branch(tmpr, 'el[0-9]-stream-\w')
(Branch(name="el8-stream-rhel"),)

>>> has_branch(tmpr, '^el8$')
(Branch(name="el8"),)

[maxbyte9p]

has_branch can be tweaked to make it easier to work with

return tuple(filter(lambda x: branch_match(x, pb), r.get_branches())) or None

[maxbyte9p]

Experimenting with a toggle for regex mode.

if regmod == 1 : (lambda : re.match('el8', tmpb.name) is not None)()

[maxbyte9p]

One-liner to switch between regmod and normal mode.

re.match('el8', tmpb.name) is not None if regmod == 1 else tmpb.name == 'el8'

[maxbyte9p]

regmod option implemented. Off by default.

[maxbyte9p]

Pretty output of regmod

=== Oela Importer ===========================================================================
--- Import ----------------------------------------------------------------------------------
  Repo: libguestfs
  Branch Matches: ['el8-stream-rhel', 'el8', 'el9', 'el-8.8-stream-rhel', 'el-8.8', 'el-8.9-stream-rhel', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
  Repo: cockpit
  Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
  Repo: abrt
  Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
  Repo: anaconda
  Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
  Repo: anaconda-user-help
  Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
  Repo: cloud-init
  Branch Matches: ['el8', 'el9', 'el-8.8', 'el-8.9', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
  Repo: crash
  Branch Matches: ['el8', 'el9', 'el-8.8', 'el-9.2']
--- Import ----------------------------------------------------------------------------------  Repo: dhcp
  Branch Matches: ['el8', 'el9', 'el-8.8', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
  Repo: dnf
  Branch Matches: ['el8', 'el9', 'el-8.8', 'el-8.9', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
  Repo: firefox
  Branch Matches: ['el8', 'el9']

[maxbyte9p]

Pretty output of regmod

=== Oela Importer ===========================================================================
--- Import ----------------------------------------------------------------------------------
  Repo: libguestfs
  Branch Matches: ['el8-stream-rhel', 'el8', 'el9', 'el-8.8-stream-rhel', 'el-8.8', 'el-8.9-stream-rhel', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
  Repo: cockpit
  Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
  Repo: abrt
  Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
  Repo: anaconda
  Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
  Repo: anaconda-user-help
  Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
  Repo: cloud-init
  Branch Matches: ['el8', 'el9', 'el-8.8', 'el-8.9', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
  Repo: crash
  Branch Matches: ['el8', 'el9', 'el-8.8', 'el-9.2']
--- Import ----------------------------------------------------------------------------------  Repo: dhcp
  Branch Matches: ['el8', 'el9', 'el-8.8', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
  Repo: dnf
  Branch Matches: ['el8', 'el9', 'el-8.8', 'el-8.9', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
  Repo: firefox
  Branch Matches: ['el8', 'el9']

Github messed it up a little bit oh well

[maxbyte9p]

I just tested it with the Oela Box koji server. I used reg mode and even made the pretty output show the koji task ids.

[maxbyte9p]

Working on some sort of machine readable output format. Looking toward Vagrant's way of doing it for inspiration.

[maxbyte9p]

A very interesting method of making import data machine readable.

json.dumps({ 'repo': tmp[0].name, 'imported': [ { 'branch_match': i[0].name, 'import_id': i[1] } for i in tmp[1] ]})

[maxbyte9p]

Machine mode has now been implemented. When enabled Oela Importer outputs import data as a JSON encoded line per import. In order to parse one would need to iterate every line.

[maxbyte9p]

Done some cleanup and testing of the code. Added a -t option for using a github token for API calls.

[maxbyte9p]

Implemented chunk feature. Oela Importer can be supplied with -c "1:2" to specify a chunk of repositories to import.

Also implemented -u flag so the oelakoji user doesn't need to be hardcoded.

[maxbyte9p]

I also somehow made Oela Importer even faster. No idea what I did to do that, but I'll take that as a win.

[maxbyte9p]

Did some more cleanup, made the help message look neater and pushed.

[maxbyte9p]

Oela Importer is now available on Oela Box.