Closed maxbyte9p closed 5 months ago
This is likely going to be very time consuming which is not a bad thing.
I think the best course of action could be to design tools like Oela Importer as separate components which can be added to a tools directory in the Oela Box root directory.
I do fear however that this design choice could make it harder for users to comprehend commonly used tools that are not part of stock Oela Box. I've seen how messy Fedora development can be due to having a bunch of tools that interact with Koji, but seemingly no clear indication what tools are in use or how they come together.
A naming scheme I have been using is prefixing the names of tools meant for Oela Box with "Oela". Although with enough tools available I can see this being hard to keep track of.
Something else I don't want is to make Oela Box too large and hard to work on effectively. I guess to take some notes from Vagrant's plugin implementation they include builtins with stock Vagrant which are required or commonly used.
I'm working on refactoring and redesigning Oela Importer. It may be of interest to make it a fairly simple tool which can be included in Bash scripts.
I've got a basic redesign going. It takes command line arguments and it does some filtering like I want it. I'm wondering whether or not to redesign how it filters. If given a list of branches it checks against the left most and if the left most exists then the filtering returns with True. So it can't check if a repository has all the branches in the list. The script just goes "Good enough for me" then returns a boolean.
Not sure whether or not if I want to go with the design of "acceptable branches" or "required branches"
I could maybe have the ability to do both. "acceptable branches" being sloppy and just having Oela Importer check if a repository has at least one of the branches then pull the first matching branch. "required branches" being precise and checking if a repository has all the branches listed.
Doing required branches is really easy to implement. Something along the lines of this works great.
set(acceptable_branches).issubset([i.name for i in repos[0].get_branches()])
I have now figured out how I want to handle branches. Make the user explicitly say which branch they want to import from. I may also add regex support in the future for this, so any branch matching the regex can be imported.
I've got the prototype working in the Koji vm and it successfully imported 10 OpenELA repositories to Koji.
I'm implementing regex support. So far it seems to be going smoothly and it opens up a lot of opportunities.
Prototype branch_match
def branch_match(b,pb):
return re.match(pb, b.name)
Prototype has_branch
def has_branch(r, pb):
return tuple(filter(lambda x: branch_match(x, pb), r.get_branches()))
Simple regex can be used to get specific branch types or a general range.
>>> has_branch(tmpr, 'el[0-9]-stream-\w')
(Branch(name="el8-stream-rhel"),)
>>> has_branch(tmpr, '^el8$')
(Branch(name="el8"),)
has_branch can be tweaked to make it easier to work with
return tuple(filter(lambda x: branch_match(x, pb), r.get_branches())) or None
Experimenting with a toggle for regex mode.
if regmod == 1 : (lambda : re.match('el8', tmpb.name) is not None)()
One-liner to switch between regmod and normal mode.
re.match('el8', tmpb.name) is not None if regmod == 1 else tmpb.name == 'el8'
regmod option implemented. Off by default.
Pretty output of regmod
=== Oela Importer ===========================================================================
--- Import ----------------------------------------------------------------------------------
Repo: libguestfs
Branch Matches: ['el8-stream-rhel', 'el8', 'el9', 'el-8.8-stream-rhel', 'el-8.8', 'el-8.9-stream-rhel', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
Repo: cockpit
Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
Repo: abrt
Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
Repo: anaconda
Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
Repo: anaconda-user-help
Branch Matches: ['el8', 'el9']
--- Import ----------------------------------------------------------------------------------
Repo: cloud-init
Branch Matches: ['el8', 'el9', 'el-8.8', 'el-8.9', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
Repo: crash
Branch Matches: ['el8', 'el9', 'el-8.8', 'el-9.2']
--- Import ---------------------------------------------------------------------------------- Repo: dhcp
Branch Matches: ['el8', 'el9', 'el-8.8', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
Repo: dnf
Branch Matches: ['el8', 'el9', 'el-8.8', 'el-8.9', 'el-9.2', 'el-9.3']
--- Import ----------------------------------------------------------------------------------
Repo: firefox
Branch Matches: ['el8', 'el9']
Pretty output of regmod
=== Oela Importer =========================================================================== --- Import ---------------------------------------------------------------------------------- Repo: libguestfs Branch Matches: ['el8-stream-rhel', 'el8', 'el9', 'el-8.8-stream-rhel', 'el-8.8', 'el-8.9-stream-rhel', 'el-9.2', 'el-9.3'] --- Import ---------------------------------------------------------------------------------- Repo: cockpit Branch Matches: ['el8', 'el9'] --- Import ---------------------------------------------------------------------------------- Repo: abrt Branch Matches: ['el8', 'el9'] --- Import ---------------------------------------------------------------------------------- Repo: anaconda Branch Matches: ['el8', 'el9'] --- Import ---------------------------------------------------------------------------------- Repo: anaconda-user-help Branch Matches: ['el8', 'el9'] --- Import ---------------------------------------------------------------------------------- Repo: cloud-init Branch Matches: ['el8', 'el9', 'el-8.8', 'el-8.9', 'el-9.2', 'el-9.3'] --- Import ---------------------------------------------------------------------------------- Repo: crash Branch Matches: ['el8', 'el9', 'el-8.8', 'el-9.2'] --- Import ---------------------------------------------------------------------------------- Repo: dhcp Branch Matches: ['el8', 'el9', 'el-8.8', 'el-9.2', 'el-9.3'] --- Import ---------------------------------------------------------------------------------- Repo: dnf Branch Matches: ['el8', 'el9', 'el-8.8', 'el-8.9', 'el-9.2', 'el-9.3'] --- Import ---------------------------------------------------------------------------------- Repo: firefox Branch Matches: ['el8', 'el9']
Github messed it up a little bit oh well
I just tested it with the Oela Box koji server. I used reg mode and even made the pretty output show the koji task ids.
Working on some sort of machine readable output format. Looking toward Vagrant's way of doing it for inspiration.
A very interesting method of making import data machine readable.
json.dumps({ 'repo': tmp[0].name, 'imported': [ { 'branch_match': i[0].name, 'import_id': i[1] } for i in tmp[1] ]})
Machine mode has now been implemented. When enabled Oela Importer outputs import data as a JSON encoded line per import. In order to parse one would need to iterate every line.
Done some cleanup and testing of the code. Added a -t option for using a github token for API calls.
Implemented chunk feature. Oela Importer can be supplied with -c "1:2" to specify a chunk of repositories to import.
Also implemented -u flag so the oelakoji user doesn't need to be hardcoded.
I also somehow made Oela Importer even faster. No idea what I did to do that, but I'll take that as a win.
Did some more cleanup, made the help message look neater and pushed.
Oela Importer is now available on Oela Box.