Metal Integration Waypoint #2 (a?): (Revision Control) Frontend

[coreyeng]

Hello,

Way back when, I opened #136 as a way to quell one of my biggest concerns I had with using a compiled backend: the limited influence the frontend can have when key drivers originate from the backend. This is much less a worry for the core team, as we should generally have the ability to add whatever we need when we need it but, as I still dream of Origen catching on even more, this could be a larger concern. For example, what happens when someone would love to use Origen, especially the workspace and/or app management aspects, but use some other system, such as SVN, or some of the git adjacent systems (e.g., supports Git, but not really git, something like Perforce I think falls into this)? Those users would either need to open an issue and hope someone on the core team can pick it up, or learn Rust, the build process, add the systems then hope its reviewed and accepted, or IMO worse, fork the project, and just live off that fork.

To remedy this, I introduced the frontend. The idea here is that the Rust-backend can describe, via traits, what it wants to do and what it expects in return. How exactly that happens, it doesn't care and, in the case of the Dummy RC Driver that I provide here as an example, it doesn't even know that nothing actually happened.

Example Call

That's the general idea at least. To illustrate the implementation, I think it best to just walk through what a call would look like:

• A call made from Python, or from the CLI in Origen, will in some way trigger something in the backend. For example, I have this test function in Python which corresponds to this in the PyAPI.
• In this block the call rc.init() is actually called on this trait way back in the backend. This takes us from the frontend to the backend.
• Now begins the trip back to the frontend. The frontend implements this function in this trait.
• Assuming everything is in place, this call is of interest. This corresponds back to the dummy RC driver defined in Python.
• This call occurs, its return value processed and converted to what the backend trait requires, and the cycle is complete.

Frontend Initialization

When all is setup, that's pretty much how a call will look. Initializing isn't difficult and is handled here. This amounts to instantiating a struct which implements the general FrontendAPI and hands it off to the backend, which will store it as a static. This also takes care of instantiating the PyFrontend and sticking it at origen_metal._origen_metal.frontend.Frontend. This servers as the global state for now (see a bit further down).

Using The Frontend

In Rust, I have this function which will handle grabbing the frontend, checking if it exists and is initialized, and returning the trait, or raising an error. In PyAPI, I have this function to work directly with the PyFrontend. The test cases show some examples of what a user would currently need to do to set this up.

End-User Interface & Global State

In Origen, this is handled during the initialization, with the app also serving as a global state where this setup is available. In metal, we don't really have that quite yet. I can stick this on a module, as I did, but its not the most user friendly namespace. I can alias this to whatever, but I'm wondering if there's any intention on adding something like the config or some other global/current state. I could see a final interface looking something like:

# om.global is a class containing the current, global state
# this takes care of calling both 'frontend.initialize()', if needed, and setting the driver.
om.global.rc = RcClass()

A relevant SO question, for getattr overrides at the model level show its not that straightforward and a bit of a hack. I'd imagine the same would be true for setattr and I think some sort of global/current configuration or state could be beneficial.

Abstract Classes For Conformity

On the frontend side, an abstract class can be used to ensure that end users supplying their own frontend utilities are meeting the specification. I have an incomplete one for revision control, but I think it shows the idea. It also subclasses a base object from PyAPI that is currently empty, but can be built up as needed.

Outcomes

I introduced an Outcome struct to handle a generic return value. This is GenericResult in Origen but I really didn't like that name so I found something more or less synonymous instead. This servers as a generic way to pass a result/outcome of an operation, which may fail or error out, but shouldn't actually trigger an Err by itself. It may result in that at some point, but also make more sense to treat as a "failed success" (or a successful fail!) than an actual fail. It also can be built up later on (a bit more on this further down) as needed.

In Origen, I have multiple Results, which could be moved over here as well. In future, I plan to make a more generic Outcome Struct that we can use then essentially subclass off of that.

Current PR State And Next Items

The current state of this PR implements this scheme for a dummy RC driver that only deals with an init (as in, initialize the workspace, a-la git init) and nothing else. Everything else will panic with a todo, but this is just to show the basic scheme and this is simpler with just a single function for now.

I do intend to continue building on this. I'll submit more commits as I'm able, but I opened this as a legit PR as I'm fine with merging it if there's no objections. Waypoints 2b, 2c, etc. will continue building out revision control some more and move the existing stuff, including the git driver, out of Origen and into Metal. I'll also look at Origen using the Metal frontend and (not exactly sure how yet) splitting out the app-specifics from the bare-metal use cases.

Soon-To-Be-Renamed Metadata

Just as an aside, a problem does arise of passing arbitrary parameters around. But, this has also been solved through metadata in Origen and, in future, will be available here. However, this is an entirely different entity and could use some rework, generalization, peer review itself, and some rebranding. That'll probably be Waypoint 3 or 4.

__test__ Module

Getting a Python thread going from a Rust test where Rust functions would be available I think is impossible. Instead, I needed to expose some test functions through Python to call during pytest, but these shouldn't be usable in general. I conditioned these on the debug_assertions config setting, which I believe means this should be removed when --release is used. I stuck this at origen_metal._origenmetal.__test_\

[coreyeng]

Revision Control & Git

Quite a large update which moves the revision control stuff that already existed into origen_metal. Support for revision control through the frontend is also included, and is how I use it back in origen. I won't go into details here since its the same stuff that already existed. I did add some tests to just make sure the Git driver can be instantiated and such.

Note that two todo's were added (two new ones, that is) here and here - both of which related to driving Git with the current_user. Since the user stuff hasn't been brought over yet, this was removed for the time being.

Sharing PyAPI/PyAPI Metal

Although there's not a ton of code associated with this, a very big problem was encountered that coasted its way through the initial integration: PyO3 doesn't recognize the same class compiled from a different library - meaning that a class defined in PyAPI metal cannot be used in PyAPI. Attempts to do this leads to really wonky errors like "class PyFrontend cannot be extracted as a (...) PyFrontend" This slipped though as early tests and basic functions I was trying out were either returning PyO3-supported objects between Origen and Origen-Metal, or were returning a custom class defined in PyAPI or PyAPI-Metal, but were not crossing between the two.

Unfortunately, the revision control stuff exposed this and, more unfortunately, there really isn't a good, clean workaround. The cleanest solution I could come up with was just to compile _origen_metal into PyAPI and place it in _origen. Then, during Origen's, bootup, usurp the native _origen_metal and replace it. In origen_metal, this is handled very generically, so metal remains non-dependent on origen, and should any other things like origen pop up and want to add custom compiled code, they can reuse this same method.

This also means that origen_metal can still hold actual Python code. Although I think we're using the python-package more for documentation than anything, I do have the abstract base classes, for example, which I think makes more sense to be in Python-land than trying to hack something up from PyO3.

I hope to look into this more in the future, but I think this will hold together fine. The only caveat is origen must be imported before origen_metal for this replacement to occur. If you're running everything through origen, (e.g., origen i), it's fine. But, if you try to use a raw environment (like poetry run python), you need to take a bit of care. For us, this really only occurs during pytest. For example, if origen and origen_metal are swapped here, the test will fail.

Other

The only other thing I recall adding here is updating the poetry version when running the origen_metal tests to 1.1.9, to fix this issue. May need to further update in the future, but for now it seems to be holding together.

[info-rchitect]

@coreyeng How will origen_metal generally change origen usage at your place?

[coreyeng]

It doesn't directly. All origen metal is is more generic versions of what I already have. There's really no changes in how or what we'd use and I'm making it a point to not duplicate between the two (hence compiling PyAPI Metal into PyAPI to allow for code reuse there).

Really, the only impact I've had is the time spent on origen-metal stuff is time not-spent on other utilities, pattern API stuff, etc. But, a lot of what I'm moving to metal could really use some clean-up and review anyway, so its not necessarily a bad thing.