Soooo… yet another massive PR. Fortunately (-ish) the big addition here is fairly standalone and shouldn't affect much that isn't using it explicitly, but this was another time where I started with a generic goal in mind, one thing led to another, and now this PR is all over the place. I'll break it down as best I can.
I started with just trying to get application-level releasing together. The first thing I went for was updates to the Git driver to handle check in and push operations., mostly just for automatic checkins of releasing-related collateral, such as version.rb, like O1 does.
Frontend <-> Backend Cooperation
I ran into a bit a of problem though, but ultimately able to fix it by messing with the git config and changing the tokens on the Windows side. However, I also went down a path I've been interested in pursuing for a while now - one that puts a lot of my "compiled library" concerns to bed: callable frontend stuff from the backend.
I've done a few similar things but hadn't really brought it all to fruition until now. This PR adds the FRONTEND static, which contains a trait (and traits underneath it) which the actual frontend implements. With this trait, the backend can call and utilize methods written in the frontend. After defining such a function in either the frontend trait, or in one of the sub-traits, this creates a full round-trip, starting from the backend, going up through the frontend, likely calling something defined in the frontend itself, then return back to the backend. The links in the last sentence shows the backend asking the frontend to go emit a callback (covered later). Another example asks the frontend application's unit_tester, pytest in this case, to run and handles the results. The return value here is really basic, but can be expanded later as needed.
Sometimes though, this may lead to a bit of overkill. For example, since Git doesn't really do anything extra, its just a superfluous round trip. But, it could do something! Or, as I more envision it, can be inherited by something in the application to expand or workaround company internals (like our bitbucket annoyance with not allowing anonymous accesses), without requiring Rust knowledge or backend releases.
For a future item, base classes utilizing Python's abstract base class and typing could allow for an interface, ensuring anything implemented on the frontend side by end users can successfully be used in this scheme.
I truly believe that this will go a long way to quelling any issues with using a compiled backend, especially if others outside of just the core team here opt to pick it up. Now, we can have compiled, safe, and efficient backend code in Rust for general cases, but allow for user specific cases to override as needed in the frontend. They may take a bit more of a performance hit, but likely beats by a mile needing to learn Rust and go through the releasing process, or opening tickets and hoping one of us picks it up in a timely manner. Anyway, that's my sell on this scheme. In principle, this isn't much different than what I did with frontend testers, but this is much more generic and with a wider use case. If this fine though, I'll rewrite the tester stuff to use this scheme as well.
Callbacks
Next up, I also added callbacks, but I'm doing callbacks somewhat differently this time around. At least twice a year, I run into a situation where some block somewhere is adding a callback, usually startup, that breaks other stuff in certain instances. O1 doesn't allow for any kind of tracking, re-ordering, or bypassing individual callbacks and that's one thing I wanted to make sure O2 has.
I also wanted to have more explicit callbacks. That is, if you provide a method, you know its got the purpose of being a callback and aren't surprised when it behaves as such. Python offers decorators that fills both these requirements. End users will explicitly mark methods or functions with the callback's decorator and the decorator allows me to cache and track a full list of callbacks. Unfortunately, this did get somewhat messy on the implementation side - primarily because of how Python handles methods for classes where the class hasn't actually been instantiated yet, but I think I've got a solution to get this to work as we need.
So far, I only have a few callbacks. You can see here for the test scripts, and here, here, here and here for more setup.
The actual user methods to query, re-order, disable/enable individual callbacks, etc. aren't there yet, but the structures should be in place to start adding this.
Returning to the frontend trait stuff a bit, callbacks can also registered and emitted from the backend. I only have one so far, but see here for that.
I have a laundry list of other things I'm going to build off of this, but I think this is a good snapshot of the current progress and I don't foresee any large scale changes to the implementation incoming - hopefully will just be adding additional support or various helper methods.
Small Stuff
A footnote to this PR: when I merged everything it exposed a few bugs I added quick fixes for. First, the targeted tester processor needs to be run, so I added that, and second, the syntax dut.reg("x").bits["y"] wasn't yielding a full bitfield, which it should. I also expanded on converting from a PyErr, which is useful when an error originates in Python (e.g., during a callback or frontend method) but needs to undergo a trip though the backend before being presented to the user.
Soooo… yet another massive PR. Fortunately (-ish) the big addition here is fairly standalone and shouldn't affect much that isn't using it explicitly, but this was another time where I started with a generic goal in mind, one thing led to another, and now this PR is all over the place. I'll break it down as best I can.
I started with just trying to get application-level releasing together. The first thing I went for was updates to the Git driver to handle check in and push operations., mostly just for automatic checkins of releasing-related collateral, such as
version.rb, like O1 does.Frontend <-> Backend Cooperation
I ran into a bit a of problem though, but ultimately able to fix it by messing with the git config and changing the tokens on the
Windowsside. However, I also went down a path I've been interested in pursuing for a while now - one that puts a lot of my "compiled library" concerns to bed: callable frontend stuff from the backend.I've done a few similar things but hadn't really brought it all to fruition until now. This PR adds the FRONTEND static, which contains a trait (and traits underneath it) which the actual frontend implements. With this trait, the backend can call and utilize methods written in the frontend. After defining such a function in either the
frontend trait, or in one of the sub-traits, this creates a full round-trip, starting from the backend, going up through the frontend, likely calling something defined in the frontend itself, then return back to the backend. The links in the last sentence shows the backend asking the frontend to go emit a callback (covered later). Another example asks the frontend application's unit_tester, pytest in this case, to run and handles the results. The return value here is really basic, but can be expanded later as needed.Sometimes though, this may lead to a bit of overkill. For example, since Git doesn't really do anything extra, its just a superfluous round trip. But, it could do something! Or, as I more envision it, can be inherited by something in the application to expand or workaround company internals (like our bitbucket annoyance with not allowing anonymous accesses), without requiring Rust knowledge or backend releases.
For a future item, base classes utilizing Python's abstract base class and typing could allow for an interface, ensuring anything implemented on the frontend side by end users can successfully be used in this scheme.
I truly believe that this will go a long way to quelling any issues with using a compiled backend, especially if others outside of just the core team here opt to pick it up. Now, we can have compiled, safe, and efficient backend code in Rust for general cases, but allow for user specific cases to override as needed in the frontend. They may take a bit more of a performance hit, but likely beats by a mile needing to learn Rust and go through the releasing process, or opening tickets and hoping one of us picks it up in a timely manner. Anyway, that's my sell on this scheme. In principle, this isn't much different than what I did with
frontend testers, but this is much more generic and with a wider use case. If this fine though, I'll rewrite the tester stuff to use this scheme as well.Callbacks
Next up, I also added
callbacks, but I'm doing callbacks somewhat differently this time around. At least twice a year, I run into a situation where some block somewhere is adding a callback, usually startup, that breaks other stuff in certain instances. O1 doesn't allow for any kind of tracking, re-ordering, or bypassing individual callbacks and that's one thing I wanted to make sure O2 has.I also wanted to have more explicit callbacks. That is, if you provide a method, you know its got the purpose of being a callback and aren't surprised when it behaves as such. Python offers decorators that fills both these requirements. End users will explicitly mark methods or functions with the callback's decorator and the decorator allows me to cache and track a full list of callbacks. Unfortunately, this did get somewhat messy on the implementation side - primarily because of how Python handles methods for classes where the class hasn't actually been instantiated yet, but I think I've got a solution to get this to work as we need.
So far, I only have a few callbacks. You can see here for the test scripts, and here, here, here and here for more setup.
The actual user methods to query, re-order, disable/enable individual callbacks, etc. aren't there yet, but the structures should be in place to start adding this.
Returning to the
frontend traitstuff a bit, callbacks can also registered and emitted from the backend. I only have one so far, but see here for that.I have a laundry list of other things I'm going to build off of this, but I think this is a good snapshot of the current progress and I don't foresee any large scale changes to the implementation incoming - hopefully will just be adding additional support or various helper methods.
Small Stuff
A footnote to this PR: when I merged everything it exposed a few bugs I added quick fixes for. First, the targeted tester processor needs to be run, so I added that, and second, the syntax
dut.reg("x").bits["y"]wasn't yielding a full bitfield, which it should. I also expanded on converting from a PyErr, which is useful when an error originates in Python (e.g., during a callback or frontend method) but needs to undergo a trip though the backend before being presented to the user.