Refactor for View Agnostic Controller

[bahostetterlewis]

I have started the process of the refactor and before I commit to this I want to update you as this is a fairly large change to how the program works. In the end the look and feel would be identical but the code base would be different. By different I mean that it would be rearranged but a large portion of the codebase would be what you have already written just moved to new locations. End result is separation of the views and the regex into modules, with identical functionality.

I have mapped the first version of what I think this will look like and created this diagram that shows what the changes will potentially look like and the modules that would be created as well as the classes that would be created.

@bittercode I know you have put a lot of time into this project so before I move forward with this I wanted to show you what it would look like. Let me know what you think and if this is something you would like see happen.

PS any change to the diagram will be immediately updated in the link so any changes we discuss will be reflected there.

Todo:

• [x] regChange
• [x] strChange
• [x] replaceChange
• [x] processRegex
• [x] findAllSpans
• [x] processEmbeddedFlags
• [x] Integrate new controller
• [x] Embedded flags

[bittercode]

I'm fine with it. My one request would be that you try and make sure that you get it so that I have at least some working pieces with pyqt so that if you aren't able to continue, I have examples to keep going. You know a lot more about programming than I do and if you have to stop I'll need to be able to emulate what you did and try to keep moving it forward. So I'll need something that shows me all the way to the end - even if it doesn't have all the functionality in place. That make sense?

[bahostetterlewis]

Yeah that definitely makes sense. My goal isn't to make something you can't work on at all. I am actually going to reuse (read: copy and paste) almost all the code you wrote but move them to different functions. The idea is that by taking the parts of code you wrote to deal with evaluating the regular expressions and strings into its own code you will be free to just call those functions.

Example: take findallSpans. What it will look like in the future is it will call the regex function that will hold your logic that you already wrote (maybe slightly tweaked to the new format) and that function will return a list of spans. Then you can display the spans the exact same way you already do in that function. But now when someone wants to create an alternate windowing system they just make their own findallspans that calls the regex function and then displays them however they want.

The code itself will change minimally but we will just move it to where it will be more reusable.

[PythonicChemist]

So what do you have left to do after today's major git push? I only scrolled over pree.py and tested the program and I haven't found any bugs. I had some time and already built most of the Tk gui. It is only missing scrollbars for textboxes and the about window right now. It is far from finished since it is written purely procedural, but it is only about 90 LoCs long right now.

[bahostetterlewis]

Sorry I havn't had a chance to email you yet, I can do that later today. As far as what I have left I think I just need to migrate the workspace into pree.py on my branch remove calls to the pree strings replace them with calls to the controller and then QA it. Then I will go through pree line by line again and just see if there is anything else to clean up. For example you can see how process regex was cleaned up in pree.py.

[bahostetterlewis]

Hey @bittercode and @GreenBikerDude . I think everything is working. I would love some QA on it. I have tested it myself but the more eyes the better. Once I get some feedback I will do a rebase so that I can clean up the history a bit and send in a PR. I think it reads a lot easier now. There are a ton of changes from moving the controller to cleaning up some pieces of code to be just a little more reader friendly (I wanted to write code that you could read as quickly as possible). Obviously there is always room for improvement and so I am open to anything. Anyway, you can find the branch here. I am excited to see what everyone thinks of this as it should make writing guis just a matter of calling functions to get the regex data we already use ready to be displayed.

[PythonicChemist]

I will read the branch tomorrow or maybe this evening. Just scrolled over it and could grasp the logic easily ,so it looks fine. I will test the code tomorrow or the day after tomorrow. Only question I have is why the controller got its own folder on the toplevel and not in the modules folder? Most projects have one modules/src folder where everything is contained except for the executing file. Personally I like it that way, but as always is open for discussion.

[bahostetterlewis]

The reason it has its own folder is I am using it as its own package. I did this because I want to clean up some things using some package specific things(basically set some stuff up in init for a cleaner interface). If it doesn't work like I want then I would move it into just a module.

[PythonicChemist]

I read the source code and it is free of gui code, which was the goal. I didn't test the program too much, but haven't encountered any bugs.

What I noticed though was that we need a is_valid_regex() checker. If you don't press pause the shell will be cluttered with errors, while you are entering your regex. Also performance wise it might be nice to have a timer option to check after X ms or so. I run this program on my netbook and the program freezes sometimes while processing the regex.

Long story short: The refactor fullfilled its job and that is what matters in this discussion. :)

[bahostetterlewis]

I made some changes on it so that it no longer throws garbage errors during compile. It now displays a debug message warning of an incomplete regex. The problem is based on there is no way to validate the regex without running it through some type of parser. I decided to just use the compile as that parser so basically it will attempt to compile, either error and print debug message, or accept. It should clean up those errors. I don't know what is causing the slowdown for you though because I didn't change how the regex works (it was always compiled before as well). Once we hear from @bittercode I can move forward with the PR I just don't want to change his codebase this much without him seeing it first :stuck_out_tongue:

[PythonicChemist]

I had try-except in mind as solution if re module doesn't provide other means of validating. This should make debugging easier in later iterations as the output isn't polluted as much anymore.

I think the slowdowns are coming from regex being cpu heavy afaik and my atom processor isn't powerful. Using Pause/Unpause works around my hardware limitations. I will test the try-except code piece, I heard that except errors are cheaper in python than compiler errors.

[bahostetterlewis]

The problem with the pause unpause solution was the entire program was setup to catch textchanges in the various text boxes. That means that if we were to say pause it and you changed one value while it was paused the entire system wouldn't update again until you hit another key. This wouldn't at all be clear to the user as to why nothing changed when one of their strings changed. Worse this would be a matter of luck to the end user who wouldn't know about this necessarily so sometimes a change will update the system at once and sometimes they would need to change it (potentially more than once) to get everything to update properly. To handle this we would need to create some function that would watch for changes and log that they happened then execute an update at certain intervals which could work but now we would have to deal with asynchronous issues. In other words during a change we would need to lock everything so that during a change a person couldn't be typing which again is possible but might hurt the end result if we have to lock the windows every so often.

Another possibility would be to keep a state changed value and the most recent change then every x amount of time compile and update the views but that can lead to outdated views potentially.

Anyway I am not 100% sure on how to remedy the compile issue and am just spitballing potential fixes and potential issues with those fixes. I think the second solution would be the easiest to implement. We might want to tinker around and see what the cost is and find any potential bugs because again we will basically need to be working with a threaded system so that one can be on a timer while everything else executes then every interval we update the system.

[PythonicChemist]

Another possibilty would be after x amount of time after the last input compile the regex. The waiting time x should be rather small just to throttle cpu usage. This reminds me kind of old games which would run at the fastest speed they could and when better hardware was released they were unplayable, because the game was running way too fast.

EDIT: I tested the new except-try code and the output is so much cleaner! I can also see in the console what was matched. This is quite nice.

[bahostetterlewis]

We just need to figure out the concurrency issues for a thread that does the waiting and compiling since it will be modifying state asynchronously.