Modularize Shellista

[transistor1]

In my mind, the goal for the short term is still to modularize Shellista, according to briarfox's model. The codebase is growing too much to maintain everything in one module, and this is the way that shells traditionally operate (i.e. "spawning" processes from the main process -- though I am aware that this won't really "spawn" anything).

Here are my thoughts on roadmap; please contribute suggestions/comments/profanities as you see fit:

• I want to keep Shellista as a one-file copy and paste, in the spirit of the original. This makes it easy for newbies to get started. Therefore, the code for module download should be in the main file.
• Core modules (ls, mkdir, cd, etc...) should be downloaded upon first run
• For non-core modules (e.g. Git), I'd like to have a way for the user to pick and choose what modules s/he is interested in using - not as critical in the short term, because we don't have that many modules yet.
  • I wonder if it's possible to use submodules in the Git repo to make core vs optional downloads easier. Core modules could be stored in one submodule, and then each optional module would have it's own submodule. Not sure if GitHub supports this, also might require specific additions to the Git module for committing from iPad
• I would prefer to keep things Python-agnostic, if possible... for individual modules, it's probably not as critical.
  • Maybe modules should adhere to an interface, which has some sort of check_dependencies() function

[transistor1]

Used briarfox's ShellistaExt and a new Module installer class to make a proof of concept thing. @jsbain, @briarfox I would like to get your input and suggestions & ideas on how/if to proceed

My idea was to bootstrap it with the existing code & wean off of the old code base as the modules all are transitioned. Briarfox has already written a bunch (most?) which could be used.

Hoping also to re-use the ModuleInstaller class for dependency downloads too.

[briarfox]

@transistor1 Looks like a good roadmap. I like the idea to keep shellista a single file and download all other core modules on run.

We could add non-core modules to a text file in the repo which has the corresponding github to the project and version number. Then shellista can display available modules in the ui and allow the user to select which modules to download/update.

Better yet, keep all non-core module info in an ini file, check it on run.

[transistor1]

@briarfox - That is a great idea. I like the idea of the user being able to have a list of downloadable modules, and potentially any updates.

Would you be willing to use your ShellistaExt.py file, with the addition of the ModuleInstaller class, and your existing modules? It seems like the best path is to re-use the code you have already written, and I can rewrite my contributions as separate modules (I thought you said you did something with the Git functions, so if you have put a lot of work into that I do not want it to be wasted, though)

My proposal, if you are willing, would be to open a separate development branch in this repo, whose shellista.py file would be a combination of your ShellistaExt file named as "shellista.py", and my ModuleInstaller class. Essentially, it would be this, but I would get rid of my hackish addition to your Shellista class, particulary the introduction of the _import_legacy_shellista(self, modname) function, so basically your part of the code would be unchanged, except for checking for the existence of modules.

Once all the commands were implemented as modules, we could merge development to master and "cut over" to the new system. I imagine it would not take long since most of the work is done.

What do you think about keeping the modules themselves in a separate repo? It seems like it would be nice to have a bootstrap repo, and a separate module repo. I have tested the download using your ShellistaExt repo, and it worked well. I was thinking that we could re-purpose yours to store the module code in?

[briarfox]

@transistor1 That sounds like a great idea. Feel free to take any part of ShellistaEXT that makes it easier. The loading method of modules felt pretty solid to me. Very easy to expand. I'll help out as I can, it's been a crazy few weeks at work. I never finished adding a way for the modules to declare thier dependencies for use in your download module. The git module isn't much more then the git you have in shellista, I just wrapped it in the template I used for shellistaExt. Work should be slowing down in the next day or to and I'll help out.

[transistor1]

@briarfox - awesome! Thank you.

I will also need your help figuring out what the best way is for using the iPad specific modules when needed without breaking shellista everywhere. That's what I meant by Python agnostic, but that was probably the wrong term.

I want us to be able to take advantage of iPad specific modules if they seem appropriate, but not break Shellista everywhere else. Or should this not even be a goal of the project?

Technically, we could check everywhere they are used, but that feels wrong. Do we just write wrappers as we use each module?

[jsbain]

Is there a new fork that you guys are working in? I've written a bash history module (!!, !$, !*, !N, !-N, and a history command that either pops up a window or prints to stdout depending in if ui is present. It made most sense to have this live as an object in Shell, as it intercepts precmd to do history expansion, and uses postcmd to add to the history log.
Either this would need to get incorporated into the core Shell, or we need a plugin method or decorator or something to allow hooking into precmd and postcmd.

[transistor1]

I have been working in a new local branch, called dev-modular which I just pushed now so you can see - link). At the same time, I'm working in a new local branch of ShellistaExt, which will be the new modules repo. I am basically trying to get my Git module to work there, and re-use the ModuleInstaller class instead of separate download code.

I have been trying not to touch @briarfox 's Shellista class, so I think the only thing that should change in the file is the other stuff (ModuleInstaller class especially). It sounds like @briarfox would need to review the changes. Do you want to create a pull request?

[transistor1]

Note: if you are going to test out the code in the branch above, make sure you set __DEBUG__ to False.

[transistor1]

Fixed the link.

[jsbain]

Ok, I see where things are going. If I can offer a suggestion, can we consider creating @precmd_plugin, and @postcmd_plugin in Shellista class that allow other modules to be called within pre or post cmd. . I.e something along the lines of this.
That way, modules can register themselves to call in precmd or postcmd. Maybe others, but not sure what that might be. Admittedly, the only use case I see is for things which do expansion, like tab completion, or history expansion and recording, which maybe should be a part of the core Shell. At a minimum, we could just have an expansion object and history recording object, whose implementation is in a separate file.

[jsbain]

As an aside, do we need to consider a versioning and update system for plugins? I.e. A way to download newer versions of plugins. So should consider a version entry in the plugin template, even if we don't implement something right away.

[jsbain]

FYI, the bash history work was pushed here

In reviewing how bash history in general I realized the argument identifier is supposed to come after a colon, so I need to go back and make some changes to fix bashhistory.py, (not really ready to be pulled) but this gives a flavor of what needs to be added to Shell. What I have there works, but uses nonstandard syntax that I'd like to fix.

[transistor1]

@jsbain added decorators here.

To access them in a plugin, you can do:

shellista = sys.modules['__main__']

then call

@shellista.Shellista.precmd_plugin
def my_precmd_plugin(line):
  pass

I tested precmd and it seems to work fine; haven't tested postcmd.

The weird quirk about this is that you still have to have a main() method in your plugin, so Shellista adds a command when one isn't needed. That needs to be fixed.

[transistor1]

do we need to consider a versioning and update system for plugins Yes, absolutely. Just trying to get the base stuff working first though. import issues are making my head spin.

We should brainstorm on how a versioning/update system might work, maybe create a new issue for it.

One immediate problem I see with downloads from GitHub in general, is that there is no way to only download a single folder from GitHub. You have to pull down the entire repo or a single file. Plugins need to be able to have several files if needed. Not sure how to resolve this.

[transistor1]

FYI I have my current changes to the module repo here but shellista.py isn't set up to pull that directly.

Git has a number of enhancements that are rough but more or less working; namely pull, status, checkout -b. Also I have changed to dulwich's porcelain.py wherever I can, because Gittle doesn't get much attention, and dulwich's porcelain works better for some things. Porcelain's clone doesn't work as well as Gittle's, though.

[khilnani]

Is it possible to augment the plugin install to take in a url for a module if its not officially supported or in the modules github? eg. during development

[briarfox]

@khilnani for development purposes you can drop your module into shellista/plugins/extensions. However a plugin add url is a good idea.

[transistor1]

Once you have bootstrapped your initial shellista install you can also modify your local copy of plugin_urls.txt to pull in plugins from whatever source you want, for development.

Sent from my phone On Aug 19, 2014 9:43 AM, "Chouser" notifications@github.com wrote:

@khilnani https://github.com/khilnani for development purposes you can drop your module into shellista/plugins/extensions. However a plugin add url is a good idea.

— Reply to this email directly or view it on GitHub https://github.com/transistor1/shellista/issues/23#issuecomment-52634961 .

[khilnani]

that's essentially what I did, I created my own repos since I wanted to test my plugin with a clean install as well.I have a fork of the plugin repo and will look into adding the url. Install seems easy enough but not sure if update relies in plugins.txt off hand. Ill take a look

[khilnani]

Great job tho, this makes pythonista much more useable.