Testing and feedback needed for newer version

[transistor1]

Please test the newer version of Shellista, based on @briarfox's ShellistaExt. I'd like to push this version to master, and push the modules to briarfox's repo.

It currently downloads the entire plugins repo when the program starts, rather than individual modules. I'd like to get feedback on how we can download individual modules instead (GitHub repos can only be downloaded as either single files or as a full repo-- you can't download directories), and on how the update system should be implemented.

Feedback on how code can be improved is also welcome. @jsbain I tried to incorporate your changes to the Git module for password caching, but haven't really tested it.

[jsbain]

Few comments. The precmd plugin probably should be setting line to the output of the plugin function. Also, we might want to pass Self to the plugin, to give it access to the class, for instance to add local storage (I'd like to attach my bash history object to Shell in the plugin, so future calls have access.

Git pull and push code should look identical, except for the command name.
I.e all the password caching stuff, and origin caching.

[jsbain]

Re versioning/updates... One thought is that the modules could be individual repos. The main repo would have a file pointing to the other ones. Update would simply be git pull.

Another thought, re updates would be to use pypi.. A little more overhead to setup the pypi entry. But this would also allow third party devs to create shellista plugins.

[transistor1]

Thanks for the quick response and feedback!

The precmd plugin probably should be setting line to the output of the plugin function.

You are right - added that, also precmd/postcmd now need to take `self`` as the first argument.

Also, we might want to pass Self to the plugin, to give it access to the class, for instance to add local storage

I agree with this also but do not have time to make that change this morning as it affects all plugins. @briarfox, any reason why this would not be a good idea?

Git pull and push code should look identical, except for the command name.

Will look, probably later tonight

Re versioning/updates... One thought is that the modules could be individual repos. The main repo would have a file pointing to the other ones. Update would simply be git pull.

I thought of individual repos, but would that be too hard to maintain if they are all in different repos? Anyone know how Git submodules work, and/or if there is any support for them in GitHub?

I am definitely not against the idea of individual repos - in fact I like it a lot. The benefit is that each dev maintains their own plugins. I would definitely like some other people weighing in before we make that change, just in case there is some other problem with that, that we're not thinking of.

Another thought, re updates would be to use pypi. A little more overhead to setup the pypi entry. But this would also allow third party devs to create shellista plugins.

Using setuptools was actually my first thought, but it didn't work in Pythonista, which is why I wrote the ModuleInstaller class. Doesn't pypi use setuptools for packaging (I know that easy_installer used it, but I thought pip just built on top of that framework)?

[transistor1]

Edited my comment - sorry my 1-year old hit the comment button for me! :)

[briarfox]

@transistor It seems to be working well. I think passing self to each module would be the correct way to go about things. I'll see if I can squeeze some time in at work and start converting over the shellistaExt modules.

I think individual repos might be the best way to go about it. Keep all the core modules in a single repo and allow other people to maintain their repos for other extensions. I agree with keeping a file that stores extension urls. Seems like the easiest way to go about it. I'd say if you want to use the ShellistaExt repo for the core, then we should reorganize it. However it might be better to move all core files into thier own repo.

As far as current extensions I'm thinking that the ssh module may need to be pulled. I re wrote it but full ssh access is seeming to cause a little bit of trouble. Commands like nano or python cause the client to hang and no data to be displayed. I'm thinking ssh might best be left to a third party app. The only command that I feel is vital to shellista from ssh tools is scp. I'm just about done with an scp module that will make it easier to get/put files from/to a server.

[transistor1]

@briarfox - excellent! Thanks for the feedback.

I think individual repos might be the best way to go about it. 

That works for me. @jsbain had a very good point about doing updates with git pull. This could make things very easy. Should Git be part of the initial download so we can manage plugin installation/updates with Git clone/pull?

Any ideas on the best way to architect the extension file plugins/update system? Do we use ConfigParser, XML, etc... do we maintain version info of the plugins ourselves, or maybe have the project store something like a version.txt file?

 I'd say if you want to use the ShellistaExt repo for the core, then we should reorganize it. However it might be better to move all core files into thier own repo.

Im open to either way. The dev version is currently pointing to a clone of your repo , which hosts my plugins. I had tried to push the branch to ShellistaExt repo in Shellista but got a "private" error. Not sure if that is specific to Dulwich or some other issue.

I'm thinking that the ssh module may need to be pulled.

That is a shame but I can certainly understand why. I can imagine things might get complex. Scp will be a great addition though. I wonder how hard it would be to make an sftp plugin.

[transistor1]

@briarfox in case you hadn't gotten around to making the self change, it is done here. I don't know what your local Git looks like -- let me know if you'd like me to create a pull request.

I just ran this on the repo in Linux to replace the main() args

find plugins | grep -v 'local-modules' | while read i; do if [[ "$i" == *.py ]]; then cat $i | sed "s/def main(/def main(self, /g" > .tmp; mv .tmp "$i" ; fi; done

[jsbain]

Regarding git and single files.... I know we've moved on, but found this while fixing gistcheck. The github json api does allow single files.

See for example https://api.github.com/repos/jsbain/shellista/contents Then, if one were to follow url for the third filename, we get https://api.github.com/repos/jsbain/shellista/contents/shellista.py?ref=master Which has a content field, base64 encoded.

Doesn't pypi use setuptools

I'm definitely no expert, but I believe one can use distutils to package. Seuptools has some improvements, but I'm not exactly clear what the differences are. Not sure we'd want to, but in some ways pipista is a lot simpler than git, especially for the average usr who is not necessarily interested in a full git repo with history, etc, but just wants the latest version. The search capability is nice too.

[transistor1]

That is a good find -- but single files weren't the problem I meant (you can get to a single raw file on GitHub by going to the "raw" URL) What I meant was that you can't download a single directory from GitHub, which would make it hard to manage plugins from a single repo. Since we voted to use individual repos as plugins, I went ahead and coded up a quick and dirty implementation.

Latest dev-modular bootstraps (1) core modules, (2) Git, and (3) a "plugin plugin" which adds the command 'plugin' to search & install plugins via github. New repos can be added here. Anyone can edit it and it should be a simple merge.

In its current state it doesn't hot-plug plugins (you have to re-start Shellista to get it to recognize them) but I am working on it.

I have not been successful with pipista install, but maybe you have had a better experience. If you can setup an example for this we can certainly bootstrap pipista too.

[transistor1]

For new plugin repos - put your __init__.py and *_plugin.py in the root folder so the git clone works properly.

[briarfox]

@transistor1 I reworked the ssh plugin which includes scp. I added support for nano/edit to edit remote files. I still need to add a shortcut scheme for hosts but it's working. It still needs some reworking but I'd like some feedback if possible. I'll add it to the plugin list.

shellista-ssh https://github.com/briarfox/shellista-ssh

[transistor1]

Merged... did you have to fork the plugins repo to add the url? My hope was that devs could edit it directly using GitHub's built in text editor (click 'edit' in the upper right)

[transistor1]

Ssh plugin sounds great! Will definitely try it out when i get a chance.

[transistor1]

Can you also add your other non core modules like pipista?

[briarfox]

Yeah I hit edit then commit. It auto forked it. Sure I'll add pipista tonight. I'll also clean up and add better docs to ssh. I am still unsure on how to handle python and any return prompts, such as github login when using git pull. Btw amazing job on the plugin installer!

[briarfox]

Woops ssh key won't generate in shellista. I'll get it fixed.

[briarfox]

Added pipista and fixed ssh.

Could we use the github release mechanism for tracking module versions and updates?

[transistor1]

Thanks man! SSH looks awesome, but I can't seem to get it to connect. I tried connect user@host.com and it doesn't seem to respond. Should it prompt me for a password? I'm sure i'm doing something wrong.

The latest dev-modular plugins should now automatically hot-plug the commands when you do a plugin install xyz. I had to make a couple changes to your Shellista class, so the plugin class could use the code that adds the do_command function.

Git add/rm should now properly calculate relative paths properly. This was causing repos not to be pushed properly.

Could we use the github release mechanism for tracking module versions and updates?

How does it work? I don't have any experience with it. It sounds like something we should consider.

BTW - the core modules are currently being bootstrapped from here. They are entirely your code except with directory changes - please feel free to host them in your repo & change the link. In either case, I'm going to add you as a collaborator to that repo.

[briarfox]

@transistor1 Thats odd. From shellista do ssh. Then connect user@host.com. It will first try to use a key pair to connect, then prompt for a password. If you connect with will change the prompt to user@host.com>. I'm just about done with the shortcuts and ill have a look.

On github you can select a commit from a branch and mark it as a release with version number. It's a git tag. If you click on release on a project you will see what I mean.

[transistor1]

I am sure it's something I'm doing wrong. My guess is that the iPad is probably not resolving my internal host correctly for some reason. I will take another look

On Thursday, July 3, 2014, Chouser notifications@github.com wrote:

@transistor1 https://github.com/transistor1 Thats odd. From shellista do ssh. Then connect user@host.com javascript:_e(%7B%7D,'cvml','user@host.com');. It will first try to use a key pair to connect, then prompt for a password. If you connect with will change the prompt to user@host.com javascript:_e(%7B%7D,'cvml','user@host.com');>. I'm just about done with the shortcuts and ill have a look.

On github you can select a commit from a branch and mark it as a release with version number. It's a git tag. If you click on release on a project you will see what I mean.

— Reply to this email directly or view it on GitHub https://github.com/transistor1/shellista/issues/26#issuecomment-47980762 .

[transistor1]

Erm.... yeah -- just so you know, your SSH plugin doesn't work when the host is turned off! Can you fix that? ;-) This looks great man! Nice work!

On Thursday, July 3, 2014, Steve Russo svrusso1@gmail.com wrote:

I am sure it's something I'm doing wrong. My guess is that the iPad is probably not resolving my internal host correctly for some reason. I will take another look

On Thursday, July 3, 2014, Chouser <notifications@github.com javascript:_e(%7B%7D,'cvml','notifications@github.com');> wrote:

@transistor1 https://github.com/transistor1 Thats odd. From shellista do ssh. Then connect user@host.com. It will first try to use a key pair to connect, then prompt for a password. If you connect with will change the prompt to user@host.com>. I'm just about done with the shortcuts and ill have a look.

On github you can select a commit from a branch and mark it as a release with version number. It's a git tag. If you click on release on a project you will see what I mean.

— Reply to this email directly or view it on GitHub https://github.com/transistor1/shellista/issues/26#issuecomment-47980762 .

[transistor1]

Love the ability to edit remote files in Pythonista. That is awesome!

[transistor1]

On github you can select a commit from a branch and mark it as a release with version number. It's a git tag. If you click on release on a project you will see what I mean.

Updating by tags is a great idea. Do you think it would be enough to check if the latest tag on master is different than the latest local tag?

[briarfox]

@transistor1 I updated ssh with shortcuts. They are very handy. I added shortcuts for my login and put/get [scp] for my shellista projects, saves a ton of typing.

If the host doesnt respond it will eventually time out, I'll see if I can shorten it.

As far as tags that sounds like it would work if we are checking master. Do you think we could check github releases instead of master?

[briarfox]

@transistor1 I can't seem to use plugin install pipista. It creates the extension directory but thats it. after install pipista it returns Error:

I am also unable to use plugin update ssh same 'Error:'

[transistor1]

@briarfox update isn't implemented yet- it is throwing a NotImplementedError, but I'm not sure why it isn't displaying it, though. Implementing update will be the next steps. For now, to update you should just be able to delete the extension folder and re-install.

I could reproduce that error with pipista, but then I cloned a new copy of shellista in another folder, and then wiped out the initial installation and restarted Pythonista and it worked fine. There must have been some criteria leading up to that error, but it's not happening for me now. Can you try commenting out the try/except in plugin_plugin.py and see if you can get a stack trace?

[briarfox]

@Transistor1 I ended up reloading shellista and it works. When are you looking to push dev-modular?

[transistor1]

@briarfox I was going to send a message yesterday but never finished it. What do you think needs to be done before we merge to master? I havent had time to figure out how to get release tags working with gittle/dulwich. Right now, plugin update does a git pull from master.

Sent from my phone On Jul 11, 2014 2:44 PM, "Chouser" notifications@github.com wrote:

@Transistor1 https://github.com/Transistor1 I ended up reloading shellista and it works. When are you looking to push dev-modular?

— Reply to this email directly or view it on GitHub https://github.com/transistor1/shellista/issues/26#issuecomment-48767827 .

[briarfox]

@Transistor1 you are right we need to get the versioning system down before the merge. I've been busy with the family but I'll play with tags tonight and see if it'll work. We should display any updates needed when shellista is loaded. We also need a way to notify if there is a core module update.

[transistor1]

@briarfox Sounds good. You are right about the core module. Maybe the Git module could download first and then clone the core repo... You might have a better idea.

[briarfox]

Looks like we can use the api to get a list of releases. You can push git tags to github but you need to edit them for them to receive the Latest Release or pre-release tags.

[transistor1]

Core module now gets checked out with git after downloading so it is eligible for upgrade through git pull. You may want to wipe out your installation & reinstall to test this, I moved all the files in shellista-core one level up for this change.

[jsbain]

Where is the latest version to test now? Dev-modular? I'm getting an error in line 347, main takes exactly 2 arguments ( one given) when typing any comment except help.

[transistor1]

Yes - it's dev-modular. Please try copying just shellista.py (dev-modular version) into its own directory. Then restart pythonista to make sure the old version's modules arent still in memory, and then run shellista.py. I havent touched the code in a couple weeks for lack of time but I thought I left it in a working state.

Note that this version uses the "plugin" command to install other commands. You should be able to do "plugin list" (no quotes) to get a list of available plugins. "plugin install " lets you install a module

Sent from my phone On Jul 29, 2014 12:46 AM, "jsbain" notifications@github.com wrote:

Where is the latest version to test now? Dev-modular? I'm getting an error in line 347, main takes exactly 2 arguments ( one given) when typing any comment except help.

— Reply to this email directly or view it on GitHub https://github.com/transistor1/shellista/issues/26#issuecomment-50434109 .

[jsbain]

My bad, I had so many copies of shellista floating around, I was running an older verison.

Did the precmd_plugin stuff change recently? I recall testing this previously, but now that I'm actually trying to get this working! I'm getting errors in importlib that precmd_plugin is undefined. I used @precmd_plugin before a function inside a history_plugin.py that is getting imported

[jsbain]

Never mind... Previous comment covered how to get to precmd_plugin

[jsbain]

Did you figure out a good way to reload a module that has already been loaded ( when doing plugin development for example). I think I saw a comment in one of the git checking that you worked on a method? I have to rename the file, or restart pythonista... Surely there is a better way?

[transistor1]

I don't know of any better way to do it than to reload Pythonista, which, besides being very annoying, also forces the user to have to restart Pythonista when our shellista modules get upgraded, if they want to use the new functionality. This is one reason why I have been doing a lot of the development work in Linux. Plus PyCharm is pretty awesome.

If you can figure out a way to do it, that would be wonderful.

For what it's worth (in case you weren't already aware) if you hold down the "Clear" button in the interactive shell screen for a couple of seconds, there is a "Restart Pythonista" option.

[transistor1]

@jsbain I am also adding you as a contributor to the shellista-git repo. Any chance you could get the password caching working properly on that? Someone asked for it, and you're the expert on that.

I implemented dummy wrappers called ios_console.py and ios_keychain.py, which should get automatically used in place of console and keychain modules, when they aren't available on a platform, so hopefully that should automatically keep it working on Linux.

I had hoped to eventually create some sort of settings associated with shellista, to allow the user to turn password caching and other options on or off (some of us are paranoid), so there is a line up at the top, which is really just a placeholder. By default, it's true:

SAVE_PASSWORDS = shellista.Shellista.settings.get('save_passwords', True)

[transistor1]

... and, if you're feeling really ambitious, there seems to be a bug in the git module related to tags with messages, too, which I haven't had any time to look at.

[jsbain]

Fwiw, del sys.modules['plugins.extensions.bashhistory.hist_plugin'] for example removes a given module, allowing to be imported fresh next time. Likewise,

for mod in sys.modules.keys():
  if mod.startswith('plugins'):
    del sys.modules[mod]

Would delete all plugins, which should allow updating...maybe.

I found an issue with the pre_cmd plugin method... The decorator only gets called when the plugin is first imported. Thus, if I stop shellista, run again, the precmd plugin doesn't get registered.
Perhaps we should get rid of the decorator style, and do something similar to the alias stuff... I.e we'd have in the main hook function

for hook in lib.precmdhooks:
    self.precmd_plugin(hook)

I'll try this out.

I'll take a look at git password stuff. Do you want me to have it use keyring when not on ios, so that it'll work on linux/windows?

[transistor1]

Thank you - we definitely need to look into this for the updates. There is something that deletes the do_ commands in the plugin module when you do a plugin remove but how I did it escapes me at the moment. I had googled the topic at the time, and it seemed that there was no foolproof way of unloading modules, b/c they don't go away until there are no references to them & it can be challenging to find all references. In our case, I think some modules may be getting imported under multiple namespaces, which is confusing and probably not good.

The way Pythonista holds on to stuff even when it's no longer being used is definitely a challenge here. Looking forward to seeing what you have in regards to the precmd_plugin

Thanks for looking at the password stuff -- I had edited my post above but probably after GitHub sent you an e-mail. There are two "dummy" classes, in the core module -- in ios_console.py and ios_keyring.py which should keep things working in Shellista without you having to do anything special. You can just use the classes normally and it should "just work" on Linux.

[briarfox]

@transistor1 I've updated rm with a -r flag for directory deletion. I pushed it to ShellistaExt. Where are you storing the core modules?

[transistor1]

They are in shellista-core- you are a contributor in that project. But check your local copy & see if there is a .git folder in the top plugins folder. If your edits are in a fresh copy of shellista, I think (but pls double check) you can just do a git push -u briarfox from the plugins folder.

[transistor1]

If you chdir into plugins easiest way to check is to do git remote

[briarfox]

There should be a way to update plugins like dulwich. I had an older version and it took me a bit to figure out why git wasn't working. Deleted and re downloaded the current version from shellista.

[briarfox]

I added nano to shellista-core. I needed a way to edit the .git files. This creates/edits files.

[jsbain]

Passing this on, because it was useful to me while developing plugins. The following code prompts for a regexp, then removes the modules that match from sys.modules, thus allowing import to work on the updated file. For example, typing plug would match everything in plugins folder, thus would be useful after plugin update, or ext to remove all extensions.

import console, re

s=console.input_alert('enter regex for finding modules to delete (includes path)')

for m in sys.modules.keys():
    if re.findall(s,m):
        del sys.modules[m]

[jsbain]

As an aside... Is it time to push dev modular to master?

[cclauss]

I think someone would need to update Readme.md to reflect all that you have added.