CLI

[yfeldblum]

Fog could provide a CLI for simple operations. Example use:

$ fog rackspace servers list --format json
$ fog rackspace servers create --name "test.corp.com" --flavor-id 2 --image-id 69
$ fog rackspace servers show --name "test.corp.com" --format json
$ fog rackspace servers ssh --name "test.corp.com" --user root

The CLI would provide functionality covering a subset of the API.

[geemus]

I've heard other people mention this as well, but I at least greatly prefer the existing fog interactive mode, in most cases there is enough context involved that I would rather have ruby as a dsl than having to pass lots of command line arguments. What use case do you have in mind for the CLI?

[yfeldblum]

The use case is using multiple tools, one after the other, to perform a complex task. The shell is where that is done. Having to go into fog's interactive mode is great when one needs to do multiple fog steps in a row, but not as useful when one needs to do one thing with fog, then another thing with ssh, then another thing with knife, then another thing with vagrant, then another thing with git. One might need to do that kind of a workflow manually in the shell, or one might need to write a shell script that does that kind of a workflow.

Of course the reason Fog exists is for its Ruby API. But a CLI layered on top of that, generated at runtime from introspection on the models, would help with this alternative type of use case, one which makes use of multiple tools and only does one thing with each of them.

[geemus]

My feeling about that sort of workflow has tended to be that scripted is probably better. In the same way that I wouldn't recommend to someone that they should manually configure their servers each time they boot them, I think it makes good sense to say that provisioning workflow should be code that is versioned and stored off somewhere. I think ultimately its hard to do this sort of thing well for the general case, but I've seen some pretty compelling use cases where people have created a kind of hybrid that worked really well. In particular they built up a set of rake tasks that encapsulated a lot of their specifics (avoiding the large number of specific command line arguments) but still maintaining the ability to use it in the sort of workflow you describe. That seems like a more compelling flow in my mind, and one I would love to better facilitate. I just worry that maintaining a general purpose CLI like this would be a disservice as it can get hairy quickly and is hard to repeat.

[yfeldblum]

For repeatable work - versioned scripts are better.

For experimentation - versioned scripts get in the way. If I want to poke around and see what works (including for the purpose of writing versioned scripts) then poking around, then going into fog interactive mode and poking around some more, and then quitting from fog interactive mode to poke around at something else - that is a form of context-switching which, while fog makes such learning possible in the first place, sometimes slows down the learning. In my own case, I have been opening a new console tab and leaving fog interactive mode open in there.

As an analogy, when I know what my code is supposed to do, I might start off with a test case demonstrating that my code in fact does that, then write the code and check it against the test case. But if I'm attempting to learn a technique, API, or language, or if I'm attempting to debug a method, or if I have no idea what my code is supposed to do but I want to experiment, then I might skip the test case altogether.

For Fog as a tool of execution, doing it scripted is doing it right. But for Fog as a tool of learning (specifically, of how to do achieve higher-level goals, making use of multiple tools including Fog), doing it scripted is awkward (I don't know what the script is supposed to, since so far I can only envisage the first step; I can envisage envisaging the second step but only once the first step is completed; and so on).

Perhaps it's better to have a Fog CLI as a separate project, and to leave Fog as a Ruby API. But a common CLI for interacting with multiple clouds would be rather useful.

[geemus]

First off, just to be sure (text doesn't do tone well), I'm not trying to shoot this down outright. I just don't have the same workflow you do so I'm trying to better understand it so I can either suggest stuff or implement stuff. This may have already been clear but I didn't want to come off as the grumpy old man telling you to get off my lawn or something.

I agree that versioned scripts are a pain when experimenting. For experimenting I almost always just work inside a console though, so I'm wondering if a lack of good docs on how to do that kind of stuff is the problem more so than the lack of CLI. For instance, I usually use #bootstrap to place ssh keys for me and then #ssh to actually run stuff and experiment. That usually gives me everything that I need to figure out how to actually write a versioned script. If there are just one or two other things that are common we could maybe add them to fog (to make experimentation easier, but also to make eventually writing the scripts easier too).

Ultimately I just worry that a CLI may be a lot of work that doesn't gets used that much and/or gets used for the wrong purpose (we both understand that at some point you should move to versioned scripts, but newer people might assume that they would just always use the CLI). Again, this may well be a documentation/education issue more than a technical one, but right now no one is likely to be mistaken into thinking that CLI is the long term solution.

At the end of the day I want to make experimentation in fog super easy and helpful, so I'm glad we are discussing what that will take. For me it already is easy and helpful in interactive mode, but I have already spent an unhealthy amount of time in the code base ;). So really I think I'd like to see us tackle this in two phases. 1) identify what you can't easily do from inside fog and consider incorporating it 2) consider how to bridge the gap to things that can't be easily integrated (CLI could be a good fit here). Also, as to whether it should be an internal or external project, I'm not really sure. We can cross that bridge when we get to it, but I imagine some stuff will need to happen in fog to help facilitate this either way.

Thanks again for all the feedback!

[yfeldblum]

You come off as friendly and interested in my scenario!

Another use case for a CLI, in addition to experimentation, is a shell script which wants to use Fog to control a cloud resource, as a part of what the script does, in addition to many other shell-scripty things. Currently we would need 1) to "shell out" to Ruby from the shell script (ruby -e "#{generate_fog_code}") or 2) to write a wrapper CLI script in Ruby alongside the shell script and use that wrapper from the shell script or 3) to write a Ruby script instead of a shell script, and to shell out to the shell from the Ruby script when not using Fog. The solution you gave in your penultimate comment is a variant of 2) above, possibly mixed with 3).

I'll have to look at the #bootstrap and #ssh methods. I didn't know about them. They may help a little bit for my use case, but I don't think solve it, because these methods are still geared towards Fog as being used from Ruby scripts or from the Fog interactive mode. My use case is using Fog as part of a larger overall shell session for experimentation or even ad-hoc systems administration, or as part of a larger overall shell script, rather than as part of an IRB session or a Ruby script.

I am comparing this in my mind with Knife/Shef. I often prefer, when at the shell, to knife node list than to load Shef to ask the same question. Knife is the CLI while Shef is the interactive mode for the Chef API - whereas the Fog API has the interactive mode but does not have a CLI.

The thing you can't easily do from inside Fog is to go back-and-forth to and from the shell or the rest of the shell script (this answers the question in phase #1). Personally, I have found it more convenient to use Fog from a separate terminal tab/window dedicated to a running Fog interactive session, and switch between that tab and the tab dedicated to the rest of the shell session. That is more convenient for me than loading a Fog session, doing one thing, closing the Fog session, doing a shell command in the shell, loading another Fog session, doing onother thing, closing the Fog session, doing another thing, etc. While Fog is fantastic, this tells me that I find using Fog from shell sessions or shell scripts to be less seamless (a little more seamful?) than it could be in a perfect world, and I confirm this by looking at how I use Chef compared to how I use Fog from the shell. Likewise, AWS has a command-line API for its services while both AWS and Rackspace have HTTP APIs which one can talk to with curl (although I would prefer to use Fog than to use the particular cloud APIs).

As a suggestion for starting and seeing what others think, perhaps a read-only CLI would be helpful. At least initially, if I want Fog to answer a question, I might use the read-only CLI (fog rackspace server list --format json > servers.json); whereas if I have a more complex problem I would go into the interactive mode to solve it or I would wrap up the solution in a Ruby script.

[geemus]

Glad its coming across ok, sometimes I'm not sure (and I was in a grumpy mode yesterday for unrelated reasons), so I figured I better double check.

Interesting, there is one other (incredibly poorly documented) option that may serve your purposes but I always forget about. By default the first argument to the interactive mode is what credential set to use (if you omit it then it uses :default). The optional second argument will actually be executed immediately and returned as json (rather than starting the interactive mode). Maybe this will serve your purposes. As an example, to get your list of rackspace servers you might do this: $ fog default Rackspace.servers Which will return (at the command line) your list of servers converted to json.

I'll warn you though, that since I forget about this and people (to the best of my knowledge) mostly don't use this, there are likely to be some edge cases and bugs that I hadn't considered. I originally added this as a way to shell out from a non-ruby process and read data from fog (this is slow, so wouldn't be good for your production setup, but can be great for experimentation).

This also provides well for my desire to be lazy and not write a new dsl/interface ;)

Would this provide for your needs? If so I think we could probably iron out the edges and make sure it is better documented, if not then we can hopefully improve it (or do a more traditional CLI like you mentioned).

[yfeldblum]

A nice CLI would be nice, but just executing the second argument goes a long way!

It is a giant hack. It requires the user to be diligent about encoding the second argument to be interpreted as a single string by the shell. It doesn't have the niceties of a CLI built with Thor or Mixlib, such as help (fog rackspace server would show what actions can be taken, such as list, show NAME, etc).

But for a simple way to use Fog from the command line doing one-line things, it works.

[geemus]

You can actually be quite lax about the encoding it takes 2nd and subsequent arguments, joins them with spaces and then evals that. So generally you don't have to worry about escaping or encoding or anything, I think it should 'just work'.

You should be able to get some of what you mentioned by using the kind of stuff I would use in the interactive usually. For instance Rackspace.collections and Rackspace[:compute].requests or, somewhat less helpfully Rackspace.servers.first.methods or something. This certainly is not as nice as the CLI stuff could be though.

Not sure how much that helps your concerns, but I had thought about at least the encoding issue when I wrote it.

[yfeldblum]

I was thinking more along the lines of backslashes \, dollar signs $, and semicolons ; - things that the shell interprets and doesn't pass through. In such cases you would have to quote the Ruby statement, and then encode any strings within the Ruby statement or use different quote characters, etc.

But it works:

fog default 'Hash[Rackspace.servers.select{|s| s.name =~ /^test\./}.map{|s| [s.name, s.addresses["public"][0]]}]' > test-servers.json

[geemus]

ah, yes. I forgot about those things (I'm not the most experienced shell-er). FWIW, I think you could replace s.addresses["public"][0]] with s.public_ip_address and make it a bit clearer whats going on. Any thoughts on making the quoting stuff less hairy?

[yfeldblum]

If you're going to pass in complex Ruby code as a string parameter on the command line, you've got to know how to deal with quotes and special characters, and you're probably going to have to do it by hand much of the time. I can't see much way around that.

[geemus]

Very true. Do you think we should start planning for a 'real' CLI or is this a good enough stop gap that we can put it on the back burner for now?

[yfeldblum]

I can only tell you what I would like to use - but what I would like is not necessarily optimal or applicable to a wide variety of potential users.

For the full feature-set, the only possibility is the Ruby API. No question about that.

But I don't think the equivalent of ruby -e is user-friendly, neither for beginner nor for expert, because Ruby is best in Ruby scripts and interactive Ruby sessions and CLIs are best for shell scripts and interactive shell sessions. ruby -e is alright for me some of the time as a stop-gap. But it is more complicated than it could be and, while I could get along without it, I would prefer to use a CLI when I need to use Fog and I am working in the shell or in a shell script. But that doesn't necessarily make it worth the effort.

For a partial or limited feature-set, I have been using the Chef CLI (Knife) and find it somewhat easy to use. Knife is a good comparison here because it has a similar model to Fog: multiple providers, multiple models per provider, and multiple queries/actions per model.

In terms of planning: it might be useful first to find out what other people would like. Is there enough interest in a "real" CLI and is there enough varied experience out there to inform what a "real" CLI ought to look like? That's the way to find out whether to go forward with a "real" CLI or to put the idea on the back burner. Are there people at EY who would find a Fog CLI useful in addition to the Fog API?

[geemus]

Those are all good questions, but not ones I have the most insight into. Mostly I only hear about bugs, rather than use cases.

Also, fwiw, knife uses fog internally for most of that stuff. So in some ways it already is the fog CLI (at least for that fairly limited use case).

[geemus]

I'm still thinking about this and making more notes for myself, but not much code to show for it yet (mostly just working on forming my opinions on what a good CLI might look like for us). Hopefully I'll have more to show for it and/or a proposal in the not too terribly distant future. Seems like it might be a good feature for part of the ongoing march toward 1.0.

Currently at EY I don't think there is too much call for a Fog CLI, but now that I think about it more this is likely because we have abstractions on top of that which are useful/powerful and we have CLI tools for those instead.

Anyway, nothing in particular to add at the moment, just wanted to let you know the investigation is ongoing.

[geemus]

I have been entirely convinced there should be a CLI, but I don't think an issue is the easiest place to discuss it. It is probably still a little ways off (trying to hold off to finish a 1.0 first). That said as it gets closer I hope to draft a proposal and throw it out for discussion on the ruby-fog google group (so look for more info on this there as we move forward).

[yfeldblum]

I agree, finishing a 1.0 with core functionality is more important than delaying a 1.0 to work on non-core functionality.