Support CLI plugin extensibility

[phaumer]

The current Zowe VS Code extension only supports the default CLI that is implemented using zOSMF. However, the CLI can be extended with plugins that could provide other means of accessing z/OS. For example USS could be accessed using FTP or SSH protocols. (We are currently in the process of implementing a CLI FTP plugin that would do such a thing for clients who do not have zOSMF configured.)

The Zowe VS Code extension should support such plugins and provide an API for other VS Code extension to contribute such implementations dynamically based on what customers are installing.

We propose that the Zowe VS Code extension defines a set of API interfaces that abstract from using zOSMF (the zowe.method calls in the current codebase). It should also provide and API registry that has the means for other VS Code extensions to contribute its implementation of the API interfaces that should be used in addition of the default zosmf implementation. (similar to a plugin implementing an Eclipse extension point)

An end-user scenario would be:

• customer installs the Zowe CLI and the FTP CLI plugin and create an ftp profile
• customer installs the Zowe VS Code extension
• customer installs another VS Code extension (e.g. an optional Zowe VS Code FTP expansion) that registers its FTP plugin-based implementation of the Zowe VS Code APIs
• in the Zowe VS Code USS Viewer when the user clicks the + icon to add a profile she can choose from all available zosmf as well as ftp CLI profiles to add them to the tree.
• if they add both then each will be used with a different API implementation that is selected based on the profile type

[dkelosky]

You mentioned that you're implementing a CLI FTP plugin, is it different than this: https://github.com/zowe/zowe-cli-ftp-plugin ?

[phaumer]

Oh dear, I did not realize that existed @dkelosky :-) Thanks for pointing us to that. We will look at that and see if we can use it instead and make extensions to that one if needed to support the scenarios outlined above..

[MikeBauerCA]

@phaumer do you see significant interest from the mainframe community in using FTP over z/OSMF? I'm in favor of enabling plug-in extensibility and allowing folks to use the services they have/want, but I am trying to evaluate the priority of this initiative.

[phaumer]

@MikeBauerCA yes, this is important for us as we talk to a lot of customers who do not have zOSMF configured. This item provides two important enhancements:

1. An extensibility mechanism adding the ability for other third party VS Code extensions to inject other API implementations into Zowe Explorer to use for populating the tree browsers.
2. An implementation using the FTP plugin as an example for such an alternative API.

I am working on a branch that I can share soon for feedback.

[jelaplan]

I'm not sure I fully understand how this works. It sounds like the Zowe Explorer extension has an extension of it's own - that is extensions to the extensions. But, regardless, the story is interesting in that a user can create a new connection and use either zOSMF or FTP to connect to the Mainframe.

In your story, the user can choose from a combined list of zOSMF and FTP profiles. This leads me to wonder whether you've thought of ways to author an FTP profile from scratch. Therein the user would need to choose between one or the other. It might be done by adding an additional prompt to the new connection dialog. A downside to that is adding yet another question makes that workflow longer.

It would be good to get some customer feedback after the fact on whether and how they are using FTP. If it is popular, I'm a proponent of making the connection type (ftp or zosmf) an option for the existing commands. It is a bit messy to add the FTP group to the CLI which duplicates much of the commands from ZOS-FILES and ZOS-JOBS groups. We've had customers complain that the command syntax is a lot to digest and by adding a FTP group, we are increasing the complexity.

[phaumer]

@jelaplan yes, I want to make the extension extensible as there are many other APIs out there that we want to consider supporting. There are even two more in Zowe in addition to ftp: Zowe REST and ZSS and ssh could be another way this could be implemented. Plus there are vendors like ourselves who have additional APIs that we want to "plug in".

So yes, there is one more question which is the profile type and then the following questions would all be specific to each profile type.

I think the user-experience of these quick pickers is indeed not very good. As soon as you click away to copy your password or connection string the pickers might go away and you have to start over. I think they are only valid for one to three questions where you do not have to type difficult things. Zowe CLI profiles are just yaml files. So I think it would be more consistent with the VS Code user experience of letting users just write yaml files with code templates and code completion just the same way you use json and code completion to write user and workspace settings in VS Code. Another idea could be writing yeomans or yeoman-like commands for the Zowe CLI for creating connection files perhaps.

[phaumer]

I pushed a branch that implements a first prototype that is merged with master up to the 23rd: https://github.com/zowe/vscode-extension-for-zowe/tree/extension-api

It currently does the following:

• defines API Interfaces for USS and MVS
• updates the Profiles class to load all profiles (not only zosmf) and to use the API Registry to filter down the profiles lists based on the profile types for the ones that have API implementations registered.
• provides an API registry that can be retrieved updated from third party VS Code extensions that want to contribute additional implementations of the APIs.
• implements USS and MVS APIs for zOSMF and replaced all occurrences where zosmf CLI APIs were called directly with calls to the api registry based on the profile.
• implements all USS operations using the Zowe FTP CLI plugin; user can choose to add their zosmf or zftp profiles or both to the USS Explorer (zftp profiles will not be offered for the other two trees as they have not implementations for zftp, yet)

TODOs:

• As the FTP CLI plugin is not available on npmjs you need to clone the repo in a parallel directory, build it and do an npm pack; then do an npm install in the extension-api branch to load it.
• etag conflict merge UI during Save currently broken for ftp, because ftp content is streamed; need to add a Promise that waits
• add localizations
• update/fix tests
• define and implement zosmf JES API
• implement MVS and potentilly JES for zftp

Add one new dependency that conveniently computes temp file names: https://github.com/raszi/node-tmp

@Colin-Stone @zFernand0 would you guys have time to review and provide feedback?

[Colin-Stone]

Thanks Peter, this represents great evolution of Zowe Explorer I really like the ability to register additional extensions that add function and value to Zowe Explorer.

If I am reading this correctly we can augment VSCode with an additional extension that provides FTP support. Is this cumulative?. Can we add more and more enabling Zowe explorer to support multiple profile types?. Ultimately this means that the user is able to pick and choose from different API's simply by adding a profile of a specific type.

The functionality we have currently such as list files, get content etc is not bounded by a pre-defined interface and has been designed literally by the available zosmf functionality. Although a decent set of API's they may also be restrictive which is why I don't think that providing an explicit API that implements USS, Jobs, Datasets etc based on zosmf is necessary and potentially a backward step.

Considering the way a VSCode extension/plugin is architected. It's a complete self-contained unit containing all the code and definitions it needs. Central to it's behavior are the command definitions in the package file which associates functionality with a particular object type. So for example an object with the context of "uss_session" can have explicit operations and commands associated that are then implemented in the code. The session object is key here since it relates directly to the profile. The "uss_session" has functions like filter and list which should explicitly create "uss_files". This has a function which can download content either as binary or text which is a restriction on the API. It's possible an alternative profile/API that isn't restricted could alternatively provide say a "uss_binary_file" for example.

So an extending or "complimentary" extension registers and provides support for a new profile type (e.g. FTP). It is in the package.json file for this VSCode extension where we then define a new set of commands for "ftp_session" which relate specifically to code within the "complimentary" extension, providing alternative and additional function. Apart from the behavior of the "ftp_session" object I don't see any need for the original the API. Although this sounds like a lot of code replication there is a branch Inter2 which shows how we can build upon the ZoweTree.ts interfaces you recently provided to Zowe Explorer and provide both core interfaces and abstract functionality of the items without specifying implementation or indeed behavior.

I think this comes down to a binary decision on how we handle different profiles within Zowe Explorer.

1. Create a hard interface describing the operations that each object has to perform.
2. Allow different profiles to behave differently, even inconsistently because they are providing alternative information to the user.

[phaumer]

Thanks @Colin-Stone. I agree with what you are outlining will provide additional value as one could add additional behavior to the tree browser. It would be good to explore and list a couple of examples of added behavior that would be worth adding.

I do not think, though, that this is a binary decision, but rather can be seen as an additional layer of added complexity for an extender. The interfaces I am proposing keep it simpler and could be the first level of extensibility. They are just the core functions needed for a USS and MVS explorer such as list files, get contents, write contents, get as binary. Something that each data provider can easily implement without any extra gimmicks. This will be enough to quickly add support for the APIs already on the horizon for Zowe such as ZSS, SSH, FTP. For additional layers of functionality I agree we could work on another layer of more dynamic extensibility patterns.

For choosing the right abstractions I went through several iterations. For example, I also though at first that the CLI session would be a valid abstraction to use, but then realized that session is only an abstraction for http and breaks down when using other protocols and it is not extensible, so I replaced it with profile, which is extensible by design. The CLI and Imperative provided almost all the abstractions needed in addition to that, such as IZosFilesResponse, IUploadOptions, IDataSet etc. which were all mostly zosmf and http independent.

[Colin-Stone]

Thanks for that explanation @phaumer, I recognize the potential value of the specific interface and as you describe it doesn't prohibit alternative extensions implementing completely different strategies. Regarding embedding the FTP implementation into Zowe Explorer would this not be better as an extension to Zowe Server based upon your interfaces?. That way we don't put more code into the base Zowe Explorer minimising complexity and maintenance and FTP specific users are able to pull it in through the marketplace?.

[phaumer]

FTP example of API is here now: https://github.com/phaumer/vscode-extension-for-zowe-api-sample

[phaumer]

Merged.