PowershellFrameworkCollective / psframework

A module that provides tools for other modules and scripts
MIT License
433 stars 41 forks source link

Suggestion: Module Dependency Incorporation #648

Open merddyin opened 1 month ago

merddyin commented 1 month ago

One of the items that holds me back on leveraging PSFramework for module dev, if only slightly, is the need to have a separate dependent module. While this is fine for many modules run from standard workstations, I tend to do a lot of work focused on implementation of zero-trust. Because of this, I often end up needing to run modules from systems without internet access, such as servers or privileged admin workstations. I know I can copy the dependent module over, but might it not be far easier to incorporate the required elements more directly? These are just thoughts, so feel free to disregard of course.

My thoughts/suggestion here is to achieve integration along the lines of one of the approaches outlined below.

Explicit Integration

This approach involves adding more options, in terms of specific features of PSFramework that the module maker wishes to implement. For example, providing the options to incorporate configurations, classes, protected commands, and logging, while excluding other features. In this scenario, required component elements would be copied to the target module as private component elements that would load as native to the module, perhaps as internal cmdlets. To reduce complexity for the user, the build process could also institute checks for PSF command calls for features that weren't included initially, and offer the option to update the module to add the required bits.

Pros:

Cons:

Implicit Integration

This approach could be achieved by implementing a sort of 'plugin' type approach, in similar manner to that used by ModuleBuild. In this scenario, the entire PSFramework module can semi-easily be incorporated into the main module as a single package as sort of an additional set of "private" cmdlets by only loading the plugin within module scope. I've attempted this approach in the past with ok results, though there are some loading challenges that would need to be worked through.

Pros:

Cons:

You could also, in theory, sort of combine the two approaches. In that scenario, you'd probably want to separate the main module into discrete feature sets that get copied to the Plugins location set by the template. You'd also likely want to incorporate some elements by default. There are lots of quality of life things that would make sense to include in every build, such as the protected command functionality.

FriedrichWeinmann commented 1 month ago

Heya @merddyin , thanks for the quite thorough feedback and thought spent on this! This discussion has come up repeatedly through the years (especially at the conferences I usually hang out), and I don't think this is ever going to end. Not a bad thing either, as it shows, that people are interested in the toolkit, so I much appreciate it. I also get the opposite feedback, where people are happy about having one dependency handle all their needs (and don't care about the extras it offers that they don't use).

I've put together my conceptualization on module types at the beginning of this year's conference talk on templating and module mass production. Summarizing: PSFramework is basically most of my Framework stack, splitting it up into components is against its design.

From a technical perspective, it is not designed to be split up, and other than some of the less integrated extra features that do not make up much of its code, they all tie into each other - logging uses configuration, configuration uses messages, messages use logging, messages use configuration, configuration uses debug, messaging uses debug, tab completion uses messages, tab completion uses debug, configuration uses tab completion, ... Trying to meaningfully break it down would be an incredible effort and a lot more fragile. It's an effort I cannot deliver on, even if I agreed with the approach.

Nesting components internally has further risk of invisible collisions between different consuming code parts and I shudder at the support impact that would cause. Could it be done? Possibly. The complexity required would make this a full-time effort I am very, very far from being able to commit to, sorry.

In the meantime, internal module deployment to offline environments can be simplified using an internal PSGallery, which you can even host in a file-share. This would allow you to run install-module from that offline computers and solve the package management requirement. It would also allow you to integrate internal code-signing before re-distributing approved modules. Here's some docs describing that

On October, 15th, there is an online mini-conference from the PSConfEU team - this may be for you, as Sydney Smith will talk on that very topic and what's the current state, including the new versions of packagemanagement. If you can distribute PS7 internally, your offline machine will have the new tools available out of the box - though I can understand if that is not possible for you from an organizational perspective (native PS7 integration with OS is a common issue requested and discussed).

What I do with several customers is maintain an internal repository and satisfy the zero-trust needs by manually verifying all modules before introducing them to the internal environment. For that we clone the github repo into an internal repository and code-review there. This allows us to later do a delta-PR and see what has changed, rather than having to do a full review for the entire module each time I want to update. Then we build it from source and publish internally (all my module repositories include the automation tools needed to build them locally without being bound the a particular service, such as Azure DevOps or Github).

merddyin commented 1 month ago

As always, appreciate your kind consideration of my thoughts.

In terms of the amount of effort involved in splitting things up, I totally get it, which is why I didn't really think that approach would fly. I suggested it partly from a perspective of being thorough, and partly just in case. That said, I guess I hadn't dug through the code quite enough to realize the integrations ran so far or so deep.

From the perspective of nesting however, I am unsure of what you mean as far as risk of invisible collisions. As I mentioned, I have experimented with this approach some with PSFramework while I work through how to leverage the module. In general, the module loads into the private scope with minimal issues, and everything appeared to work well enough. I do vaguely recall seeing an error of some sort last time I worked on this, but I admit it has been some time and I would need to dig for the project to retest so I can provide accurate information.

In terms of both PS7 and the private repo approach, I am intimately familiar with both. Any system under my active use always gets PS7, Windows Terminal, and my preferred set of modules. The challenge I generally face is that I am a consultant, so most of the environments I work on are not my own. I always recommend and encourage deployment of PS7, but most orgs don't see the point as it is an extra step they have to take when WPS5 is already there and works "well enough". And don't even get me started on the perspective of most InfoSec teams, who all seem to want as little PowerShell as possible in their environments because they don't know how to properly secure it. Private repos are a bit easier sell, but unless it's part of the focus for my project, it generally goes nowhere because the customer teams are "too busy" and they'll "put it on their roadmap", even though I tell them it can be as simple as a file share.

The inability to reliably have an internal repo is probably the biggest driver for my suggestions here. Much of the work I do is in higher security environments, or environments attempting to achieve higher security at least. I frequently end up building a lot of modules with tooling to ease both deployment and ongoing administration overhead since most models are simply too complex for most orgs to maintain otherwise. Being able to incorporate the Framework directly within the module, even as a plugin (maybe even more-so as a plugin), would enable me to avoid the need to document out the dependencies or needing to provide additional instruction to teams that, while usually at least somewhat familiar with PowerShell, are more often just franken-scripters than anything else. That said, I can understand your reluctance to take on such an effort.

While I do not have a functional project leveraging the Framework as a plugin presently, it is still something on my back projects list. Integrating as a plugin keeps the entire module fully intact in it's own folder, and I always include references in the notes and adhere to all licensing for the modules, though typically I am unable to share my work since it is part of my projects. Since the time I posted issue 576, I've been able to find a few more projects (including some of your newer ones) leveraging the Framework, which will make it easier for me to implement effectively, so I'm hoping to make more use of the Framework soon. My prior experiments were while I was between projects, but I've been too busy since to get back to them lately...need to fix that lol.

In the event that I am able to successfully put together an integration path using the plugin approach, would you be interested in taking a look, and maybe offering feedback? Either way, feel free to close this particular issue if you like.

FriedrichWeinmann commented 1 month ago

I always recommend and encourage deployment of PS7, but most orgs don't see the point as it is an extra step they have to take when WPS5 is already there and works "well enough". And don't even get me started on the perspective of most InfoSec teams, who all seem to want as little PowerShell as possible in their environments because they don't know how to properly secure it.

Oh how I know this situation ... It's a constant struggle. What I generally try to do for one-off engagements, is provide bootstrap logic for a file-share based repository and have the project bootstrap itself from that repo.

From the perspective of nesting however, I am unsure of what you mean as far as risk of invisible collisions.

Both the DLL and the type extensions can only exist once in the process. The issue arises when multiple projects try to coexist. Let's say we have two, independently developed modules:

If they now have a console and first import ModuleA (which will work fine), trying to import ModuleB will fail, complaining about an error in its dependency (PSFramework). The Contoso ltd. admin will have no indications, that your project contains an internal PSFramework version - it is invisible.

Splitting PSFramework into plugins - if successful - would still have a type-name collision between different projects using either the full Framework or its own plugin with a shared subset. Some more components are actually, intentionally aimed at shared use across different projects (e.g. logging, configuration) or have external engine integrations that would collide (Tab Expansion). So ... yeah, that's going to be a struggle.

In the event that I am able to successfully put together an integration path using the plugin approach, would you be interested in taking a look, and maybe offering feedback?

Absolutely. I may not look to support this and it may go against my intentional design decisions, but the nerd in me would absolutely be fascinated by the implementation. You may also want to reach out to James Brundidge, he's spent a lot of thought on this matter and has some significant tooling. Just ... bring some time when you approach him - he'll be enthusiastic to interact with a likeminded fellow :)

merddyin commented 1 month ago

Ah, I understand your concerns around the use as a plugin. and I can see where there might be a problem, though not in the way you are describing. The approach I have been using addresses the use case you outlined to a limited degree, though there are other collision risks that I can see now.

From a plugin vs public scenario, the plugin import process uses specific per plugin load/unload ps1 files. Within this file, I always check for the presence of the module in the public scope first. If the module exists, the load process for that plugin gets aborted, and implicit import is used when I call cmdlets from that module. The loader also checks for the plugin module already being resident in memory in an accessible location, though this only protects against a scenario where the unload process wasn't fired and a re-import within the same session is attempted.

The bigger collision risk comes from two modules that are using PSFramework as a plugin on the same system. In that scenario, since ModuleA only loads the plugin within its private scope, ModuleB would not see it in either the publicly available modules or as already resident and would attempt to load normally. Based on your description, the DLL and type extensions would fail to load, so the module would also fail to load for ModuleB. Since the commands loaded by ModuleA are private, ModuleB fails to run.

That said, it should be easy enough to try/catch during load and handle the DLL/type loading another way. There are some tricks with remoting that can functionally load a module into an isolated context, provided it is available and the user has local permissions, though this may cause other issues because of how output is handled. In reviewing this article and this GitHub sample, it seems like there might be some possible avenues to explore using AssemblyLoadContext. Not sure if I've taught myself enough .NET yet to understand or implement some kind of wrapper or not, but seems like an interesting bit of research.

From reading the content, I'm aware that this may introduce limitations, particularly in areas like the engine integration elements you mentioned, so the question I guess I'd have to answer is whether the trade-offs are worth it to make loading easier. Some things, like the tab expansion bits, I seem to recall being made easier in PS7, so perhaps I can leverage that as an additional reason for pushing PS7? Thoughts?

As for not being in alignment with your design, I completely get that, and I know some in the community can be...less than happy when people go their own way anyway. It's awesome that you embrace the nerdier aspects! As for reaching out to James, I'll certainly do that (I assume he's on GH somewhere?). I'm of a similar type to him it sounds like, in that I enjoy thoroughly nerding out and talking for hours on topics I am passionate about. Thanks for the info!

FriedrichWeinmann commented 1 month ago

AssemblyLoadContext would solve the majority of the collision risks ... if they were viable. PSFramework is a hybrid module, not a binary one - the script parts need to interact with the C# parts, and that means the C# parts need to be in the main Context of the PowerShell process. Also that only works on 7+ and I very strongly strive for 5.1 compat (and even 3.0 for many features).

For the C#-only bits it would work though.

On the other hand, it would also prevent features from cooperating that intentionally should work together with all consumers in the process (e.g. logging, configuration).

I assume he's on GH somewhere?

https://github.com/StartAutomating