search

gaelcolas / DscInfraSample

A Control Repo for a DSC Infrastructure as Code
MIT License
47 stars 22 forks source link

What is the command you use to actually fire off this build process? #8

Open VertigoRay opened 6 years ago

VertigoRay commented 6 years ago 
PS C:\Temp\DscInfraSample-DEV> .\.Build.ps1
Invoke-Build : The term 'Invoke-Build' is not recognized as the name of a cmdlet, function, script file, or operable
program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\Temp\Git\DscInfraSample-DEV\.Build.ps1:39 char:9
+         Invoke-Build $Tasks $MyInvocation.MyCommand.Path @PSBoundPara ...
+         ~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Invoke-Build:String) [], CommandNotFoundException
    + FullyQualifiedErrorId : CommandNotFoundException

I don't see an Invoke-Build function anywhere that you've provided. Does this repo function by itself or are there dependencies that I'm not noticing? I can only assume you're needing https://github.com/nightroman/Invoke-Build as a dependency.

Even with InvokeBuild installed (Install-Module InvokeBuild -Force), I still have errors:

PS C:\Temp\Git\DscInfraSample> .\.Build.ps1
DSC_Configurations
Build . C:\Temp\Git\DscInfraSample\.Build.ps1
Task /./Clean
------------------------------------------------------------------------------
                         CLEAN UP
------------------------------------------------------------------------------
Done /./Clean 00:00:00.0365730
Task /./PSModulePath_BuildModules
Done /./PSModulePath_BuildModules 00:00:00.0160784
Task /./test

    Directory: C:\Program Files\WindowsPowerShell\Modules

ModuleType Version Name   ExportedCommands
---------- ------- ----   ----------------
Script     0.0.30  datum  {Get-DatumRSOP, Get-FileProviderData, Get-MergeStrategyFromPath, Invoke-ProtectedDatumAction...}
Script     3.4.0   Pester {Describe, Context, It, Should...}

Done /./test 00:00:00.1213200
Task /./LoadResource
ERROR: The term 'Invoke-PSDepend' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.
At C:\Temp\Git\DscInfraSample\.Build.ps1:60 char:13
+             Invoke-PSDepend -Path $PSDependResourceDefinition -Confir ...
+             ~~~~~~~~~~~~~~~
At C:\Temp\Git\DscInfraSample\.Build.ps1:57 char:5
+     task LoadResource {
+     ~~~~~~~~~~~~~~~~~~~
At C:\Temp\Git\DscInfraSample\.Build.ps1:52 char:5
+     task . Clean,PSModulePath_BuildModules,test,LoadResource,LoadConf ...
+     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Build FAILED. 5 tasks, 1 errors, 0 warnings 00:00:00.7169683
Invoke-PSDepend : The term 'Invoke-PSDepend' is not recognized as the name of a cmdlet, function, script file, or
operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try
again.
At C:\Temp\Git\DscInfraSample\.Build.ps1:60 char:13
+             Invoke-PSDepend -Path $PSDependResourceDefinition -Confir ...
+             ~~~~~~~~~~~~~~~
    + CategoryInfo          : ObjectNotFound: (Invoke-PSDepend:String) [], ParentContainsErrorRecordException
    + FullyQualifiedErrorId : CommandNotFoundException
VertigoRay commented 6 years ago

Key was to install PSDepend (Install-Module PSDepend -Force). This should probably be handled as a first step in the .Build.ps1. Something like this, maybe:

[CmdletBinding()]
param(
    [switch]$Force
)

$Install_Module = @{
    'Name' = @(
        'PSDepend'
    );
    'Force' = if ($Force.isPresent) {$Force.isPresent} else {(-not ([Environment]::UserInteractive -and (-not ([Environment]::GetCommandLineArgs() | ?{ $_ -like '-NonI*' }))))};
}

Write-Verbose "Installing Modules: $($Install_Module | ConvertTo-Json)"
Install-Module @Install_Module

That won't fix it all though, PSDepends prob needs to allow TLS 1.2: image

However, you can do it as well:

Write-Verbose "Set PowerShell to use TLS v1.2 ..."
[Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12

Then you can run any of the following and it would just works on a new system; including a clean CI build system:

I said would because there are still a couple of errors left:

Could not find the module '<SharedDscConfig, 0.0.3>'.

Could not find the module '<Chocolatey, 0.0.46>'.

I'll keep working through them.

VertigoRay commented 6 years ago

Thanks for the update, I've just started the build and will let you know of progress I make.

VertigoRay commented 6 years ago

ERROR: The required module 'xDscResourceDesigner' is not loaded. Load the module or remove the module from 'RequiredModules' in the file 'C:\Temp\Git\DscInfraSample\BuildOutput\modules\DscBuildHelpers\DscBuildHelpers.psd1'.

Import-Module : The specified module 'xPSDesiredStateConfiguration' was not loaded because no valid module file was found in any module directory.

Installing them manually for now: 

Install-Module xDscResourceDesigner,xPSDesiredStateConfiguration -Force

I have these installed on my dev machine. I'm testing this in a clean container so I know it'll work in CI.

gaelcolas commented 6 years ago

Hi, you missed the -resolveDependency parameter on the build script. That pulls PSDepend for you and sets the TLS settings for the session. Look at the video on my blog:

Building DscInfraSample

I had missed xDscResourceDesigner, and xPSDesiredStateConfiguration from the PSDepend.build.ps1. I've just added them, you can pull and you should be all set for your CI.

It currently needs to run as admin, that's because of xDscResourceDesigner, which is used in another module use to compress the Modules for the pull server. I'll remove the dependency so that's not required and can be ran as normal user.

VertigoRay commented 6 years ago

Works great now. The video is perfect. Thanks so much for taking the time to make it and help me through this.

The final BuildOutput folder has the following structure (as evident in your video as well):

BuildOutput
├─DscModules
├─MetaMof
├─Modules
└─MOF

This isn't the same as what's defined in the README. That's okay, it's pretty straight forward. Usage of the DscModules and MOF folders are pretty obvious to me; they need to be served by the Pull server from the modules and configurations folder, respectively. I know what the other two are, but I'm curious about usage:

I'm not familiar with InvokeBuild, but I'm so glad that I'm becoming familiar with it and some of these other tools. I've used Pester, but I've always just written my own code to handle all the build foo in Jenkins or GitLab CI. I'm glad I'm seeing other avid POSHers and their strategies.

gaelcolas commented 6 years ago

Was updating the documentation, feel free to have another look and provide feedback ;) Thanks for taking the time trying this, I know it's still very rough...

At work I use PSDeploy to deliver those MOF, MetaMOFs, and DscModules where they belong, I'll update that eventually here.

Re your questions:

My DSC build pipeline takes the same approach than my Module pipeline, in my SampleModule project.

VertigoRay commented 6 years ago

How would you handle variable injection within the YAML? I want the Node to look for a configuration with it's $env:ComputerName as the file name. Maybe I'm configuring the LCMs wrong?

LCM_Config:
  Settings:
    ActionAfterReboot: ContinueConfiguration
    AllowModuleOverwrite: true
    ConfigurationMode: ApplyAndAutocorrect
    ConfigurationModeFrequencyMins: 15
    RebootNodeIfNeeded: true
    RefreshFrequencyMins: 90
    RefreshMode: Pull
  ConfigurationRepositoryWeb:
    ServerURL: https://dsc.server:8080/PSDSCPullServer.svc
    RegistrationKey: c2339b32-0505-4269-9650-863da0632901
    ConfigurationNames: $Node.Name
  ResourceRepositoryWeb:
    ServerURL: https://dsc.server:8080/PSDSCPullServer.svc
    AllowUnsecureConnection: false

You use a ConfigurationID, but that's just around for backwards compatibility, right? Eitherway, I don't want to manage a GUID to Computer Name map.

VertigoRay commented 6 years ago

I wonder if .build, RootConfiguration.ps1, and RootMetaMOF.ps1 should all be made into functions and made to be part of Datum ...

I feel like this repo should be just DSC_ConfigData, DSC_Configurations, DSC_Resources, and a .Build.ps1 file that would have one line in it; unless something really unique was needed:

Invoke-DatumBuilder `
    -ConfigData .\DSC_ConfigData `
    -Configurations .\DSC_Configurations `
    -Resources .\DSC_Resources `
    -BuildOutput .\BuildOutput

Those would probably be the default parameter values so could make it just a function call, but you get the idea.

Thoughts?

VertigoRay commented 6 years ago

I keep getting following error when running .Build.ps1, and I'm feeling a bit stuck:

PSDesiredStateConfiguration\node : The term 'xWallpaper' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

ERROR: Compilation errors occurred while processing configuration 'RootConfiguration'. Please review the errors reported in error stream and modify your configuration code appropriately.

Obviously, you would think that xWallpaper is not available as a DSC Resource, but I've added Import-DscResource -ModuleName 'xWallpaper' to the RootCofnig. Also, it shows up with the Get-DscResource 'xWallpaper' command.

I also have a very extensive Pester test that runs on this machine and passes 100%; which includes tests like:

Invoke-DscResource `
    -Name 'xWallpaper' `
    -ModuleName 'xWallpaper' `
    -Method $null `
    -Property $Property

So, it's definitely there. I've checked the $env:PSModulePath during runtime and it looks right. I've called Get-DscResource 'xWallpaper' right before this error, and the module is returned.

To be more specific, it's the following command that's failing in the RootConfig:

Get-DscSplattedResource `
    -ResourceName $ConfigurationName `
    -ExecutionName $ConfigurationName `
    -Properties $Properties

I've extracted all the variables and the splat hash would look like this:

$Get_DscSplattedResource  = @{
    ResourceName = 'xWallpaper'
    ExecutionName = 'xWallpaper'
    Properties = @{ ... }
}

I'm not sure if something comes to mind for you ...

gaelcolas commented 6 years ago

How would you handle variable injection within the YAML?

You don't, you have to specify it. The key here is to use merge behaviour, so that you specify all those defaults somewhere in a generic layer, and only specify the ConfigurationNames (and RegistrationKey if you feel like it) at the Node layer.

Variable interpolation is not implemented yet, but you can do somethings similar using DatumHandlers, another undocumented feature... In short, the DatumHandler can do a string matching to pass some data to a function that you can create yourself. If that function has a parameter Name that exists in the parent scope (invocation context), then it will be automatically passed to it. So if your {0}\Invoke-{1}Action function, with {0} -> Your module Name, and {1} -> Datum Handler name, has a parameter namedNode, then the variable $Node will be passed to the function, and you can make it return$node.Name`. I'm afraid I don't have much time at the moment to implement natively, but It's on my list.

Variable Interpolation will not handle all things starting with $, but more likely all string matching something like #{} or something else recognisable so that we can do something like #{$username}@#{$domain}.

I wonder if .build, RootConfiguration.ps1, and RootMetaMOF.ps1 should all be made into functions and made to be part of Datum ...

Probably not, Datum is actually a generic hierarchical data store, with a key/value store kind of experience. I also have a couple of other usages in mind, hence the reluctance of making it too tightly coupled with DSC.

I have another module that could be a better fit though, gaelcolas/DscBuildHelpers, but it's still a bit early to integrate. I know there's a few features that needs to be implemented first (actually, ported over from my work repository :/), to improve the development and usage experience.

Yep, the Invoke-DscBuild was my first approach (from Steve Murawski's DSC tools), but DscInfraSample needs to mature a bit before making it happen.

Also, I still prefer the Task runner approach for some of it, because it gives you a better customisation experience for the workflow (adding your own tasks, and gates). The other problem is that, for now, you need to customise RootConfig.ps1 with your Import-DscResource.

Obviously, you would think that xWallpaper is not available as a DSC Resource, but I've added Import-DscResource -ModuleName 'xWallpaper' to the RootCofnig.

Almost definitely an issue with the DSC resource not being found correctly, and that's one of the improvements I'm meant to add: Error handling (capturing and enhancing...).

Most likely, it can find 2 versions, or is not loading into the right context (the PSDesiredStateConfiguration.psm1 makes it tricky to hook into, and splatting needs some massaging).

First advice would be to use the -noInvoke parameter to the Get-DscSplattedResource function doing like:

(Get-DscSplattedResource `
    -ResourceName $ConfigurationName `
    -ExecutionName $ConfigurationName `
    -Properties $Properties`
    -NoInvoke).Invoke($Properties)

The reason for that is scope (isn't it always?!?)... The -NoInvoke parameter returns the generated ScriptBlock, that you invoke with .Invoke($Properties) so that it's executed in the right context (DSC Composited Resource, aka Configuration, instead of the Root configuration). That's also why Get-DscSplattedResource is a ScriptToProcess, and note a function of the Module... very annoying :( I wish splatting was native to PSDesiredStateConfiguration.psm1 !!!

The second advice is to explicitly Version the module you're referencing in your RootConfiguration.ps1: 

Import-DscResource -ModuleName 'xWallpaper' -ModuleVersion 0.3.0

There's multiple benefits to this such as:

It also simplifies the job to find the version and troubleshoot issues with DSC, which is very annoying, as you're starting to see.

Here's a few tips and tricks for troubleshooting: If 2 versions of same module available in $Env:PSModule it will most likely fail to compile. If a Module with DSC resources is present but has dependencies not identical in version has what you are using, you're screwed... As an example, you have two Composite Resources importing xPSDesiredStateConfiguation (using Import-DscResource), and one is specifying version 8.0.0. If the other is specifying another version, or has already loaded another version, it will fail, probably with the same error message you stated above.

'seeing' the DSC resource with Get-DscResource 'xWallpaper' is a good start, but not sufficient.

If not enough info is returned, remember to explorer the $Error variable for some more details. For instance, I always do $Error[0..4] when a weird error happened, so see the 5 last error messages. DSC (again, that PSDesiredStateConfiguration.psm1) tends to swallow the errors and only return a generic issue.

This bit is really painful at the moment, especially when you get started. After you start getting used to it, and you're explicit with the versioning, it's ok, just a habit to have. What I really dislike, is that the error handling is rubbish, and we should do better to fail faster (read better tests). Also, we are duplicating the source of truth from the PSDepend files, and this RootConfig.ps1. My idea is that we could dynamically generate the RootConfig.ps1 scriptblock and invoke it with all Import-DscResource -ModuleName ... -ModuleVersion, but that's low in term of priorities (for now... I should create an issue to start get some vote maybe...).

Thanks for your feedback ;)

VertigoRay commented 6 years ago

I finally got it working in a way that I'm happy with. It includes some splatting of the resource. I'll fork this repo and show my changes when I have time to cleanup the private foo. In the meantime, below are some of the key changes.

PS> Lookup 'Configurations'
Wallpaper

RootConfiguration.ps1

[CmdletBinding()]
param()

Write-Host "---------->> Starting Configuration" -ForegroundColor 'Blue' -BackgroundColor 'Yellow'
$BuildVersion = $Env:BuildVersion

Write-Verbose "Import DSC_Configurations that appear to be needed: $((Lookup 'Configurations') -join ', ')"
foreach ($Configuration in (Lookup 'Configurations')) {
    $ConfigurationToImport = (Get-ChildItem '.\DSC_Configurations\' -Filter "${Configuration}.ps1" -Recurse).FullName
    if ($ConfigurationToImport -is [string]) {
        Write-Debug "Found 1 Configuration; importing: ${ConfigurationToImport}"
        . $ConfigurationToImport
    } else {
        Write-Debug "Found $(($ConfigurationToImport | Measure-Object).Count) Configurations; importing: $(ConfigurationToImport -join ', ')"
        foreach ($cti in $ConfigurationToImport) {
            . $cti
        }
    }
}

configuration RootConfig {
    Import-DscResource -ModuleName PSDesiredStateConfiguration

    if ($env:BuildVersion) {
        & $(Get-Module PSDesiredStateConfiguration) {param($tag) Set-PSTopConfigurationName "MOF${tag}_"} $env:BuildVersion
        Write-Verbose "PSTopConfigurationName: $(& $(Get-Module PSDesiredStateConfiguration) {Get-PSTopConfigurationName})"
    }

    node $ConfigurationData.AllNodes.NodeName {
        Write-Host "Processing Node $($Node.Name) : $($Node.nodeName)" -ForegroundColor 'Blue' -BackgroundColor 'Magenta'

        foreach ($ConfigurationName in (Lookup 'Configurations')) {
            $Properties = $(Lookup $ConfigurationName -DefaultValue @{})
            Write-Host "Properties: $($Properties | Out-String)" -ForegroundColor 'Blue' -BackgroundColor 'Cyan'
            & $Configuration $Configuration @Properties
        }
    }
}
RootConfig -ConfigurationData $ConfigurationData -Out $(if ($BuildRoot) { "${BuildRoot}\BuildOutput\MOF\" } else { "${PSScriptRoot}\BuildOutput\MOF\" })

DSC_Configurations\Wallpaper.ps1

Configuration Wallpaper {
    Param(
        [ValidateSet('Absent','Present')]
        [string] $Ensure = 'Present'
        ,
        [ValidateSet('AllUsers','LockScreen','AllUsersAndLockScreen')]
        [string] $TargetWallpaper = 'AllUsersAndLockScreen'
        ,
        [ValidateScript({ [System.IO.FileInfo]$_ })]
        [Parameter(Mandatory=$true)]
        [string] $Path
        ,
        [ValidateNotNullOrEmpty()]
        [string] $PathAgeDays = 1
        ,
        [ValidateNotNullOrEmpty()]
        [string] $SourcePath = 'https://unt-wallpaper-dev.azurewebsites.net/?w={0}&h={1}' # https://github.com/UNT-CAS-ITS/Wallpaper-Server
    )

    Import-DscResource -ModuleName xWallpaper -ModuleVersion 0.1

    xWallpaper $InstanceName {
        Ensure          = $Ensure
        TargetWallpaper = $TargetWallpaper
        Path            = $Path
        PathAgeDays     = $PathAgeDays
        SourcePath      = $SourcePath
    }
}
gaelcolas commented 6 years ago

If that works for you and help you moving towards Infra as Code, then great, I'm happy! Let me warn you though, that your current model is risky, and might become a problem/constraint. From what I can see, you're relying on configurations, as PS1, included in your control repo, instead of DSC Composite Resources, built and packaged in their own and respective pipeline - artifacts. The control repo should only assemble and compose different tested artifacts (shift left), by gluing them with the Configuration data.

VertigoRay commented 6 years ago

Very observant, but this was definitely a conceptual simplification.

We already manage our Linux systems with Saltstack, and I'm already a huge advocate of infrastructure as code. I started development on a version of Chocolatey long before Chocolatey was a thing. We use it internally to update, detect, and install/uninstall software on our desktops; supporting over 75 licensed and free to use software titles. All of the code is in our internal GitLab server, and there are pipelines that test the installation/detection/uninstallation (such as Firefox with our proxy) automatically on Win7, 8.1, and 10 (32 and 64 bit of each). These pipelines must pass before anything is released to production.

Since Chocolatey is such a powerful tool, I've thought about rolling a lot of the work we've done into Chocolatey. However, the lack of a detection method in Chocolatey (ConfigMgr Apps) has kept that at bay. I realize that Chocolatey has a licensed version that can do some lite detection, but it's overly simplistic and not sufficient.

I was debating using DSC, but DSC doesn't like running in Push/Pull mode side by side. Maybe as Partial Configs, but I would be definitely getting into the realm of non-supported usage.

Anyway, I'll try to make some time to get this all published publically after some other deadlines I have pass. I mean, I really want to get these published, but ... you know ... work is paying the bills. ;)

VertigoRay commented 6 years ago

Since Datum is a generic hierarchical data store, should tools like Get-DscSplattedResource be broken out into another project/repo?

I've done some design changes to this function for use in a composite resource I've been working on, and I'm currently wanting to break out that function out into it's own repository.

gaelcolas commented 6 years ago

Yup, it's meant to go in DscBuildHelpers ;)

On Wed, 11 Apr 2018, 13:26 Raymond Piller, notifications@github.com wrote:

Since Datum is a generic hierarchical data store, should tools like Get-DscSplattedResource https://github.com/gaelcolas/Datum/blob/master/Datum/ScriptsToProcess/Get-DscSplattedResource.ps1 be broken out into another project/repo?

I've done some design changes to this function for use in a composite resource I've been working on, and I'm currently wanting to break out that function out into it's own repository https://github.com/UNT-CAS/OneDriveDsc/issues/13.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gaelcolas/DscInfraSample/issues/8#issuecomment-380584669, or mute the thread https://github.com/notifications/unsubscribe-auth/AIjANXcxZZjqStTqHMJMop_k6a4oxZdgks5tnmcDgaJpZM4S5Wso .

gaelcolas commented 6 years ago

D'oh, yeah, Invoke-Build by Roman! Are you at summit by any chance?

On Wed, 11 Apr 2018, 13:46 Gael Colas, gaelcolas@gmail.com wrote:

Yup, it's meant to go in DscBuildHelpers ;)

On Wed, 11 Apr 2018, 13:26 Raymond Piller, notifications@github.com wrote:

Since Datum is a generic hierarchical data store, should tools like Get-DscSplattedResource https://github.com/gaelcolas/Datum/blob/master/Datum/ScriptsToProcess/Get-DscSplattedResource.ps1 be broken out into another project/repo?

I've done some design changes to this function for use in a composite resource I've been working on, and I'm currently wanting to break out that function out into it's own repository https://github.com/UNT-CAS/OneDriveDsc/issues/13.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/gaelcolas/DscInfraSample/issues/8#issuecomment-380584669, or mute the thread https://github.com/notifications/unsubscribe-auth/AIjANXcxZZjqStTqHMJMop_k6a4oxZdgks5tnmcDgaJpZM4S5Wso .

VertigoRay commented 6 years ago

Unfortunately, I didn't know about it in time to get a them request submitted.