[RFC] Define a schema for the directory layout & a file naming convention

[lucdekens]

During the evolution of the project, the participants have stumbled on several situations where the initial folder layout, and the initial file naming conventions do not cover every possible situation they encounter.

This RFC shall attempt to discuss and define:

• the directory layout (under Source\VMware.vSphereDSC)

• file naming conventions to cope with all possible resources (including composite resources)

The final result of this RFC shall be included in the project documentation.

[lucdekens]

After some online discussions, we have the following proposal for composite resources.

DSCResources

• CompositeResource -- CompositeResource.schema.ps1 -- CompositeResource.psd1 -- Resource1.ps1 -- Resource2.ps1 -- Resource3.ps1

For the CompositeResourcename we propose to follow the names from vSphere Managed Inventory Objects .

This list includes the following composite resources: Cluster, (VM)Host, StandardSwitch, DistributedSwitch, StandardPortgroup, DistributedPortgroup, VM, VApp, Template ... (list to be evaluated and completed)

The individual resources (Resource1, Resource2...) should reflect which component from the composite resource they represent.

As an example, the Cluster resource

DSCResources

• Cluster -- Cluster.schema.ps1 -- Cluster.psd1 -- HACluster.ps1 -- DRSCluster.ps1

[lucdekens]

Which format and filetype shall we use to describe the folder layout?

[dmilov]

Hello,

I'm glad to see initiative for discussing project structure. Here are my thoughts from the code reviews I've made

1. Having classes, enums, and resources in separate folders under the root source folder is inconvenient to me.
2. Every class name has DSC suffix, I don't see value in this
3. Resources folder has DSC prefix which is not necessary IMO, it should be clear that under Source/VMware.vSphereDSC resources are DSC resuorces

My proposal is as follows

1. Create folder Source/VMware.vSphereDSC/Resources to hold all the code related to the resources
2. Create folder Source/VMware.vSphereDSC/Resources/Types to hold classes and enums
3. Remove DSC suffix of classes

[lucdekens]

To further the discussion, this is the current layout under Source\VMware.vSphereDSC

Classes
Configurations
    Ansible
    Chef
    PowerShell
        ESXiConfigs
            VMHostVss
            VMHostGraphics
            VMHostNetwork
                VMHostNic
                VMHostPhysicalNic
                VMHostVssPortgroup
        PowerCLISettingsConfigs
        vCenterConfigs
            Cluster
    Puppet
DSCResources
    Cluster
    VMHostGraphics
    VMHostNetwork
        VMHostPhysicalNic
        VMHostVss
        StandardSwitch
        VMHostVssPortgroup
Enums
    Cluster
    PowerCLISettingsConfigs
    VMHostVss
Tests
    Unit
        Cluster
        TestHelpers
            Mocks
            VMware.VimAutomation.Core
        VMHostGraphics
        VMHostNetwork
            VMHostVssPortgroup
        VMHostVss
    Integration
        Cluster
        Configurations
            Datacenter
            DatacenterFolder
            DrsCluster
            Folder
            HACluster
            <others>
        TestHelpers
        VMHostGraphics
        VMHostNetwork
            VMHostVssPortgroup
        VMHostVss

[lucdekens]

The same organisational issue comes back multiple times. How to organise 'simple' (by lack of a better name) and composite resources? This is currently used.

Cluster
VMHostGraphics
VMHostNetwork
    VMHostPhysicalNic
    VMHostVss
    StandardSwitch
    VMHostVssPortgroup

The following could be a working model.

<Optional_Subdivision> 
    <Composite_Resource_Name>
<Composite_Resource_Name>

For the Composite_Resource_Name we adopt the naming schema from vSphere Managed Inventory Objects.

The actual folders would then look something like this.

Cluster
VMHostGraphics
VMHostNetwork
    VirtualNic
    PhysicalNic
    StandardSwitch
    StandardPortgroup

For composite resources, the composite resource file (.ps1) and the schema will use the same name as the composite resource folder. Other components of the composite resource shall have a name that preferably clearly defines which component is in there.

As a practical example

Cluster
    Cluster.ps1
    Cluster.schema.psm1
    ClusterHA.ps1
    ClusterDRS.ps1

Note that the component resource filenames deviate from the current situation. I would propose to start each filename with the name of the composite resource, followed by the component name. I.e. ClusterHA.ps1 instead of HACluster.ps1.

[lucdekens]

As a side remark, I'm not 100% sure that vSphere Managed Inventory Objects is the best reference document for the naming.

Too many resources are not covered in there. For example, VSS, VMHostGraphics ...

[lucdekens]

2. Every class name has DSC suffix, I don't see value in this
3. Resources folder has DSC prefix which is not necessary IMO, it should be clear that under Source/VMware.vSphereDSC resources are DSC resuorces

@dmilov Shouldn't we try to follow the MSFT naming guidelines? See DSC Resource Naming Guidelines? I'm not sure if there is more recent (and more official) document on this.

If we do adopt your proposals , point 2 would be somewhat contradicting these guidelines.

Agreed on point 3.

[dmilov]

@lucdekens Oh, I wasn't aware of this naming convention for DSC Resources. Thanks for sharing!

Why do you think classes names contradict the convention? “Dsc” suffix is stated to be used in case the module delivers Functions and Resources. In this case it is said that resources should use "Dsc" suffix. point 2 is about the name of the base classes.

I'm not firmly against the suffix we can keep it if you think it's reasonable.

[lucdekens]

@dmilov It is a bit confusing which document currently is the actual guideline.

I also discovered DSC Resource Naming, which has a slightly different definition (as I read it). But that only seems to discuss the name of the module and the resources, not the file names.

That document also points to Developing a new resource, which shows a template folder layout.

The least we can conclude, this is a subject still under discussion :-)

[lucdekens]

On another note, and perhaps not completely related to this RFC, we should probably also try to follow the guidelines in Creating a High Quality DSC Resource Module.

At least the points concerning folder layout and file naming.

[lucdekens]

On a side note, what is the best way to describe a folder layout and a file naming convention?

Anyone know a tool or a language for this?

[gaelcolas]

Hi Folks. Thanks @lucdekens for pointing me to this, and I'll summon @PlagueHO & @johlju for more opinions.

We're trying to compile some of the info about the DSC resource kit & all here: https://dsccommunity.org and we'll eventually complete the move of Style guideline & naming convention. That said, current reference is indeed the HQRM guidelines you found.

You can also look at what we have in the DscResource.Template, although It might not go deep enough in areas and does not cover Class based resource yet (intentionally).

They won't all apply to you (and some missing) because Class based DSC Resource are not yet the standard we recommend (but I think we'll get there one day).

So with regarding to appending DSC to the all base classes names is not necessary, it's just recommended for the Resources name.

As a general guideline, my personal advice is to bear in mind the package containing DSC Resources is first and foremost a PowerShell Module. As such, it helps to keep a layout familiar to PowerShell module authors, and similarly, arrange the sources in a way that the community seem to be converging on. Not something we've been good at in the DSC Resource kit. For this reason, I'd prefer putting all my DSC Resources (and Composites) under a subfolder of that name (DSCResources, careful with the case here...).

That being said, I know of severe limitations doing so when using classes heavily.

Regarding the Composite resources, they should be in their own folder similarly to resources (ideally). This design may have some drawback when using Classes heavily (if you use nestedmodule and want to expose a class).

ModuleRootDsc
  +--ModuleRootDsc.psd1
  +--en-US/
  |  +--about_ModuleRoot.txt
  +--DSCResources/
    +--Resource1/
    |  +--en-US/
    |  |  +--Resource1.strings.psd1
    |  +--Resource1.Schema.mof
    |  +--Resource1.psm1
    +--ClassResource2/
    |  +--en-US/
    |  |  +--ClassResource2.strings.psd1
    |  +--ClassResource2.psm1
    |  +--ClassResource2.psd1
    +--Composite3/
       +--en-US/
       |  +--Composite3.strings.psd1
       +--Composite3.schema.psm1
       +--Composite3.psd1

[johlju]

I have not read the entire thread here, but just to point out that the reason for sticking each resource in a seperate folder under DSCResources (mof-based resource) or DSCClassResources (class-based resource) is to be able to localize the resource, to have the locale folder (e.g en-US) and its localized strings.psd1 files separate for each resource. Otherwise you end up with a bunch of strings for many resource in the same localized strings file.

The latest documentation around the folder and files structure used in the DSC Resource Kit is here https://github.com/PowerShell/DscResource.Template/blob/master/TEMPLATE_README.md. But it does not cover composites resources (but should in my opinion be treated as a mof-resource) and netiher the folder DSCClassResources. But the folder structure of DSCClassResources should look exactly like DSCResource if i would update the documentation. :)

[SimeonGerginov]

Well based on the discussion above, I propose to have the following structure for the VMware.vSphereDSC module:

1. For the Composite DSC Resources in the module:

VMware.vSphereDSC
  +--VMware.vSphereDSC.psd1
  +--VMware.vSphereDSC.psm1
  +--DSCResources/
    +--Cluster/
    |  +--Cluster.psd1
    |  +--Cluster.schema.psm1
    |  +--HACluster.ps1
    |  +--DrsCluster.ps1

And the same structure will go for the other Composite DSC Resources in the module like StandardSwitch, StandardPortGroup, etc. All DSC Resources that are part of the Composite DSC Resource should be in the same folder. Here we don't have a lot of options because the Composite DSC Resources can only be found by the Get-DscResource cmdlet if they are located in a folder below the DSCResources folder of the module where the folder should have the same name as the Composite DSC Resource.

2. For the classes and enums in the module:

VMware.vSphereDSC
  +--VMware.vSphereDSC.psd1
  +--VMware.vSphereDSC.psm1
  +--Types/
    +--Classes/
    |  +--Base.ps1
    |  +--VMHostBaseDSC.ps1
    |  ...
    +--Enums/
    |  +--EntityType.ps1
    |  ...

3. For the DSC Resources in the module:

VMware.vSphereDSC
  +--VMware.vSphereDSC.psd1
  +--VMware.vSphereDSC.psm1
  +--DSCResources/
    +--Network/
    |  +--VDSwitch.ps1
    |  ...
    +--Storage/
    |  +--VmfsDatastore.ps1
    |  ...
    +--VMHost/
    |  +--VMHostRole.ps1
    |  +--VMHostPermission.ps1
    |  ...
    +--VCenterInventory/
    |  +--Datacenter.ps1
    |  +--Folder.ps1
    |  ...

4. The Unit Tests should follow the directory structure of the DSC Resources and for every DSC Resource:

VMware.vSphereDSC
  +--VMware.vSphereDSC.psd1
  +--VMware.vSphereDSC.psm1
  +--Tests/
    +--Unit/
      +--VCenterInventory/
        +--DatastoreCluster/
        |  +--DatastoreCluster.Unit.Tests.ps1
        |  +--DatastoreCluster.Mocks.ps1
        |  +--DatastoreCluster.Mocks.Data.ps1

5. The Integration Tests should follow the directory structure of the DSC Resources and for every DSC Resource:

VMware.vSphereDSC
  +--VMware.vSphereDSC.psd1
  +--VMware.vSphereDSC.psm1
  +--Tests/
    +--Integration/
      +--VCenterInventory/
        +--DatastoreCluster/
        |  +--DatastoreCluster.Integration.Tests.ps1
        |  +--DatastoreCluster.Integration.Tests.Helpers..ps1
        |  +--DatastoreCluster_Config.ps1

Module structure:

VMware.vSphereDSC
  +--VMware.vSphereDSC.psd1
  +--VMware.vSphereDSC.psm1
  +--Configurations/
  +--DSCResources/
  +--Tests/
    +--Unit/
    +--Integration/
  +--Types/

[gaelcolas]

Hi @SimeonGerginov, After more experiments I have changed my approach a fair bit. I've found that the most solid pattern to have Class Based DSC Resources is to merge them in the root Module's PSM1. This can be done at compile time (i.e. using ModuleBuilder, as the DSC Community is doing through Sampler). Using other PSM1 and the "nestedModules" approach is not great, as it's severely limits how you can use the classes you create, and other ways can make discovery a bit more painful too.

I'd obviously prefer for your project to be a bit more aligned to what the DSC community does, so I'll just summarize the main things we do differently from your proposition and why.

We separate the source from other bits of code that won't end directly in the module (CI files and configurations). We don't use many DSC classes yet, but we've started to experiment and discuss (Bartek Bielawski will do a presentation about his findings during next Community call, the 20th of May). I'd put them in a Classes Source folder as per below (note that it's not under DSCResources folder which is here only for composites or MOF-based as you noted above). Then the compilation step of our pipeline takes every ps1 files from the folders Enum, Classes, Private, Public and merge them into a VMware.vSphereDSC.psm1 (in the output folder). DSCResources and other folders are copied as-is without transformation.

So the git repo would look like this:

VMware.vSphereDSC
  +--build.ps1
  +--build.yml
  +--azure-pipelines.yml
  +--GitVersion.yml
  +-- ...
  +--source/
    +--VMware.vSphereDSC.psd1
    +--DSCResources/
    +--Classes/
    |  +--Base.ps1
    |  +--VMHostBaseDSC.ps1
    |  ...
    +--Enums/
    |  +--EntityType.ps1
    |  ...

While the output would look like (same folder but ignored by git):

VMware.vSphereDSC
  +--source/ # content not shown
  +--output/
    +--VMware.vSphereDSC/ # artefact built from the sources.
    |  +--VMware.vSphereDSC.psd1
    |  +--VMware.vSphereDSC.psm1 # all classes in there
    |  +--DSCResources/ # for composites only
    +--RequiredModules/
    |  +--Pester/
    |  ... #all modules required during the build steps

Finally, I've made a video a while ago to show the principle of Sampler (the template and build library we use): https://www.youtube.com/watch?v=UWGKgVB4VNU It evolved a fair bit from this, but the principle remains the same. It allows us to improve the build process of many modules by just using this as a dependency we can improve asynchronously from the repositories using it. In case of issue, those repositories can still pin an older version.

Happy to discuss this on Slack, I'm there most days.

[lucdekens]

@SimeonGerginov in your point 1, you didn't mention the schema part in the composite resource .psm1 file. That could perhaps lead to misinterpreting that these are 'real' nested modules.

Related to that, @gaelcolas dumping all classes into the module's .psm1 file in the Output folder is not really a big deal. But how would you organise the composite resource schema files? These are not nested modules.

[gaelcolas]

Yeah composites or MOF based resources would still go in a DSCResources folder.