Determine Scope for Refactoring Repository Layout

[markcowl]

Description

We plan to split repository into 3 pieces:

(1) Common code, Profile, and Tools (2) ResourceManager (3) ServiceManagement

Prototype part of this modeling using git submodules or git subtrees, decide on the best mechanism, roughly size the work to do this and file issues.

Cost: 10

[MiYanni]

Preliminary Investigation

Someone made another option called git-subrepo that tried to resolve the issues with git-submodule and git-subtree. I think it is worth a try to see if collaborators would need to install this to develop on our repos. https://github.com/ingydotnet/git-subrepo

Here is a good doc on git-subtree: https://www.atlassian.com/blog/git/alternatives-to-git-submodule-git-subtree

A basic blog post on git-submodule: https://github.com/blog/2104-working-with-submodules Lessons learned post: https://blogs.ancestry.com/ancestry/2015/02/26/lesson-learned-sharing-code-with-git-submodule/

An alternative idea is that we could simply make the AzPsCommon repo [working title] a separate repo entirely. We have it shared as a NuGet package, and AzPsRm and AzPsSm get the AzPsCommon package from NuGet. This blog post has some information related to our situation, and they gave up with trying to use git-subtree and simply used a NuGet feed: http://chrisoldwood.blogspot.com/2016/08/sharing-code-with-git-subtree.html I'm leaning toward this idea since it is easiest for our RPs to use. No crazy Git commands involved. And, we can control when to update the dependency on our AzPsCommon for the solutions in AzPsRm and AzPsSm.

[MiYanni]

Primary Concept

Separation

• 3 different repos
• one for modules (azure-powershell), one for common (azure-powershell-common), and one for service management (azure-powershell-servicemanagement)
• https://softwareengineering.stackexchange.com/a/263787/108631
• Service teams only work in azure-powershell. We (scottphers 😉) work within azure-powershell and azure-powershell-common, primarily. azure-powershell-servicemangement is in maintenance mode and hopefully does not need changes beyond the re-release after this separation process.
• azure-powershell-common is to simplify dependencies for the modules. This package is released on a separate cadence than azure-powershell. Module projects simply reference the NuGet package produced by azure-powershell-common. This allows us to manage common references and infrastructure code. All issues in this repo are for our team.
• azure-powershell has all modules. We use this repo to communicate with external customers, have service teams produce/update modules, and use it basically the same way we do right now. Nearly all issues in this repo are for the service teams. The modules can be separated by DataPlane and Management if need be.
• azure-powershell-servicemanagement only remains to have a codebase for servicemanagement modules. This will be self-contained, and not rely on azure-powershell or azure-powershell-common in any way. It will be in maintenance mode.

Issue Resolution/Simplification

• Removes depth in folder structure
  • Each repo has src and has the project folders under that (or possibly a categorization, then projects). No need to have the current elaborate depth we currently use.
  • We were hitting the depth limit in producing the packages. We will flatten this significantly with this strategy.
• Allows module solution files to only contain projects necessary for the module
  • In most cases, it will contain 1 main project and 1 test project. Currently, most solutions have 12-15 projects for the common projects that are required.
• Reduces build time
  • Common components are precompiled since they are acquired from NuGet
• Service teams clone less code
  • They'll only clone the module code and no common/service management code. They'll have what they need to build modules, which is what they're concerned with.
• Modules have their own solutions and we automatically create the solution for CI build
  • No need to have an 'umbrella' solution in source control. We can generate this as part of CI.
  • azure-powershell would only contain module solutions. Very straightforward.

[maddieclayton]

Please include this issue in the creation of the service management repo: https://github.com/Azure/azure-powershell/issues/6379

[MiYanni]

Here is the basics of restructuring the repo layout. There is definitely more to be fleshed out as the work is being done, but this is a good starting point.

In terms of work items,

• Common (2 items linked inside): https://github.com/Azure/azure-powershell/issues/6104
• RM: https://github.com/Azure/azure-powershell/issues/6395
• SM: https://github.com/Azure/azure-powershell/issues/6396

Repository Structures

Based on:

• https://gist.github.com/davidfowl/ed7564297c61fe9ab814
• http://nvie.com/posts/a-successful-git-branching-model/#the-main-branches

Other references:

• https://github.com/kriasoft/Folder-Structure-Conventions
• https://github.com/fsprojects/ProjectScaffold
• https://github.com/dotnet/docs/blob/master/docs/core/porting/project-structure.md

General

• Stack being created as a separate branch
• DataPlane modules will merge into base modules (in the future)
• Change default branch to dev
• Investigate .lock files

azure-powershell

Folders

• .github [*.md - Templates (Issue, PR, etc.)]
• build
  • installer [Was 'setup' before]
  • build.proj
  • *.targets files
• src
  • Aks
  • Aks
    • bin
    • obj
    • Properties
  • Aks.Test
  • Aks.sln
  • CHANGELOG.md
  • README.md
  • BREAKINGCHANGES.md
  • AnalysisServices
  • AnalysisServices.DataPlane [Moved and renamed]
  • ApiManagement
  • ApplicationInsights
  • Automation
  • Backup [Renamed from 'AzureBackup']
  • Batch [Renamed from 'AzureBatch']
  • Billing
  • Cdn
  • CognitiveServices
  • Compute
  • Compute.ManagedService
  • Consumption
  • ContainerInstance
  • ContainerRegistry
  • DataFactories [v Rename or merge into 1 module? v]
  • DataFactoryV2 [^ Rename or merge into 1 module? ^]
  • DataLakeAnalytics
  • DataLakeStore
  • DataMigration
  • DeviceProvisioningServices
  • DevTestLabs
  • Dns
  • EventGrid
  • EventHub
  • HDInsight
  • Insights
  • IotHub
  • KeyVault
  • LocationBasedServices
  • LogicApp
  • MachineLearning
  • MachineLearningCompute [Change to 'MachineLearning.Compute'??]
  • ManagedServiceIdentity
  • ManagementPartner
  • MarketplaceOrdering
  • Media
  • Network
  • NotificationHubs
  • OperationalInsights
  • PolicyInsights
  • PowerBIEmbedded
  • Profile
  • RecoveryServices
  • RecoveryServices.Backup
  • RecoveryServices.SiteRecovery
  • RedisCache
  • Relay
  • Reservations
  • Resources
  • Scheduler
  • ServiceBus
  • ServiceFabric
  • Sql
  • Storage
  • Storage.DataPlane [Moved and renamed]
  • StreamAnalytics
  • Subscription [Should be merged with Profile???]
  • Tags
  • TrafficManager
  • UsageAggregates
  • Websites
• docs [Was 'documentation']
• local-packages [Was 'LocalFeed']
  • README.md
• samples [Samples geared towards partners]
• tools
  • Module tools
  • May be used on-build
  • May be moved to separate repo in future

Local Folders [Not in git repo]

• .git
• .vs
• packages
• artifacts [Output of build.proj - Replaces 'Package']

Files

• .dockerignore
• .gitattributes
• .gitignore
• .travis.yml
• appveyor.yml [Remove this]
• Azure.PowerShell.sln
• Azure.PowerShell.sln.DotSettings [Team-shared Resharper project settings]
• Azure.PowerShell.Test.sln
• Azure.PowerShell.Test.sln.DotSettings [Team-shared Resharper project settings]
• Dockerfile [What is this for?]
  • If keeping, move to 'build' folder
• LICENSE.txt
• nuget.config
  • Note: This is the preferred casing
• packages.config [Remove this]
• README.md

Local Files [Not in git repo]

• Azure.PowerShell.sln.DotSettings.user [User-specific Resharper project settings]
• Azure.PowerShell.Test.sln.DotSettings.user [User-specific Resharper project settings]

azure-powershell-common

Folders

• artifacts [Output of build.proj]
• build
  • build.proj
• src
• docs

azure-powershell-servicemanagement

• AFAIK, preserve as much of the layout as possible since it is not worth the effort to restructure this repo