Azure Pipelines build and release pipeline

[ethomson]

At present, we have a mix of build systems that I would like to unify.

1. Azure Pipelines for Linux, Windows and macOS.
2. Travis for Linux and macOS
3. Appveyor for Windows

I'd like to standardize around Azure Pipelines. At the moment, it's only doing a build for validation. I would like the build step to:

1. Create a nuget package (see AppVeyor's build configuration)
2. Upload that nuget package to Azure Artifacts

Then I'd like to have a separate release pipeline that will:

1. Take the nuget package from Azure Artifacts
2. Upload the nuget package to nuget.org

[bording]

That sounds great to me! No need for three systems when one can handle everything.

[simondel]

I'd like to do this :)

Would you still like to keep your cmd and sh scripts to build and test or shall I replace that with yml steps? If I use the yml steps I can also upload the test results (with code coverage) to the build summary.

Also, what do you want to do with Coverity and Coveralls? The use for Coveralls could go away since the Azure Pipeline would be able to contain your code coverage. Coverity could (if you like) be replaced with SonarCloud.

You mentioned that you would like to upload the results of the build to Azure Artifacts, why would you want this instead of publishing them as regular build artifacts and using those in your release definition?

EDIT: You can view my progress on my Azure DevOps project. For now I'm using the visual editor but I'll export it to a yml pipeline once I'm done.

[simondel]

How are the paths set during testing? I'm running into the issue that the tests pretty much run on no platform.

For example, on Windows I'm getting issues like:

System.IO.DirectoryNotFoundException : Could not find a part of the path 'D:\a\1\LibGit2Sharp.Tests\Resources\testrepo_wd'.

While the file is actually in

'D:\a\1\s\LibGit2Sharp.Tests\Resources\testrepo_wd'.

I have the same issue if I manually run the tests on my machine.

Also, what's up with the extradefine property?

[ethomson]

How are the paths set during testing? I'm running into the issue that the tests pretty much run on no platform.

Sorry, in what context? How are you running them? They run fine for me locally (via xunit runner) and in AppVeyor, Travis and Azure Pipelines.

The existing Azure Pipelines build is here, which reads the existing azure-pipelines.yml to do a build.

This should be a good start. Curious what's different about your setup from the one that's in place?

[simondel]

Sorry for the slow response! I execute commands using the Azure DevOps tasks instead of running the current script. The script is in my opinion not the preferred way of building/testing because the integration with Azure DevOps is very limited.

Right now there are a lot of tests that are failing: https://dev.azure.com/simondel/Public/_build/results?buildId=147&_a=summary&view=ms.vss-test-web.test-result-details

On Windows a lot of Unix related tests seem to run (and fail).

On Linux the following command is executed:

/usr/bin/dotnet test /home/vsts/work/1/s/LibGit2Sharp.Tests/LibGit2Sharp.Tests.csproj -f netcoreapp2.0 --no-restore --no-build --logger trx --results-directory /home/vsts/work/_temp

And that's causing all tests to fail such as:

2018-10-30T13:30:45.2573042Z Failed   LibGit2Sharp.Tests.CleanFixture.CannotCleanABareRepository
2018-10-30T13:30:45.2576813Z Error Message:
2018-10-30T13:30:45.2580845Z  System.IO.DirectoryNotFoundException : Could not find a part of the path '/home/vsts/work/1/LibGit2Sharp.Tests/Resources/testrepo.git'.
2018-10-30T13:30:45.2583722Z Stack Trace:
2018-10-30T13:30:45.2585691Z    at System.IO.UnixFileSystem.FileSystemEnumerable`1.OpenDirectory(String fullPath)
2018-10-30T13:30:45.2586139Z    at System.IO.UnixFileSystem.FileSystemEnumerable`1.Enumerate()
2018-10-30T13:30:45.2587859Z    at System.IO.UnixFileSystem.FileSystemEnumerable`1..ctor(String userPath, String searchPattern, SearchOption searchOption, SearchTarget searchTarget, Func`3 translateResult)
2018-10-30T13:30:45.2588335Z    at System.IO.UnixFileSystem.EnumerateFileSystemInfos(String fullPath, String searchPattern, SearchOption searchOption, SearchTarget searchTarget)
2018-10-30T13:30:45.2590465Z    at System.IO.DirectoryInfo.InternalGetDirectories(String searchPattern, SearchOption searchOption)
2018-10-30T13:30:45.2590834Z    at System.IO.DirectoryInfo.GetDirectories()
2018-10-30T13:30:45.2592539Z    at LibGit2Sharp.Tests.TestHelpers.DirectoryHelper.CopyFilesRecursively(DirectoryInfo source, DirectoryInfo target) in /home/vsts/work/1/s/LibGit2Sharp.Tests/TestHelpers/DirectoryHelper.cs:line 25
2018-10-30T13:30:45.2592975Z    at LibGit2Sharp.Tests.TestHelpers.BaseFixture.Sandbox(String sourceDirectoryPath, String[] additionalSourcePaths) in /home/vsts/work/1/s/LibGit2Sharp.Tests/TestHelpers/BaseFixture.cs:line 234
2018-10-30T13:30:45.2594716Z    at LibGit2Sharp.Tests.TestHelpers.BaseFixture.SandboxBareTestRepo() in /home/vsts/work/1/s/LibGit2Sharp.Tests/TestHelpers/BaseFixture.cs:line 177
2018-10-30T13:30:45.2597028Z    at LibGit2Sharp.Tests.CleanFixture.CannotCleanABareRepository() in /home/vsts/work/1/s/LibGit2Sharp.Tests/CleanFixture.cs:line 30

I would expect that a dotnet test of the test csproj would work, but it looks like it doesn't. If you look in the error above, it's looking in the incorrect folder (it ignores the s folder)

[ethomson]

The script is in my opinion not the preferred way of building/testing because the integration with Azure DevOps is very limited.

Being able to run the scripts locally is a huge help in creating a reproducible CI system - debugging failures on a CI server is hard. If you can run the same script locally that the CI server runs, it's a lot easier. Obviously things like uploading test results will need to be a unique task, but using the "build" and "run my test" task is not necessarily a big win.

For libgit2 itself, we have a straightforward setup: run the build script, then run the test script. Additional steps (like the test results upload) run the Azure Pipelines tasks, but the core building happens in a script so that it's easy to understand.

In any case:

I would expect that a dotnet test of the test csproj would work, but it looks like it doesn't. If you look in the error above, it's looking in the incorrect folder (it ignores the s folder)

What's are you setting the current working directory to when you start this process? Can you share your yaml?

[simondel]

I'm not setting the current working directory. It's using the default value (for example: c:\agent_work\1\s). I understand that you want the same experience locally as on your build server, but I think it's odd that the 'regular' commands to restore, build and test the project don't work. They also don't work on my own machine. (workingdirectory = root of the repo on my disk)

My yaml file (generated) is:

resources:
- repo: self
phases:
- phase: Phase_1
  displayName: Windows

  condition: succeeded()
  queue:
    name: Hosted VS2017
    demands: 
   - msbuild
   - visualstudio
   - vstest

  steps:
  - task: NuGetToolInstaller@0
    displayName: 'Use NuGet 4.7.1'
    inputs:
      versionSpec: 4.7.1

  - task: NuGetCommand@2
    displayName: 'NuGet restore'

  - task: VSBuild@1
    displayName: 'Build solution **\*.sln'

  - task: VSTest@2
    displayName: 'VsTest - testAssemblies'

  - task: DotNetCoreCLI@2
    displayName: 'dotnet build'
    inputs:
      projects: '**/*.sln'

    enabled: false

  - task: DotNetCoreCLI@2
    displayName: 'dotnet test'
    inputs:
      command: test

      projects: '**/*Tests.csproj'

    enabled: false

- phase: Phase_2
  displayName: Linux

  condition: succeeded()
  queue:
    name: Hosted Ubuntu 1604
  steps:
  - task: DotNetCoreCLI@2
    displayName: 'dotnet restore'
    inputs:
      command: restore

  - task: DotNetCoreCLI@2
    displayName: 'dotnet build'
    inputs:
      projects: LibGit2Sharp.Tests

      arguments: ' -f netcoreapp2.0 /property:ExtraDefine="LEAKS_IDENTIFYING" /fl /flp:verbosity=detailed'

  - task: DotNetCoreCLI@2
    displayName: 'dotnet test'
    inputs:
      command: test

      projects: LibGit2Sharp.Tests/LibGit2Sharp.Tests.csproj

      arguments: '-f netcoreapp2.0 --no-restore --no-build  '

- phase: Phase_3
  displayName: Mac

  condition: succeeded()
  queue:
    name: Hosted macOS
  steps:
  - task: DotNetCoreCLI@2
    displayName: 'dotnet build'
    inputs:
      projects: '**/*.sln'

  - task: DotNetCoreCLI@2
    displayName: 'dotnet test'
    inputs:
      command: test

      projects: '**/*Tests.csproj'

[ethomson]

They also don't work on my own machine. (workingdirectory = root of the repo on my disk)

Do they work if you set the Configuration=release environment variable? We already have a working Azure Pipelines configuration checked in, to do the build and release. Offhand, I don't know why that environment variable would affect things, but it certainly does.

I agree that this should work correctly "out of the box" - PRs welcome.

[ethomson]

I think that @lupino3 fixed the test execution (finding resources) for you in #1630.

[simondel]

With the latest changes, it goes a lot better already :) I'll take a look at the two tests that are still failing on Windows (build log: https://dev.azure.com/simondel/Public/_build/results?buildId=150&_a=summary&view=logs)