search

googleforgames / agones

Dedicated Game Server Hosting and Scaling for Multiplayer Games on Kubernetes
https://agones.dev
Apache License 2.0
6.01k stars 796 forks source link

C# SDK build is flakey due to a race condition #1639

Closed pooneh-m closed 4 years ago

pooneh-m commented 4 years ago

What happened: make test sometimes fails with error:

 error MSB4018: The "GenerateDepsFile" task failed unexpectedly. [/go/src/agones.dev/agones/sdks/csharp/sdk/csharp-sdk.csproj]
/usr/share/dotnet/sdk/2.2.402/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(129,5): error MSB4018: System.IO.IOException: The process cannot access the file '/go/src/agones.dev/agones/sdks/csharp/sdk/bin/Release/netstandard2.0/AgonesSDK.deps.json' because it is being used by another process. [/go/src/agones.dev/agones/sdks/csharp/sdk/csharp-sdk.csproj]
/usr/share/dotnet/sdk/2.2.402/Sdks/Microsoft.NET.Sdk/targets/Microsoft.NET.Sdk.targets(129,5): error MSB4018:    at System.IO.FileStream.Init(FileMode mode, FileShare share) [/go/src/agones.dev/agones/sdks/csharp/sdk/csharp-sdk.csproj]
...

Anything else we need to know?:

Environment:

This happens on the cloud build for pull requests.

markmandel commented 4 years ago

@rcreasey do you have any idea why this is happening? It's a pretty frequent flake unfortunately :frowning:

markmandel commented 4 years ago

Full build example: https://console.cloud.google.com/cloud-build/builds/8ebf0627-bede-4ae7-adc4-24dacacf131d?project=258182270954

I think this is a full log:

docker build --tag=agones-build-sdk-csharp:8ad37ed9c6 /workspace/build/build-sdk-images/csharp --build-arg BASE_IMAGE=agones-build-sdk-base:48cd8562c3
Sending build context to Docker daemon  8.192kB
Step 1/8 : ARG BASE_IMAGE=agones-build-sdk-base:latest
Step 2/8 : FROM $BASE_IMAGE
 ---> 432b721aac95
Step 3/8 : RUN  apt-get update     && apt-get install -y wget
 ---> Running in f386e7684737
Hit:1 http://deb.debian.org/debian buster InRelease
Get:2 http://deb.debian.org/debian buster-updates InRelease [51.9 kB]
Get:3 http://security.debian.org/debian-security buster/updates InRelease [65.4 kB]
Get:4 http://deb.debian.org/debian buster-updates/main amd64 Packages.diff/Index [3688 B]
Get:5 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-07-1403.53.pdiff [439 B]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-13-2000.26.pdiff [552 B]
Get:6 http://deb.debian.org/debian buster-updates/main amd64 Packages 2020-06-13-2000.26.pdiff [552 B]
Get:7 http://security.debian.org/debian-security buster/updates/main amd64 Packages [204 kB]
Fetched 326 kB in 2s (176 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  wget
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 902 kB of archives.
After this operation, 3335 kB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 wget amd64 1.20.1-1.1 [902 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 902 kB in 0s (7866 kB/s)
Selecting previously unselected package wget.
(Reading database ... 
Preparing to unpack .../wget_1.20.1-1.1_amd64.deb ...
Unpacking wget (1.20.1-1.1) ...
Setting up wget (1.20.1-1.1) ...
Removing intermediate container f386e7684737
 ---> d806f7c140b4
Step 4/8 : RUN apt-get update     && apt-get install -y apt-transport-https
 ---> Running in 3cf91c208c0f
Hit:1 http://security.debian.org/debian-security buster/updates InRelease
Hit:2 http://deb.debian.org/debian buster InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following NEW packages will be installed:
  apt-transport-https
0 upgraded, 1 newly installed, 0 to remove and 3 not upgraded.
Need to get 149 kB of archives.
After this operation, 156 kB of additional disk space will be used.
Get:1 http://security.debian.org/debian-security buster/updates/main amd64 apt-transport-https all 1.8.2.1 [149 kB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 149 kB in 0s (2607 kB/s)
Selecting previously unselected package apt-transport-https.
(Reading database ... 
Preparing to unpack .../apt-transport-https_1.8.2.1_all.deb ...
Unpacking apt-transport-https (1.8.2.1) ...
Setting up apt-transport-https (1.8.2.1) ...
Removing intermediate container 3cf91c208c0f
 ---> 6b233bae866b
Step 5/8 : RUN wget -q https://packages.microsoft.com/config/ubuntu/19.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb     && dpkg -i packages-microsoft-prod.deb     && wget -q https://packages.microsoft.com/config/ubuntu/18.04/packages-microsoft-prod.deb -O packages-microsoft-prod.deb     && dpkg -i packages-microsoft-prod.deb
 ---> Running in a607199cf9e9
Selecting previously unselected package packages-microsoft-prod.
(Reading database ... 17713 files and directories currently installed.)
Preparing to unpack packages-microsoft-prod.deb ...
Unpacking packages-microsoft-prod (1.0-ubuntu19.04.1) ...
Setting up packages-microsoft-prod (1.0-ubuntu19.04.1) ...
dpkg: warning: downgrading packages-microsoft-prod from 1.0-ubuntu19.04.1 to 1.0-ubuntu18.04.2
(Reading database ... 17721 files and directories currently installed.)
Preparing to unpack packages-microsoft-prod.deb ...
Unpacking packages-microsoft-prod (1.0-ubuntu18.04.2) over (1.0-ubuntu19.04.1) ...
Setting up packages-microsoft-prod (1.0-ubuntu18.04.2) ...
Installing new version of config file /etc/apt/sources.list.d/microsoft-prod.list ...
Removing intermediate container a607199cf9e9
 ---> 6c3139b9a9f8
Step 6/8 : RUN apt-get update     && apt-get install -y dotnet-sdk-2.2
 ---> Running in 3b955844eeef
Hit:1 http://security.debian.org/debian-security buster/updates InRelease
Hit:2 http://deb.debian.org/debian buster InRelease
Hit:3 http://deb.debian.org/debian buster-updates InRelease
Get:4 https://packages.microsoft.com/ubuntu/18.04/prod bionic InRelease [4003 B]
Get:5 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 Packages [118 kB]
Fetched 122 kB in 2s (50.0 kB/s)
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
The following additional packages will be installed:
  aspnetcore-runtime-2.2 dotnet-host dotnet-hostfxr-2.2 dotnet-runtime-2.2
  dotnet-runtime-deps-2.2 liblttng-ust-ctl4 liblttng-ust0 liburcu6
The following NEW packages will be installed:
  aspnetcore-runtime-2.2 dotnet-host dotnet-hostfxr-2.2 dotnet-runtime-2.2
  dotnet-runtime-deps-2.2 dotnet-sdk-2.2 liblttng-ust-ctl4 liblttng-ust0
  liburcu6
0 upgraded, 9 newly installed, 0 to remove and 3 not upgraded.
Need to get 136 MB of archives.
After this operation, 400 MB of additional disk space will be used.
Get:1 http://deb.debian.org/debian buster/main amd64 liburcu6 amd64 0.10.2-1 [66.4 kB]
Get:2 http://deb.debian.org/debian buster/main amd64 liblttng-ust-ctl4 amd64 2.10.3-1 [105 kB]
Get:3 http://deb.debian.org/debian buster/main amd64 liblttng-ust0 amd64 2.10.3-1 [177 kB]
Get:4 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 dotnet-runtime-deps-2.2 amd64 2.2.8-1 [2664 B]
Get:5 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 dotnet-host amd64 3.1.5-1 [32.8 kB]
Get:6 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 dotnet-hostfxr-2.2 amd64 2.2.8-1 [143 kB]
Get:7 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 dotnet-runtime-2.2 amd64 2.2.8-1 [20.7 MB]
Get:8 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 aspnetcore-runtime-2.2 amd64 2.2.8-1 [22.9 MB]
Get:9 https://packages.microsoft.com/ubuntu/18.04/prod bionic/main amd64 dotnet-sdk-2.2 amd64 2.2.402-1 [91.5 MB]
debconf: delaying package configuration, since apt-utils is not installed
Fetched 136 MB in 7s (20.7 MB/s)
Selecting previously unselected package liburcu6:amd64.
(Reading database ... 
Preparing to unpack .../0-liburcu6_0.10.2-1_amd64.deb ...
Unpacking liburcu6:amd64 (0.10.2-1) ...
Selecting previously unselected package liblttng-ust-ctl4:amd64.
Preparing to unpack .../1-liblttng-ust-ctl4_2.10.3-1_amd64.deb ...
Unpacking liblttng-ust-ctl4:amd64 (2.10.3-1) ...
Selecting previously unselected package liblttng-ust0:amd64.
Preparing to unpack .../2-liblttng-ust0_2.10.3-1_amd64.deb ...
Unpacking liblttng-ust0:amd64 (2.10.3-1) ...
Selecting previously unselected package dotnet-runtime-deps-2.2.
Preparing to unpack .../3-dotnet-runtime-deps-2.2_2.2.8-1_amd64.deb ...
Unpacking dotnet-runtime-deps-2.2 (2.2.8-1) ...
Selecting previously unselected package dotnet-host.
Preparing to unpack .../4-dotnet-host_3.1.5-1_amd64.deb ...
Unpacking dotnet-host (3.1.5-1) ...
Selecting previously unselected package dotnet-hostfxr-2.2.
Preparing to unpack .../5-dotnet-hostfxr-2.2_2.2.8-1_amd64.deb ...
Unpacking dotnet-hostfxr-2.2 (2.2.8-1) ...
Selecting previously unselected package dotnet-runtime-2.2.
Preparing to unpack .../6-dotnet-runtime-2.2_2.2.8-1_amd64.deb ...
Unpacking dotnet-runtime-2.2 (2.2.8-1) ...
Selecting previously unselected package aspnetcore-runtime-2.2.
Preparing to unpack .../7-aspnetcore-runtime-2.2_2.2.8-1_amd64.deb ...
Unpacking aspnetcore-runtime-2.2 (2.2.8-1) ...
Selecting previously unselected package dotnet-sdk-2.2.
Preparing to unpack .../8-dotnet-sdk-2.2_2.2.402-1_amd64.deb ...
Unpacking dotnet-sdk-2.2 (2.2.402-1) ...
Setting up dotnet-host (3.1.5-1) ...
Setting up dotnet-hostfxr-2.2 (2.2.8-1) ...
Setting up liburcu6:amd64 (0.10.2-1) ...
Setting up liblttng-ust-ctl4:amd64 (2.10.3-1) ...
Setting up liblttng-ust0:amd64 (2.10.3-1) ...
Setting up dotnet-runtime-deps-2.2 (2.2.8-1) ...
Setting up dotnet-runtime-2.2 (2.2.8-1) ...
Setting up aspnetcore-runtime-2.2 (2.2.8-1) ...
Setting up dotnet-sdk-2.2 (2.2.402-1) ...
This software may collect information about you and your use of the software, and send that to Microsoft.
Please visit http://aka.ms/dotnet-cli-eula for more information.
Welcome to .NET Core!
---------------------
Learn more about .NET Core: https://aka.ms/dotnet-docs
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs

Telemetry
---------
The .NET Core tools collect usage data in order to help us improve your experience. The data is anonymous and doesn't include command-line arguments. The data is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

Configuring...
--------------
A command is running to populate your local package cache to improve restore speed and enable offline access. This command takes up to one minute to complete and only runs once.
Processing triggers for libc-bin (2.28-10) ...
Removing intermediate container 3b955844eeef
 ---> 5ec7d0693832
Step 7/8 : COPY *.sh /root/
 ---> 8dac7af11c36
Step 8/8 : RUN chmod +x /root/*.sh
 ---> Running in 56ed703a3a09
Removing intermediate container 56ed703a3a09
 ---> d1f353712e08
Successfully built d1f353712e08
Successfully tagged agones-build-sdk-csharp:8ad37ed9c6
make[4]: Leaving directory '/workspace/build'
+ /root/build.sh
+ cd ./sdks/csharp/sdk
+ dotnet publish -c Release

Welcome to .NET Core!
---------------------
Learn more about .NET Core: https://aka.ms/dotnet-docs
Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs

Telemetry
---------
The .NET Core tools collect usage data in order to help us improve your experience. The data is anonymous and doesn't include command-line arguments. The data is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell.

Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry

ASP.NET Core
------------
Successfully installed the ASP.NET Core HTTPS Development Certificate.
To trust the certificate run 'dotnet dev-certs https --trust' (Windows and macOS only). For establishing trust on other platforms refer to the platform specific documentation.
For more information on configuring HTTPS see https://go.microsoft.com/fwlink/?linkid=848054.
Microsoft (R) Build Engine version 16.2.32702+c4012a063 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

/usr/share/dotnet/sdk/2.2.402/NuGet.targets(123,5): error : The file '/go/src/agones.dev/agones/sdks/csharp/sdk/obj/csharp-sdk.csproj.nuget.g.props' already exists. [/go/src/agones.dev/agones/sdks/csharp/sdk/csharp-sdk.csproj]
make[1]: *** [run-sdk-command] Error 1
includes/sdk.mk:88: recipe for target 'run-sdk-command' failed
includes/sdk.mk:84: recipe for target 'run-sdk-command-csharp' failed
make: *** [run-sdk-command-csharp] Error 2
rcreasey commented 4 years ago

Hrm, it could be related to https://github.com/NuGet/Home/issues/6159

https://github.com/googleforgames/agones/blob/master/sdks/csharp/test/csharp-sdk-test.csproj#L25

    <ProjectReference Include="..\sdk\csharp-sdk.csproj" />
markmandel commented 4 years ago

Hrm, it could be related to NuGet/Home#6159

What does that do? :smile:

markmandel commented 4 years ago

Oh wait, I see it - it's \ when it should be / ?

markmandel commented 4 years ago

I tried it in #1650 ... and it still happened. :disappointed:

markmandel commented 4 years ago

I'm looking at the cloudbuild.yaml, and I'm wondering - if the csharp build process is run at ~ the same time as the test process, could they collide?

- name: "make-docker"
  id: build-sdks
  waitFor:
    - lint
    - cpp-sdk-build-restore-cache
    - ensure-build-sdk-image-base
  dir: "build"
  args: [ "-j", "4", "--output-sync=target", "build-sdks"]

#
# Run the all the automated tests (except e2e) in parallel
#

- name: "make-docker"
  id: tests
  waitFor:
    - lint
    - ensure-build-sdk-image-base
    - htmltest-restore-cache
  dir: "build"
  args: [ "-j", "5", "--output-sync=target", "test" ]

I wonder if we make test dependent on build-sdks that will solve the issue. I'll give it a shot in a draft PR.