search

dotnet / tye

Tye is a tool that makes developing, testing, and deploying microservices and distributed applications easier. Project Tye includes a local orchestrator to make developing microservices easier and the ability to deploy microservices to Kubernetes with minimal configuration.
MIT License
5.28k stars 520 forks source link

Unable to Deploy with Tye in Azure Pipelines #911

Open Phiph opened 3 years ago

Phiph commented 3 years ago

I'm working with Azure Dev ops to deploy Tye to my kubernetes cluster.

I have managed to build a pipeline that installs the tool, installs kubectl, and connects to my Kubernetes cluster.

Within the pipeline I can run kubectl get pods - which returns the pods I expect to see.

However, when I run "Tye Deploy" tye returns

Drats! 'deploy' failed: Cannot apply manifests because kubectl is not connected to a cluster.

To Reproduce

Here is the pipeline Stages I'm using ` stages:

here is the raw output from Azure Dev ops from the Kubectl install step

` 2021-01-21T11:15:51.9332957Z ##[section]Starting: KubectlInstaller 2021-01-21T11:15:51.9343725Z ============================================================================== 2021-01-21T11:15:51.9344100Z Task : Kubectl tool installer 2021-01-21T11:15:51.9344418Z Description : Install Kubectl on agent machine 2021-01-21T11:15:51.9344674Z Version : 0.175.1 2021-01-21T11:15:51.9344949Z Author : Microsoft Corporation 2021-01-21T11:15:51.9345263Z Help : https://aka.ms/azpipes-kubectl-installer-tsg 2021-01-21T11:15:51.9345635Z ============================================================================== 2021-01-21T11:15:55.9561648Z Downloading: https://storage.googleapis.com/kubernetes-release/release/stable.txt 2021-01-21T11:15:56.1447382Z Downloading: https://storage.googleapis.com/kubernetes-release/release/v1.20.2/bin/linux/amd64/kubectl 2021-01-21T11:15:58.3667033Z Caching tool: kubectl 1.20.2 x64 2021-01-21T11:15:58.4126659Z Prepending PATH environment variable with directory: /opt/hostedtoolcache/kubectl/1.20.2/x64 2021-01-21T11:15:58.8728707Z Verifying kubectl installation... 2021-01-21T11:15:58.8755870Z [command]/opt/hostedtoolcache/kubectl/1.20.2/x64/kubectl --help 2021-01-21T11:15:58.9418509Z kubectl controls the Kubernetes cluster manager. 2021-01-21T11:15:58.9419253Z 2021-01-21T11:15:58.9423328Z Find more information at: https://kubernetes.io/docs/reference/kubectl/overview/ 2021-01-21T11:15:58.9423872Z 2021-01-21T11:15:58.9424343Z Basic Commands (Beginner): 2021-01-21T11:15:58.9424861Z create Create a resource from a file or from stdin. 2021-01-21T11:15:58.9425518Z expose Take a replication controller, service, deployment or pod and expose it as a new Kubernetes Service 2021-01-21T11:15:58.9426258Z run Run a particular image on the cluster 2021-01-21T11:15:58.9426893Z set Set specific features on objects 2021-01-21T11:15:58.9427301Z 2021-01-21T11:15:58.9427727Z Basic Commands (Intermediate): 2021-01-21T11:15:58.9428259Z explain Documentation of resources 2021-01-21T11:15:58.9428789Z get Display one or many resources 2021-01-21T11:15:58.9429287Z edit Edit a resource on the server 2021-01-21T11:15:58.9429901Z delete Delete resources by filenames, stdin, resources and names, or by resources and label selector 2021-01-21T11:15:58.9430388Z 2021-01-21T11:15:58.9430809Z Deploy Commands: 2021-01-21T11:15:58.9431292Z rollout Manage the rollout of a resource 2021-01-21T11:15:58.9431857Z scale Set a new size for a Deployment, ReplicaSet or Replication Controller 2021-01-21T11:15:58.9433144Z autoscale Auto-scale a Deployment, ReplicaSet, or ReplicationController 2021-01-21T11:15:58.9433667Z 2021-01-21T11:15:58.9434119Z Cluster Management Commands: 2021-01-21T11:15:58.9434611Z certificate Modify certificate resources. 2021-01-21T11:15:58.9435402Z cluster-info Display cluster info 2021-01-21T11:15:58.9435985Z top Display Resource (CPU/Memory/Storage) usage. 2021-01-21T11:15:58.9436881Z cordon Mark node as unschedulable 2021-01-21T11:15:58.9437400Z uncordon Mark node as schedulable 2021-01-21T11:15:58.9437905Z drain Drain node in preparation for maintenance 2021-01-21T11:15:58.9438511Z taint Update the taints on one or more nodes 2021-01-21T11:15:58.9438984Z 2021-01-21T11:15:58.9439471Z Troubleshooting and Debugging Commands: 2021-01-21T11:15:58.9440046Z describe Show details of a specific resource or group of resources 2021-01-21T11:15:58.9440606Z logs Print the logs for a container in a pod 2021-01-21T11:15:58.9441138Z attach Attach to a running container 2021-01-21T11:15:58.9441658Z exec Execute a command in a container 2021-01-21T11:15:58.9442456Z port-forward Forward one or more local ports to a pod 2021-01-21T11:15:58.9443074Z proxy Run a proxy to the Kubernetes API server 2021-01-21T11:15:58.9443623Z cp Copy files and directories to and from containers. 2021-01-21T11:15:58.9444359Z auth Inspect authorization 2021-01-21T11:15:58.9445000Z debug Create debugging sessions for troubleshooting workloads and nodes 2021-01-21T11:15:58.9448321Z 2021-01-21T11:15:58.9448608Z Advanced Commands: 2021-01-21T11:15:58.9449189Z diff Diff live version against would-be applied version 2021-01-21T11:15:58.9449577Z apply Apply a configuration to a resource by filename or stdin 2021-01-21T11:15:58.9449927Z patch Update field(s) of a resource 2021-01-21T11:15:58.9450231Z replace Replace a resource by filename or stdin 2021-01-21T11:15:58.9450633Z wait Experimental: Wait for a specific condition on one or many resources. 2021-01-21T11:15:58.9451052Z kustomize Build a kustomization target from a directory or a remote url. 2021-01-21T11:15:58.9451304Z 2021-01-21T11:15:58.9451487Z Settings Commands: 2021-01-21T11:15:58.9451767Z label Update the labels on a resource 2021-01-21T11:15:58.9452081Z annotate Update the annotations on a resource 2021-01-21T11:15:58.9452461Z completion Output shell completion code for the specified shell (bash or zsh) 2021-01-21T11:15:58.9452699Z 2021-01-21T11:15:58.9452892Z Other Commands: 2021-01-21T11:15:58.9453371Z api-resources Print the supported API resources on the server 2021-01-21T11:15:58.9453989Z api-versions Print the supported API versions on the server, in the form of "group/version" 2021-01-21T11:15:58.9454360Z config Modify kubeconfig files 2021-01-21T11:15:58.9454664Z plugin Provides utilities for interacting with plugins. 2021-01-21T11:15:58.9455023Z version Print the client and server version information 2021-01-21T11:15:58.9455232Z 2021-01-21T11:15:58.9456589Z Usage: 2021-01-21T11:15:58.9456816Z kubectl [flags] [options] 2021-01-21T11:15:58.9456969Z 2021-01-21T11:15:58.9457506Z Use "kubectl --help" for more information about a given command. 2021-01-21T11:15:58.9458141Z Use "kubectl options" for a list of global command-line options (applies to all commands). 2021-01-21T11:15:58.9612013Z ##[section]Finishing: KubectlInstaller 2021-01-21T11:15:58.9643163Z ##[section]Starting: Kubernetes 2021-01-21T11:15:58.9659457Z ============================================================================== 2021-01-21T11:15:58.9659808Z Task : Kubectl 2021-01-21T11:15:58.9660223Z Description : Deploy, configure, update a Kubernetes cluster in Azure Container Service by running kubectl commands 2021-01-21T11:15:58.9660624Z Version : 1.175.3 2021-01-21T11:15:58.9660964Z Author : Microsoft Corporation 2021-01-21T11:15:58.9661298Z Help : https://aka.ms/azpipes-kubectl-tsg 2021-01-21T11:15:58.9661658Z ============================================================================== 2021-01-21T11:16:01.1288406Z ============================================================================== 2021-01-21T11:16:01.1299466Z Kubectl Client Version: v1.20.2 2021-01-21T11:16:01.1299759Z Kubectl Server Version: v1.19.3 2021-01-21T11:16:01.1300315Z ============================================================================== 2021-01-21T11:16:01.1813312Z commandOutput 2021-01-21T11:16:01.5058877Z ##[section]Finishing: Kubernetes 2021-01-21T11:16:01.5086721Z ##[section]Starting: CmdLine 2021-01-21T11:16:01.5093435Z ============================================================================== 2021-01-21T11:16:01.5093755Z Task : Command line 2021-01-21T11:16:01.5094252Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows 2021-01-21T11:16:01.5094733Z Version : 2.178.0 2021-01-21T11:16:01.5094969Z Author : Microsoft Corporation 2021-01-21T11:16:01.5095333Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line 2021-01-21T11:16:01.5095710Z ============================================================================== 2021-01-21T11:16:01.6579035Z Generating script. 2021-01-21T11:16:01.6589831Z Script contents: 2021-01-21T11:16:01.6590474Z kubectl get pods 2021-01-21T11:16:01.6591115Z ========================== Starting Command Output =========================== 2021-01-21T11:16:01.6618503Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/d3f19094-64bb-4efe-b245-f5a64c05d499.sh 2021-01-21T11:16:01.7778436Z NAME READY STATUS RESTARTS AGE 2021-01-21T11:16:01.7784504Z mic-9dc69b97b-nc2nt 1/1 Running 0 16h 2021-01-21T11:16:01.7834815Z mic-9dc69b97b-psb5f 1/1 Running 0 16h 2021-01-21T11:16:01.7835516Z pentest-management-7b5d88b677-g46jg 2/2 Running 0 16h 2021-01-21T11:16:01.7836083Z pentestapi-66cb9f79bf-wxrhm 2/2 Running 1 16h 2021-01-21T11:16:01.7888757Z ##[section]Finishing: CmdLine 2021-01-21T11:16:01.7914894Z ##[section]Starting: DotNetCoreCLI 2021-01-21T11:16:01.7923310Z ============================================================================== 2021-01-21T11:16:01.7923808Z Task : .NET Core 2021-01-21T11:16:01.7924132Z Description : Build, test, package, or publish a dotnet application, or run a custom dotnet command 2021-01-21T11:16:01.7924461Z Version : 2.179.2 2021-01-21T11:16:01.7924702Z Author : Microsoft Corporation 2021-01-21T11:16:01.7925029Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/build/dotnet-core-cli 2021-01-21T11:16:01.7925840Z ============================================================================== 2021-01-21T11:16:02.2581350Z Info: .NET Core SDK/runtime 2.2 and 3.0 are now End of Life(EOL) and have been removed from all hosted agents. If you're using these SDK/runtimes on hosted agents, kindly upgrade to newer versions which are not EOL, or else use UseDotNet task to install the required version. 2021-01-21T11:16:02.2608037Z [command]/opt/hostedtoolcache/dotnet/dotnet tool update -g Microsoft.Tye --version 0.6.0-alpha.21070.5 2021-01-21T11:16:02.3737309Z 2021-01-21T11:16:02.3819486Z Welcome to .NET Core 3.1! 2021-01-21T11:16:02.3820766Z --------------------- 2021-01-21T11:16:02.3821330Z SDK Version: 3.1.405 2021-01-21T11:16:02.3821718Z 2021-01-21T11:16:02.3822119Z Telemetry 2021-01-21T11:16:02.3822700Z --------- 2021-01-21T11:16:02.3825254Z The .NET Core tools collect usage data in order to help us improve your experience. It is collected by Microsoft and shared with the community. You can opt-out of telemetry by setting the DOTNET_CLI_TELEMETRY_OPTOUT environment variable to '1' or 'true' using your favorite shell. 2021-01-21T11:16:02.3826228Z 2021-01-21T11:16:02.3827142Z Read more about .NET Core CLI Tools telemetry: https://aka.ms/dotnet-cli-telemetry 2021-01-21T11:16:02.3827635Z 2021-01-21T11:16:02.3828349Z ---------------- 2021-01-21T11:16:02.3829256Z Explore documentation: https://aka.ms/dotnet-docs 2021-01-21T11:16:02.3830181Z Report issues and find source on GitHub: https://github.com/dotnet/core 2021-01-21T11:16:02.3831000Z Find out what's new: https://aka.ms/dotnet-whats-new 2021-01-21T11:16:02.3832676Z Learn about the installed HTTPS developer cert: https://aka.ms/aspnet-core-https 2021-01-21T11:16:02.3833702Z Use 'dotnet --help' to see available commands or visit: https://aka.ms/dotnet-cli-docs 2021-01-21T11:16:02.3834643Z Write your first app: https://aka.ms/first-net-core-app 2021-01-21T11:16:02.3835625Z -------------------------------------------------------------------------------------- 2021-01-21T11:16:05.0395256Z You can invoke the tool using the following command: tye 2021-01-21T11:16:05.0397092Z Tool 'microsoft.tye' (version '0.6.0-alpha.21070.5') was successfully installed. 2021-01-21T11:16:05.0523250Z Info: Azure Pipelines hosted agents have been updated and now contain .Net 5.x SDK/Runtime along with the older .Net Core version which are currently lts. Unless you have locked down a SDK version for your project(s), 5.x SDK might be picked up which might have breaking behavior as compared to previous versions. You can learn more about the breaking changes here: https://docs.microsoft.com/en-us/dotnet/core/tools/ and https://docs.microsoft.com/en-us/dotnet/core/compatibility/ . To learn about more such changes and troubleshoot, refer here: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/build/dotnet-core-cli?view=azure-devops#troubleshooting 2021-01-21T11:16:05.0558291Z ##[section]Finishing: DotNetCoreCLI 2021-01-21T11:16:05.0584733Z ##[section]Starting: CmdLine 2021-01-21T11:16:05.0599471Z ============================================================================== 2021-01-21T11:16:05.0599823Z Task : Command line 2021-01-21T11:16:05.0600212Z Description : Run a command line script using Bash on Linux and macOS and cmd.exe on Windows 2021-01-21T11:16:05.0600571Z Version : 2.178.0 2021-01-21T11:16:05.0600858Z Author : Microsoft Corporation 2021-01-21T11:16:05.0601255Z Help : https://docs.microsoft.com/azure/devops/pipelines/tasks/utility/command-line 2021-01-21T11:16:05.0602123Z ============================================================================== 2021-01-21T11:16:05.2144796Z Generating script. 2021-01-21T11:16:05.2156814Z Script contents: 2021-01-21T11:16:05.2158162Z tye deploy -v debug 2021-01-21T11:16:05.2158435Z ========================== Starting Command Output =========================== 2021-01-21T11:16:05.2186681Z [command]/bin/bash --noprofile --norc /home/vsts/work/_temp/38634601-fb1e-49e2-af1e-2dbf316fce5b.sh 2021-01-21T11:16:05.4483360Z Loading Application Details... 2021-01-21T11:16:05.5720441Z Restoring and evaluating projects 2021-01-21T11:16:26.3624298Z Resolved metadata for service pentestapi at /home/vsts/work/1/s/src/API/PenTestAPI/obj/Debug/netcoreapp3.1/MicrosoftTye.ProjectMetadata.txt 2021-01-21T11:16:26.3625649Z Resolved metadata for service pentest-management at /home/vsts/work/1/s/src/API/PenTest.Management/obj/Debug/netcoreapp3.1/MicrosoftTye.ProjectMetadata.txt 2021-01-21T11:16:26.3631358Z Restore and project evaluation took: 20796.5999ms 2021-01-21T11:16:26.3748353Z Found application version: 1.0.1 2021-01-21T11:16:26.3749754Z RunCommand=/home/vsts/work/1/s/src/API/PenTestAPI/bin/Debug/netcoreapp3.1/PenTestAPI 2021-01-21T11:16:26.3750198Z RunArguments= 2021-01-21T11:16:26.3750709Z TargetPath=/home/vsts/work/1/s/src/API/PenTestAPI/bin/Debug/netcoreapp3.1/PenTestAPI.dll 2021-01-21T11:16:26.3751148Z PublishDir=bin/Debug/netcoreapp3.1/publish/ 2021-01-21T11:16:26.3751422Z AssemblyName=PenTestAPI 2021-01-21T11:16:26.3751709Z IntermediateOutputPath=obj/Debug/netcoreapp3.1/ 2021-01-21T11:16:26.3752000Z Found target framework: netcoreapp3.1 2021-01-21T11:16:26.3752385Z Parsed target framework name: netcoreapp 2021-01-21T11:16:26.3753000Z Parsed target framework version: 3.1 2021-01-21T11:16:26.3763016Z Found shared frameworks: Microsoft.NETCore.App, Microsoft.AspNetCore.App 2021-01-21T11:16:26.3766296Z IsAspNet=True 2021-01-21T11:16:26.3766884Z Evaluation Took: 2.7207ms 2021-01-21T11:16:26.3808213Z Found application version: 1.0.10 2021-01-21T11:16:26.3808882Z RunCommand=/home/vsts/work/1/s/src/API/PenTest.Management/bin/Debug/netcoreapp3.1/PenTest.Management 2021-01-21T11:16:26.3809598Z RunArguments= 2021-01-21T11:16:26.3810099Z TargetPath=/home/vsts/work/1/s/src/API/PenTest.Management/bin/Debug/netcoreapp3.1/PenTest.Management.dll 2021-01-21T11:16:26.3810452Z PublishDir=bin/Debug/netcoreapp3.1/publish/ 2021-01-21T11:16:26.3810726Z AssemblyName=PenTest.Management 2021-01-21T11:16:26.3810978Z IntermediateOutputPath=obj/Debug/netcoreapp3.1/ 2021-01-21T11:16:26.3811423Z Found target framework: netcoreapp3.1 2021-01-21T11:16:26.3811863Z Parsed target framework name: netcoreapp 2021-01-21T11:16:26.3812161Z Parsed target framework version: 3.1 2021-01-21T11:16:26.3812501Z Found shared frameworks: Microsoft.NETCore.App, Microsoft.AspNetCore.App 2021-01-21T11:16:26.3812773Z IsAspNet=True 2021-01-21T11:16:26.3812993Z Evaluation Took: 0.2903ms 2021-01-21T11:16:26.3844583Z Verifying kubectl installation... 2021-01-21T11:16:26.5000525Z Verifying kubectl connection to cluster... 2021-01-21T11:16:26.6340934Z Drats! 'deploy' failed: 2021-01-21T11:16:26.6343234Z Cannot apply manifests because kubectl is not connected to a cluster. 2021-01-21T11:16:26.6498323Z ##[error]Bash exited with code '1'. 2021-01-21T11:16:26.6588520Z ##[section]Finishing: CmdLine

`

Further technical details

Phiph commented 3 years ago

Might of found my problem.

Looking through the source, I can see to check the status of the cluster Tye uses kubectl cluster-info.

Running this command in Azure Dev ops I get this

Generating script. Script contents: kubectl cluster-info ========================== Starting Command Output =========================== /bin/bash --noprofile --norc /home/vsts/work/_temp/c5ace85b-69c6-4560-b148-f30f0503b264.sh

To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'. Error from server (Forbidden): services is forbidden: User "system:serviceaccount:default:***" cannot list resource "services" in API group "" in the namespace "kube-system"

[error]Bash exited with code '1'.

Finishing: CmdLine

I'll go and fix my service account - and that might solve my problemo!

Phiph commented 3 years ago

I couldn't find the proper way to do this in Kubernetes.

I think I've fixed it though - but I'll leave this issue up for comments/feedback.

I changed my Kubernetes Login to this - and it works. 

    connectionType: 'Azure Resource Manager' 
    azureSubscriptionEndpoint: 'xxxxxxx' 
    azureResourceGroup: 'PenTest-Management' 
    kubernetesCluster: 'apollo' 
    useClusterAdmin: true 
    command: 'login'
Celakanth-VSI commented 3 years ago

I have the same issue when running the tye command using a powershell or bash task I get this error Verifying kubectl installation... Drats! 'deploy' failed: Cannot apply manifests because kubectl is not installed. I have tried on a self hosted agent with kubectl installed and working but I still get the error

Phiph commented 3 years ago

Yeah, you need to install KubeCTL on the agent using the pipeline.

Try using parts of my build steps, and test your connection out with the command line step.

So once you've authed with the Cluster in your pipeline, try getting the pods.

task: CmdLine@2 inputs: script: 'kubectl get pods'

If that works, then try getting the cluster-info as this is the command Tye uses to validate the connection

task: CmdLine@2 inputs: script: 'kubectl cluster-info'

If this fails with the same error I had, you just need to either change the way you connect to your cluster either as an admin, or another means.

Cheers,

P