Closed theTestTube closed 2 years ago
My first reaction, reading the log, is that Aggregator is not able to connect to Azure DevOps.
You mentioned that checked the connection using wget --no-check-certificate https://
... which rule out DNS issues, so I suspect an SSL issue. Are you tried using Aggregator_AzureDevOpsCertificate
to add AzDO certificate to the trusted ones?
Dear @giuliov, thanks for your fast response.
Yes, an Aggregator_AzureDevOpsCertificate
has been specified and I've checked twice the /secrets/tfs.cer
used for trust is present and readable,
/app # ls -lta $Aggregator_AzureDevOpsCertificate
-rwxrwxrwx 1 root root 1889 Oct 8 11:23 /secrets/tfs.cer
Please notice this is the Azure DevOps 2020 server certificate including the root and intermediate CAs,
May be another clue could give you some more info, testing the webhook from "Project Settings; Service hooks" works properly,
info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
Executing ObjectResult, writing value of type '<>f__AnonymousType1`1[[System.String, System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e]]'.
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
Executed action aggregator_host.Controllers.WorkItemController.PostAsync (aggregator-host) in 19.4591ms
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'aggregator_host.Controllers.WorkItemController.PostAsync (aggregator-host)'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 22.0227ms 200 application/json; charset=utf-8
While invoking through WIT update fails,
dbug: Aggregator[0]
Searching 'test1' in /rules
dbug: Aggregator[0]
Rule code found at /rules/test1.rule
dbug: Aggregator[0]
Connecting to Azure DevOps using PAT...
info: Microsoft.AspNetCore.Mvc.Infrastructure.ObjectResultExecutor[1]
Executing ObjectResult, writing value of type 'System.String'.
info: Microsoft.AspNetCore.Mvc.Infrastructure.ControllerActionInvoker[2]
Executed action aggregator_host.Controllers.WorkItemController.PostAsync (aggregator-host) in 29970.8469ms
info: Microsoft.AspNetCore.Routing.EndpointMiddleware[1]
Executed endpoint 'aggregator_host.Controllers.WorkItemController.PostAsync (aggregator-host)'
info: Microsoft.AspNetCore.Hosting.Diagnostics[2]
Request finished in 29973.967ms 400 text/plain; charset=utf-8
Sorry for the delay, I am on the road with just one laptop.
Recapping:
That connection is controlled by a Polly policy in RuleExecutor.ConnectToAzureDevOpsAsync
method. It attempts a connection and retries when AzDO replies with an HTTP status 429 (Too Many Requests). We see the message Waiting 00:00:30 before retrying
infact. I do not understand the A task was canceled
message happening so quick, you should see additional retry messages.
Dear Giulio,
Your recap is right. I've also checked that no callback to AzDO is taking place at the time it states Connecting to Azure DevOps using PAT...
.
To check this I've,
HostName
and OrganizationName
bash vars match those received in the Agg HTTP rule1
execution webhook and correct UserName
, and Aggregator_Token
are also specifiedwget -o stdout --no-check-certificate "https://$UserName:$Aggregator_Token@$HostName/$OrganizationName/_apis/projects?api-version=2.0&nonsense1"
test1
execution by making a WIT changeConnecting to Azure DevOps using PAT...
, also Waiting 00:00:30 before retrying (attemp #1/3)...
and even A task was canceled
wget ...?api-version=2.0&nonsense2"
I inspected Azure DevOps 2020 Server access logs and found no request from my Docker machine between the nonsense1
and nonsense2
HTTPS requests.
I wonder where the hell is Agg making the HTTPS request to. How could I trace it? Is there any option to log debug info including the target URL for the Connecting to Azure DevOps using PAT...
request?
Apart, is there any guide to make Agg work outside Docker (as a .Net WebAPI in IIS, for example), in order to debug the project with Visual Studio 2019 and have all this information on our own?
Many thanks.-
Building on your machine is trivial. There is a single solution for all flavours. In this scenario I launch aggregator-host.csproj
project.
Debugging is a bit complex.
The host project must use TLS, with a certificate trusted by AzDO, reachable with a proper DNS name, port must be open on local firewall.
You must define the API key(s), the Aggregator_SharedSecret
environment variable to enable the CLI.
The rule files can be in the bin
directory or in the directory defined by Aggregator_RulesPath
.
HTH
I haven't heard you in a while. Have you solved the issue? You mentioned that DNS resolution from the machine hosting the container, but is it working from within the container?
Thanks in advance! 👦
Steps to reproduce
Configuring runtime and a new
test1
rule following Docker setup instructions,Running docker image,
Testing Ok status,
Log in to ADO,
Mapping a local rule,
Updating a work item and correctly receiving REST request on
aggregator-host
Docker process.I've also correctly checked connectivity from the Docker machine shell to Azure DevOps with
wget --no-check-certificate https://...
Expected behavior
Output from
test1
execution was expected.Actual behavior
Before actually running the
test1
function last successfull message isConnecting to Azure DevOps using PAT...
followed byWaiting 00:00:30 before retrying (attemp #1/3)...
and anA task was canceled
error message.Diagnostic logs
These are the traces upon work item update, including the
fail
message,These are the traces for a correct Azure DevOps project webhook test execution,
Environment
Running Azure DevOps Server 2020.1 on-premise.
Running the
tfsaggregator/aggregator3:latest
, I wouldn't mind running theaggregator-host
process forv1.1.0
release in my IIS or a custom-built docker image with trace logs if I had known how to do it.