This is demo project for RabbitMQ.TraceableMessaging with Application Insights.
It shows how communications between microservices can be built over RabbitMQ with distributed tracing. You can benefit from scrutinizing this project if you:
You don't need to know C# to reach first 2 goals.
You need Azure with Application Insights instance. If you don't have Azure by now, start from https://azure.microsoft.com/free/. In Azure, please, create App Insights and find instrumentation key there:
Probably, you'll prefer to use Docker to start everything or at least RabbitMQ. Go to Docker Desktop then install it or check if you have docker and compose:
docker version
docker-compose version
I have tested with Docker Engine - Community 18.09.2 and docker-compose version 1.23.2. I believe higher versions should support the same instructions.
If you like to start in VS Code or Visual Studio then you need .NET Core SDK 2.2. Download it from here then install or check if you have it:
dotnet --list-sdks
It's not necessary to have .NET Core SDK or VS Code / VS, you may just docker-compose run cli
, but you may like to debug with it.
Neither is required, use an approach to build and run that is convenient for you. So below three of them are described - everything in Docker, approach based on VS Code, another on Visual Studio.
git clone https://github.com/dmlarionov/RabbitMQ.TraceableMessaging-example1.git
docker-compose run cli
q
to force telemetry flushing to the cloud.Ctrl
+C
to stop cli
container.docker-compose down
docker-compose -f .\docker-compose.rabbitmq.yml up -d
Ctrl
+ Shift
+ B
then choose build
task.RUN ALL
compound. CLI will be opened in external terminal.q
to force telemetry flushing to the cloud.Shift
+ F5
six times).docker-compose down
docker-compose -f .\docker-compose.rabbitmq.yml up -d
apigw
, bang
, bar
, cli
, fib
, foo
, but not lib
. So check solution properties:q
to force telemetry flushing to the cloud.Shift
+ F5
).docker-compose down
After you enter Application Insights instrumentation key CLI sends it to every microservice and to API gateway, awaits for ready message from all of them, then (if ok) displays the following menu:
Which scenario should we test with API gateway?:
1) HTTP GET /api/demo/fooping1
2) HTTP GET /api/demo/barping2
3) HTTP GET /api/demo/fibping3 (Internal Server Error)
4) HTTP GET /api/demo/fibping4 (Forbidden)
Each HTTP call to API gateway turnes into series of RabbitMQ calls. You can see a distributed trace in your Application Insights instance in Azure portal. You can test scenarios in any order.
To quit press 'q'.
You can execute scenarios in any order multiple times. After each execution the same menu will be presented to you.
CLI communicates with API gateway over HTTP. API gateway communicates with microservices (foo
, bar
and fib
) over RabbitMQ. The provided scenarios implement the following call chains:
foo
-> bar
, then bar
calls bang
and fib
consequently.bar
, then bar
calls fib
and bang
in parallel.fib
-> bar
-> bang
, then bang
generates Exception
.fib
-> foo
, then foo
generates NotImplementedException
.If any service at any level replies with Unauthorized
or Forbidden
or Fail
status then such reply is retransmitted upstream. At the API gateway this become translated to appropriate HTTP reply:
Unauthorized
-> HTTP status 401 Unauthorized
.Forbidden
-> HTTP status 403 Forbidden
.Fail
-> HTTP status 500 Internal Server Error
.After 401 Unauthorized
CLI asks for username, generates JWT and starts using it as bearer token with every new request. API gateway, foo
, bar
and fib
retransmit token downstream over RabbitMQ.
So you will be asked for username after your first request. If you start with scenarios 2-4 then API gateway will not execute any subsequent requests for your first call, because endpoints 2-4 are protected with [Authorize]
attribute, but if you start with scenario 1, foo
will be called without bearer token.
Here is short explanation of essential things about the code of API gateway and microservices. Real applications can be built with similar approach. You have to know C# to understand this part.
If you just want to see what distributed traces looks like go to Scrutinizing Application Insights.
RabbitMQ RPC client of RabbitMQ.TraceableMessaging library throw exceptions for Unauthorized
, Forbidden
and Fail
reply status codes. This way unsuccessful replies become exceptions for further translation higher in the stack.
In API gateway there is ExceptionFilter
which translates exceptions to HTTP reply statuses (see apigw/ExceptionFilter.cs
):
public void OnException(ExceptionContext context)
{
if (context.Exception.GetType() == typeof(UnauthorizedException))
context.Result = new UnauthorizedResult();
else if (context.Exception.GetType() == typeof(ForbiddenException))
context.Result = new ForbidResult();
else
context.Result = new StatusCodeResult(500);
}
HTTP controllers are extremely simple:
public async Task<IActionResult> FooPing1() =>
Ok(await _fooClient.GetReplyAsync<Pong1>(new Ping1(),
HttpContext.GetTokenAsync("access_token").Result));
JWT bearer validation is configured in Program.cs
in the following way:
// token validation parameters
var tokenValidationParameters = new TokenValidationParameters
{
ValidateAudience = false,
ValidateIssuer = false,
ValidateIssuerSigningKey = true,
IssuerSigningKey = new SymmetricSecurityKey(tokenKey)
};
// set host configuration
var hostBuilder = new WebHostBuilder()
...
.ConfigureServices(services =>
{
...
services.AddAuthentication(JwtBearerDefaults.AuthenticationScheme)
.AddJwtBearer(options =>
{
options.TokenValidationParameters = tokenValidationParameters;
options.SaveToken = true;
});
...
})
SaveToken = true
is needed for HttpContext.GetTokenAsync("access_token")
in controller to succeed.
In real application you'll have to validate issuer and audience (if audience is an API gateway).
In the shared project lib
there are Service
class which implements IHostedService
, declares request queue, initializes channel and RpcServer<JwtSecurityContext>
(of RabbitMQ.TraceableMessaging). This class has abstract OnReceive
method for invocation on request. See lib/Service.cs
.
Each microservice has service class inheriting from lib.Service
with OnReceive
similar to this:
protected override void OnReceive(object sender, RequestEventArgs<TelemetryContext, JwtSecurityContext> ea)
{
try
{
// processing depends on request type
switch (ea.RequestType)
{
case nameof(Ping1):
... // perfoming job for request of type Ping1
Server.Reply(ea.CorrelationId, ...);
break;
case nameof(Ping3):
... // perfoming job for request of type Ping3
Server.Reply(ea.CorrelationId, ...);
break;
default:
throw new NotImplementedException($"{ea.RequestType} is not implemented");
}
}
catch(Exception ex)
{
Server.Fail(ea, ex);
}
}
If some microservice is a client for another microservices it declares response queues for downstream calls and initializes RpcClient
for each downstream microservice. Such microservice's service class looks like:
public sealed class FooService : Service
{
private readonly RpcClient _barClient;
public FooService(
IConnection conn,
IConfiguration config,
SecurityOptions securityOptions)
: base(conn, config, securityOptions)
{
var channel = conn.CreateModel();
var queue = $"foo-bar-reply-{Guid.NewGuid().ToString()}";
channel.QueueDeclare(queue);
_barClient = new RpcClient(
channel,
new PublishOptions(config["RabbitMQ:Services:Bar"]),
new ConsumeOptions(queue),
new JsonFormatOptions());
}
protected override void OnReceive(object sender, RequestEventArgs<TelemetryContext, JwtSecurityContext> ea)
{
...
// you saw example above
}
}
JWT bearer validation is configured in Program.cs
in the following way:
// token validation parameters
var tokenValidationParameters = new TokenValidationParameters
{
ValidateAudience = false,
ValidateIssuer = false,
ValidateIssuerSigningKey = true,
IssuerSigningKey = new SymmetricSecurityKey(tokenKey)
};
// security options for RabbitMQ service
var securityOptions = new SecurityOptions(
// map of request types to scope names
new Dictionary<string, string>()
{
{ nameof(Ping1), "1" },
{ nameof(Ping2), "2" },
{ nameof(Ping3), "3" },
{ nameof(Ping4), "4" }
},
tokenValidationParameters);
// set host configuration
var hostBuilder = new HostBuilder()
...
.ConfigureServices((context, services) => {
...
services.AddSingleton<SecurityOptions>(securityOptions);
...
});
Instance of SecurityOptions
is passed to service class with constructor injection. Class SecurityOptions
is defined in lib
project as descendant of JwtSecurityOptions
with Authorize
delegate which uses dictionary to match request types with scopes. In the example above token have to possess scope "1" for request of type Ping1
, possess scope "2" for request of type Ping2
and so on.
In real application you'll have to validate issuer and may be audience (if audience is a microservice), also you may have your own security options implementation with different logic.
After short period of time, when all statistics sent to Application Insights will be processed, at application map you may see something similar to this:
If you want to align services upstream to downstream from left to right you can switch to hierarchical map:
Yes, we made tricky dependencies, but can untwine: bar
is called by API gateway and by foo
, foo
is called by fib
and API gateway, fib
is called by bar
and API gateway. This pictures doesn't mean that you have any single request type passing all legs of the graph, what you see is dependencies between microservices.
Number of instances is an aggregate made due to execution (in my case) of infrastructure multiple times, but for real application you probably will see the counts of service instances that works in parallel.
Let's take a look why 100% of calls from fib to foo are red:
You see that there are 45 failed requests and 28 are forbidden. Let's dive into failed requests:
You can see the time periods when failures and dependency calls had spikes. Let's scope time around the spike and drill into these 25 dependencies:
Application Insights analyses dependency calls for similarity and suggests you most representative call instances for investigation. You can select one and see what happened:
We found out that some exception in bang
led to failure of dependency call from fib
to foo
, which was made to serve HTTP request GET Demo/FibPing3 at the top of the distributed execution stack.
We see what principal identity name was. With your own implementation you can add to telemetry any information you need, which also would be analyzed by Application Insights to find similarities between failures to suggest investigations.
Let's see what was exception in the bang
microservice:
We got it! The reason of investigated failure of fib
-> foo
dependency call which led to failure of HTTP GET Demo/FibPing3 request was System.Exception
with message "I can't withstand this!" happened in bang
microservice.
Application Insights is designed to track events, exceptions, requests and dependency calls, analyze failures and performance issues, allowing you to develop your own solutions.
Other APM tools have similar abilities. Most of them are based on Open Tracing framework which contains implementations for dozens of languages.
Both Application Insights (AI) and Open Tracing (OT) bind (upstream) requests to (downstream) dependency calls by putting some information into message headers then tailor up telemetry events in the cloud.
This is demo of implementation of AI distributed tracing for RabbitMQ (RabbitMQ.TraceableMessaging) that complements AI distributed tracing implemented for HTTP by Microsoft.
Can you benefit from that? If yes please feel free to contribute to RabbitMQ.TraceableMessaging by implementing support for other APM tools, tests, performance improvements or by making publications about technology.