Question: SnapshotCollector

yahorsi commented 5 years ago

Classic ASP.NET Do we need to do anything extra to make SnapshotCollector working? Or, in other words, should SnapshotCollector.SnapshotCollectorTelemetryProcessor be explicitly added in the ApplicationInsights.config or adding corresponding nuget package is enought? If yes, could you please provide sample.

Core ASP.NET What should be done to make snapshots collection working in the ASP.NET Core application?

PS: Sorry for spam, trying to make AI working 3rd day (( The docs are not great (

pharring commented 5 years ago

Sorry you're having trouble. The documentation is here: https://docs.microsoft.com/azure/azure-monitor/app/snapshot-debugger#enable-application-insights-snapshot-debugger-for-your-application

I think they're pretty complete, but do let us know if you have feedback on the docs. e.g. You find something confusing, or your scenario isn't covered.

Classic ASP.NET

Add the nuget package. This should automatically add the SnapshotCollectorTelemetryProcessor to your ApplicationInsights.config file. If not, manually add the following:

<TelemetryProcessors>
    <Add Type="Microsoft.ApplicationInsights.SnapshotCollector.SnapshotCollectorTelemetryProcessor, Microsoft.ApplicationInsights.SnapshotCollector">
    </Add>
    <!-- other telemetry processors here, as needed for your app -->
</TelemetryProcessors>

ASP.NET Core

Add the nuget package.
Call the AddSnapshotCollector extension method from within ConfigureServices in your application's startup code (usually Startup.cs). e.g.
```
using Microsoft.ApplicationInsights.SnapshotCollector;
...
// In ConfigureServices
services.AddSnapshotCollector();
```

If you're still having trouble, please describe your application a bit more and how/where you intend to run it. (Azure App Service, custom VM, docker container, etc.)

yahorsi commented 5 years ago

Hi guys, I just want to say that todays Microsoft (OSS/Github) makes me happy :) It is now so much easier to get help, so thatk you guys :)

First, about app we build this time. It is classic .NET 4.7.2 and ASP.NET WCF service (no we cannot use WebAPI and .NET Core, we are forced to use WCF to integrate with external services). Cherry on the cake, as we all want to be in the trend we use classic .NET with NEW project system aka SDK style project. And, after some time using .NET Core I used to add nuget packages manually to the project file, and this is how I add them now...

Now, back to the topic. I of course did read docs, most of them several times and unfortunately currently they are confusing in many ways. Below I'll try to explain what I read, what I understand, and how to fix.

Please assume I'm the one who want's to add stapshot collector first time in my life... So let's start from the link you provided: https://docs.microsoft.com/azure/azure-monitor/app/snapshot-debugger#enable-application-insights-snapshot-debugger-for-your-application

Include the Snapshot collector NuGet package in your application, and optionally configure collection parameters in ApplicationInsights.config. Snapshots appear on exceptions in the Application Insights portal. How I read that - just add the nuget package and tyhat's it, changes in the 'ApplicationInsights.config' are optional. And that I did, I have opened my project file and add the package. Issues:

Doc does not mention that after you add the package your 'ApplicationInsights.config' will be modified and will now include the record for the snapshot debugger.
Doc does not provide any sample how 'ApplicationInsights.config' (and web.config as well) should look like
Doc does not mention what order processors should be added (if I add it manually)

Now Imagine I'm not interested in anything else but exceptions/snapshot's. How the system should be configured, what HttpModules must be in the web config and how ApplicationInsights.config should look like.

So, there are ton's of stuff in the config and Ideally I'l love to have docs about every single telemetry module/initializer/channel. What it is used for, what configuration params available, what scenarion I might need it and when not

General Comments: There were several problems I got in the last few days and the biggest is that now it is really really hard to troubleshoot AI. When it works - it is ok, but when it does not it is very hard to understand why.

Just for instance, I made several changes to me 'ApplicationInsights.config' and AI silently stopped working, I do not see eny errors in the telemetry events (( And even after I rolled back my changes to the pretty muck stock config telemetry still does not work and how to understand why it does not is not clear

pharring commented 5 years ago

I replied to your other question about troubleshooting ApplicationInsights.config problems (https://github.com/microsoft/ApplicationInsights-dotnet-server/issues/1243#issuecomment-514730374)

Here's the doc for ApplicationInsights.config: https://docs.microsoft.com/azure/azure-monitor/app/configuration-with-applicationinsights-config

For your scenario -- just Exceptions and Snapshots on .NET Framework, using PackageReference in the project file (instead of a packages.config) -- you need only a nuget reference to Microsoft.ApplicationInsights.SnapshotCollector (latest version). The base Application Insights SDK (Microsoft.ApplicationInsights) will automatically be included (since it's a dependency of SnapshotCollector), although I would recommend manually include the latest stable version of Microsoft.ApplicationInsights so you get all the bug fixes. The minimal ApplicationInsights.config for this scenario has just an InstrumentationKey and the TelemetryProcessors entry I mentioned above. However, that truly is the minimal config and probably isn't very useful in practice.

yahorsi commented 5 years ago

Interesting, I actually just have created sample (actually tried twice, for empty ASP.NET app and WebAPI app) ASP.NET app and add SnapshotCollector nuget package using UI. This is what I've got, have a look to the bottom of the config, it does not look correct to me.

<?xml version="1.0" encoding="utf-8"?>
<ApplicationInsights xmlns="http://schemas.microsoft.com/ApplicationInsights/2013/Settings">
    <TelemetryInitializers>
        <Add Type="Microsoft.ApplicationInsights.DependencyCollector.HttpDependenciesParsingTelemetryInitializer, Microsoft.AI.DependencyCollector"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.AzureRoleEnvironmentTelemetryInitializer, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.AzureWebAppRoleEnvironmentTelemetryInitializer, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.BuildInfoConfigComponentVersionTelemetryInitializer, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.Web.WebTestTelemetryInitializer, Microsoft.AI.Web"/>
        <Add Type="Microsoft.ApplicationInsights.Web.SyntheticUserAgentTelemetryInitializer, Microsoft.AI.Web">
            <!-- Extended list of bots:
            search|spider|crawl|Bot|Monitor|BrowserMob|BingPreview|PagePeeker|WebThumb|URL2PNG|ZooShot|GomezA|Google SketchUp|Read Later|KTXN|KHTE|Keynote|Pingdom|AlwaysOn|zao|borg|oegp|silk|Xenu|zeal|NING|htdig|lycos|slurp|teoma|voila|yahoo|Sogou|CiBra|Nutch|Java|JNLP|Daumoa|Genieo|ichiro|larbin|pompos|Scrapy|snappy|speedy|vortex|favicon|indexer|Riddler|scooter|scraper|scrubby|WhatWeb|WinHTTP|voyager|archiver|Icarus6j|mogimogi|Netvibes|altavista|charlotte|findlinks|Retreiver|TLSProber|WordPress|wsr-agent|http client|Python-urllib|AppEngine-Google|semanticdiscovery|facebookexternalhit|web/snippet|Google-HTTP-Java-Client-->
            <Filters>search|spider|crawl|Bot|Monitor|AlwaysOn</Filters>
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Web.ClientIpHeaderTelemetryInitializer, Microsoft.AI.Web"/>
        <Add Type="Microsoft.ApplicationInsights.Web.OperationNameTelemetryInitializer, Microsoft.AI.Web"/>
        <Add Type="Microsoft.ApplicationInsights.Web.OperationCorrelationTelemetryInitializer, Microsoft.AI.Web"/>
        <Add Type="Microsoft.ApplicationInsights.Web.UserTelemetryInitializer, Microsoft.AI.Web"/>
        <Add Type="Microsoft.ApplicationInsights.Web.AuthenticatedUserIdTelemetryInitializer, Microsoft.AI.Web"/>
        <Add Type="Microsoft.ApplicationInsights.Web.AccountIdTelemetryInitializer, Microsoft.AI.Web"/>
        <Add Type="Microsoft.ApplicationInsights.Web.SessionTelemetryInitializer, Microsoft.AI.Web"/>
    </TelemetryInitializers>
    <TelemetryModules>
        <Add Type="Microsoft.ApplicationInsights.DependencyCollector.DependencyTrackingTelemetryModule, Microsoft.AI.DependencyCollector">
            <ExcludeComponentCorrelationHttpHeadersOnDomains>
                <!-- 
        Requests to the following hostnames will not be modified by adding correlation headers.         
        Add entries here to exclude additional hostnames.
        NOTE: this configuration will be lost upon NuGet upgrade.
        -->
                <Add>core.windows.net</Add>
                <Add>core.chinacloudapi.cn</Add>
                <Add>core.cloudapi.de</Add>
                <Add>core.usgovcloudapi.net</Add>
                <Add>localhost</Add>
                <Add>127.0.0.1</Add>
            </ExcludeComponentCorrelationHttpHeadersOnDomains>
            <IncludeDiagnosticSourceActivities>
                <Add>Microsoft.Azure.EventHubs</Add>
                <Add>Microsoft.Azure.ServiceBus</Add>
            </IncludeDiagnosticSourceActivities>
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.PerformanceCollectorModule, Microsoft.AI.PerfCounterCollector">
            <!--
      Use the following syntax here to collect additional performance counters:

      <Counters>
        <Add PerformanceCounter="\Process(??APP_WIN32_PROC??)\Handle Count" ReportAs="Process handle count" />
        ...
      </Counters>

      PerformanceCounter must be either \CategoryName(InstanceName)\CounterName or \CategoryName\CounterName

      NOTE: performance counters configuration will be lost upon NuGet upgrade.

      The following placeholders are supported as InstanceName:
        ??APP_WIN32_PROC?? - instance name of the application process  for Win32 counters.
        ??APP_W3SVC_PROC?? - instance name of the application IIS worker process for IIS/ASP.NET counters.
        ??APP_CLR_PROC?? - instance name of the application CLR process for .NET counters.
      -->
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryModule, Microsoft.AI.PerfCounterCollector"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.AppServicesHeartbeatTelemetryModule, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.AzureInstanceMetadataTelemetryModule, Microsoft.AI.WindowsServer">
            <!--
      Remove individual fields collected here by adding them to the ApplicationInsighs.HeartbeatProvider 
      with the following syntax:

      <Add Type="Microsoft.ApplicationInsights.Extensibility.Implementation.Tracing.DiagnosticsTelemetryModule, Microsoft.ApplicationInsights">
        <ExcludedHeartbeatProperties>
          <Add>osType</Add>
          <Add>location</Add>
          <Add>name</Add>
          <Add>offer</Add>
          <Add>platformFaultDomain</Add>
          <Add>platformUpdateDomain</Add>
          <Add>publisher</Add>
          <Add>sku</Add>
          <Add>version</Add>
          <Add>vmId</Add>
          <Add>vmSize</Add>
          <Add>subscriptionId</Add>
          <Add>resourceGroupName</Add>
          <Add>placementGroupId</Add>
          <Add>tags</Add>
          <Add>vmScaleSetName</Add>
        </ExcludedHeartbeatProperties>
      </Add>

      NOTE: exclusions will be lost upon upgrade.
      -->
        </Add>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.DeveloperModeWithDebuggerAttachedTelemetryModule, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.UnhandledExceptionTelemetryModule, Microsoft.AI.WindowsServer"/>
        <Add Type="Microsoft.ApplicationInsights.WindowsServer.UnobservedExceptionTelemetryModule, Microsoft.AI.WindowsServer">
            <!--</Add>
    <Add Type="Microsoft.ApplicationInsights.WindowsServer.FirstChanceExceptionStatisticsTelemetryModule, Microsoft.AI.WindowsServer">-->
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Web.RequestTrackingTelemetryModule, Microsoft.AI.Web">
            <Handlers>
                <!-- 
        Add entries here to filter out additional handlers: 

        NOTE: handler configuration will be lost upon NuGet upgrade.
        -->
                <Add>Microsoft.VisualStudio.Web.PageInspector.Runtime.Tracing.RequestDataHttpHandler</Add>
                <Add>System.Web.StaticFileHandler</Add>
                <Add>System.Web.Handlers.AssemblyResourceLoader</Add>
                <Add>System.Web.Optimization.BundleHandler</Add>
                <Add>System.Web.Script.Services.ScriptHandlerFactory</Add>
                <Add>System.Web.Handlers.TraceHandler</Add>
                <Add>System.Web.Services.Discovery.DiscoveryRequestHandler</Add>
                <Add>System.Web.HttpDebugHandler</Add>
            </Handlers>
        </Add>
        <Add Type="Microsoft.ApplicationInsights.Web.ExceptionTrackingTelemetryModule, Microsoft.AI.Web"/>
        <Add Type="Microsoft.ApplicationInsights.Web.AspNetDiagnosticTelemetryModule, Microsoft.AI.Web"/>
    </TelemetryModules>
    <ApplicationIdProvider Type="Microsoft.ApplicationInsights.Extensibility.Implementation.ApplicationId.ApplicationInsightsApplicationIdProvider, Microsoft.ApplicationInsights"/>
    <TelemetrySinks>
        <Add Name="default">
            <TelemetryProcessors>
                <Add Type="Microsoft.ApplicationInsights.Extensibility.PerfCounterCollector.QuickPulse.QuickPulseTelemetryProcessor, Microsoft.AI.PerfCounterCollector"/>
                <Add Type="Microsoft.ApplicationInsights.Extensibility.AutocollectedMetricsExtractor, Microsoft.ApplicationInsights"/>
                <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
                    <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
                    <ExcludedTypes>Event</ExcludedTypes>
                </Add>
                <Add Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.AdaptiveSamplingTelemetryProcessor, Microsoft.AI.ServerTelemetryChannel">
                    <MaxTelemetryItemsPerSecond>5</MaxTelemetryItemsPerSecond>
                    <IncludedTypes>Event</IncludedTypes>
                </Add>
            </TelemetryProcessors>
            <TelemetryChannel Type="Microsoft.ApplicationInsights.WindowsServer.TelemetryChannel.ServerTelemetryChannel, Microsoft.AI.ServerTelemetryChannel"/>
        </Add>
    </TelemetrySinks>
<!-- 
    Learn more about Application Insights configuration with ApplicationInsights.config here: 
    http://go.microsoft.com/fwlink/?LinkID=513840

    Note: If not present, please add <InstrumentationKey>Your Key</InstrumentationKey> to the top of this file.
  -->
<TelemetryProcessors>
<Add Type="Microsoft.ApplicationInsights.SnapshotCollector.SnapshotCollectorTelemetryProcessor, Microsoft.ApplicationInsights.SnapshotCollector">
<!-- Snapshot Debugging is usually disabled when debugging in Visual Studio, but you can enable it by setting this to true. -->
<IsEnabledInDeveloperMode>false</IsEnabledInDeveloperMode>
<!-- Other properties are documented at https://aka.ms/pnv0qt -->
</Add>
</TelemetryProcessors>
</ApplicationInsights>

yahorsi commented 5 years ago

One more point, notice that: 

Now try to open the link and find 'other properties'...

pharring commented 5 years ago

I see. The TelemetryProcessors section at the bottom was added by the SnapshotCollector nuget and, unfortunately, it doesn't account for the fact that the TelemetryProcessors section has been moved inside TelemetrySinks. (This schema change happened a while ago in the base SDK, but SnapshotCollector wasn't modified to cope with it -- indeed, it may be impossible to fix due to the limitations of XML transforms). To fix it manually, you should move the <Add Type="Microsoft.ApplicationInsights.SnapshotCollector.SnapshotCollectorTelemetryProcessor ... element and its children so it's inside the <TelemetryProcessors> collection for the "default" TelemetrySink -- and then delete the (now empty) TelemetryProcessors section at the end.

The documentation for Snapshot Collector has expanded greatly since that comment (which again came from the SnapshotCollector nuget) was first authored. The aka.ms link points to the overview page (which used to be the only page we had). We recently discussed (internally) about adding a new page describing all the configuration options. When we do, we'll add a link from the overview.

@brahmnes, @jemodyMS FYI for documentation feedback.

yahorsi commented 5 years ago

Yep, there are few bugs, in nuget package installer and docs. But I'm still unable to see the snapshots (( Does TelemetryProcessors order matter and if yes where SnapshotCollectorTelemetryProcessor should be put in? Could you please share working sample with SnapshotCollectorTelemetryProcessor and all options available for the SnapshotCollectorTelemetryProcessor? It seems I'm not alone who can't make it working, there is one more https://stackoverflow.com/questions/55300791/application-insight-snapshot-debugger-snapshots-are-not-being-pushed-from-a-ne

brahmnes commented 5 years ago

Hi @yahorsi

Did the "Don't see the snapshot? troubleshoot" link show up for you? If not, it probably means snapshot collector is running, but you didn't see snapshots for some other reasons.

You can also check the troubleshooting guide to see if it helps: https://docs.microsoft.com/en-us/azure/azure-monitor/app/snapshot-debugger-troubleshoot

If you can share your application id (don't share the instrumentation key), then we can look at the telemetry to see what happened. The application id can be found by

Open the Azure portal
Navigate to your Application Insight resource
Click on the API Access on the menu on the left hand side
The application id should show up at the top

yuriyostapenko commented 4 years ago

So I've just spent almost a day trying to make snapshot debugger work in a legacy ASP.NET application and now it finally works! While there's no guarantee that the issue I've encountered is the same as yours was, I believe it is quite possible, because the symptoms I experienced were basically the same.

TL;DR: it is <assemblyBinding>.

Problem

Application Insights probably tries to be resilient to failures by preferring to (silently?) ignore them.

And that is what it does to SnapshotCollectorTelemetryProcessor. It simply silently skips it if it fails to load and proceeds happily without it. You can verify this by entering any garbage instead of Microsoft.ApplicationInsights.SnapshotCollector.SnapshotCollectorTelemetryProcessor, Microsoft.ApplicationInsights.SnapshotCollector as Type attribute value in ApplicationInsights.config.

And if you check TelemetryConfiguration.Active.DefaultTelemetrySink.TelemetryProcessors, it will simply miss those it failed to load.

A sure way to verify that problem is with assembly loading, just add this line somewhere in startup code:

var _ = typeof(SnapshotCollectorTelemetryProcessor);

That should trigger the error.

AFAIK, unlike with class library projects, Web.config in ASP.NET applications cannot benefit from <AutoGenerateBindingRedirects>, while NuGet package manager in VS won't do it either once you move to PackageReference (or to "new csproj with globs" like I did). So you have to maintain it somewhat manually.

"Fix"

As of today, assuming you have these reference versions in your csproj file,

<PackageReference Include="Microsoft.ApplicationInsights.SnapshotCollector" Version="1.3.7" />
<PackageReference Include="Microsoft.ApplicationInsights.Web" Version="2.14.0" />

you will probably need at least these redirects in Web.config

<assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
  <dependentAssembly>
    <assemblyIdentity name="Microsoft.ApplicationInsights" publicKeyToken="31bf3856ad364e35" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-2.14.0.17971" newVersion="2.14.0.17971" />
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="Microsoft.ApplicationInsights.SnapshotCollector" publicKeyToken="b03f5f7f11d50a3a" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-1.3.7.0" newVersion="1.3.7.0" />
  </dependentAssembly>
  <dependentAssembly>
    <assemblyIdentity name="System.Diagnostics.DiagnosticSource" publicKeyToken="cc7b13ffcd2ddd51" culture="neutral" />
    <bindingRedirect oldVersion="0.0.0.0-4.0.4.0" newVersion="4.0.4.0" />
  </dependentAssembly>
</assemblyBinding>

After adding those, I could see the SnapshotUploader directory created on disk and soon after simulating exceptions, some snapshots in the portal.

Extra quirks

As described above, the configuration by default is wrongly added to the root in ApplicationInsights.config and not under default sink.

Additionally, that sink configuration created by Microsoft.ApplicationInsights.Web contains duplicate entries for AdaptiveSamplingTelemetryProcessor, which will probably cause much less telemetry to be sampled than expected in some cases.

yahorsi commented 4 years ago

Tried on the latest VS and latest snapshots collector package and it still fails to install itself correctly. The AI config after package installation is corrupted

yahorsi commented 4 years ago

@uncleyo Could you please share your AI and packages configs?

Thanks!

yuriyostapenko commented 4 years ago

I haven't installed it into my existing solution. What I did is I created an "empty" ASP.NET MVC app from template, commited that to git and then added package via package manager to that, so I can see the diff. I then simply copied the resulting ApplicationInsights.config to my actual project and added the PackageReference's.

And yes, my actual solution is a "Frankenstein", which is a classic ASP.NET app with a .net core SDK based csproj (obviously targeting full framework). But that doesn't matter. As long as you get your assemblyBindings right, it should work.

I don't know what you mean when you say "it fails to install itself". Does nuget package manager fail to install package or what?

yahorsi commented 4 years ago

I don't know what you mean when you say "it fails to install itself". Does nuget package manager fail to install package or what? It's just an old bug that was not fixed yet. If you add SnapshotsCollector package it tries to modify ApplicationInsights.config but does it incorrectly that results to the broken config.

I've spent today whole day trying to make AI work for the classic app, and yes I've made simple empty app, add there AI, pushed and it does not work (( No events in the AI while I see them locally in the VS

ghost commented 3 years ago

Could you have the snapshot debugger option along with "Failure" and "Performance" Telemetry?

Objective: Enable Snapshot debugger to Diagnose session file (*.diagsession) for a specific timeframe hooked to an app service (basically debug prod)

Current State: We have Production application Insights with "Failures" and "Performance" Section Enabled - which mean we can view live telemetry.

Problem: We have experience prod exceptions which the current telemetry is unable to provide proper insights as to why they happens nor the root cause of them.

Problem with the implementation of the debugger (Microsoft Official Response): We cannot migrate the app to a workspace based without loosing the data available for Failures" and "Performance" Sections. Which means, we cannot have both solutions. Its either Debugger or Performance/Failures Metrices.

Looking for Insights or suggestions on a workaround or permanent solution to triage prod issues using the debugger or without, cause Azure AppInsights is so frustrating to use

(EFS stack implementation is scheduled for 2022, but until than, we need to manage)

Cheers!

References:

https://docs.microsoft.com/en-us/azure/azure-monitor/app/create-workspace-resource https://docs.microsoft.com/en-us/azure/azure-monitor/app/snapshot-debugger-vm?toc=/azure/azure-monitor/toc.json https://docs.microsoft.com/en-us/azure/azure-monitor/app/snapshot-debugger

github-actions[bot] commented 2 years ago

This issue is stale because it has been open 300 days with no activity. Remove stale label or this will be closed in 7 days. Commenting will instruct the bot to automatically remove the label.

pharring commented 2 years ago

I need to close this stale issue, but I see there are some unresolved comments above. @RoyDoron @yahorsi @uncleyo, if you still need help with Snapshot Collector, please open a new issue at https://github.com/microsoft/ApplicationInsights-SnapshotCollector/issues/new/choose and we'll try and address it there.

microsoft / ApplicationInsights-dotnet