AppMetrics / InfluxDB

App Metrics Extensions for InfluxDB reporting
https://www.app-metrics.io/reporting/reporters/influx-data/
Apache License 2.0
17 stars 13 forks source link

how to use it in NetCore2.0 webapi #22

Closed landonzeng closed 6 years ago

landonzeng commented 6 years ago

I want to use AppMetrics In my WebApi Application, metrics data write to influxdb,how I do? I use this code ,but my database have not data ,the Code : `using App.Metrics; using Microsoft.AspNetCore.Builder; using Microsoft.AspNetCore.Hosting; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using System;

namespace WebApplication1 { public class Startup { public Startup(IConfiguration configuration) { Configuration = configuration; }

    public IConfiguration Configuration { get; }

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {

        var metrics = new MetricsBuilder()
            .Configuration.Configure(
            options =>
            {
                options.WithGlobalTags((globalTags, info) =>
                {
                    globalTags.Add("app", "RepairApp");
                    globalTags.Add("env", "stage");
                    globalTags.Add("server", Environment.MachineName);
                });
            })
            .Report.ToInfluxDb(
                options =>
                {
                    options.InfluxDb.BaseUri = new Uri("http://127.0.0.1:9087");
                    options.InfluxDb.Database = "RepairAppInfluxData";
                    options.InfluxDb.UserName = "System";
                    options.InfluxDb.Password = "123456";
                    options.HttpPolicy.BackoffPeriod = TimeSpan.FromSeconds(30);
                    options.HttpPolicy.FailuresBeforeBackoff = 5;
                    options.HttpPolicy.Timeout = TimeSpan.FromSeconds(10);
                    options.FlushInterval = TimeSpan.FromSeconds(5);
                })
            .Build();
        services.AddMetrics(metrics);
        services.AddMetricsReportScheduler();

        services.AddMvc().AddJsonOptions(
            op =>
            op.SerializerSettings.ContractResolver
            = new Newtonsoft.Json.Serialization.DefaultContractResolver()
        );
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IHostingEnvironment env, IApplicationLifetime lifetime)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
        }

        app.UseCors(builder =>
        {
            builder.AllowAnyOrigin();
            builder.AllowAnyHeader();
            builder.AllowAnyMethod();
        });
        app.UseMvc();
    }
}

} `

alhardy commented 6 years ago

Have you looked at the sample project in this repo? https://github.com/AppMetrics/InfluxDB/tree/dev/sandbox/MetricsInfluxDBSandboxMvc

landonzeng commented 6 years ago

Thanks,My English is very poor! I use it in my project,but application is Error,the error message is: System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpClient.d58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at App.Metrics.Reporting.InfluxDB.Client.DefaultLineProtocolClient.d8.MoveNext()

The Startup.cs Code : `using App.Metrics; using Common.Model; using Microsoft.AspNetCore.Authentication.JwtBearer; using Microsoft.AspNetCore.Authorization; using Microsoft.AspNetCore.Builder; using Microsoft.AspNetCore.Diagnostics; using Microsoft.AspNetCore.Hosting; using Microsoft.AspNetCore.Http; using Microsoft.Extensions.Configuration; using Microsoft.Extensions.DependencyInjection; using Microsoft.Extensions.Logging; using Microsoft.IdentityModel.Tokens; using Newtonsoft.Json; using NLog.Extensions.Logging; using NLog.Web; using System;

namespace WebAPI { public class Startup { public Startup(IConfiguration configuration) { Configuration = configuration; }

    public IConfiguration Configuration { get; }

    // This method gets called by the runtime. Use this method to add services to the container.
    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMemoryCache();

        #region Metrics监控配置
        string IsOpen = Configuration.GetSection("InfluxDB")["IsOpen"].ToLower();
        if (IsOpen == "true")
        {
            string database = Configuration.GetSection("InfluxDB")["DataBaseName"];
            string InfluxDBConStr = Configuration.GetSection("InfluxDB")["ConnectionString"];
            string app = Configuration.GetSection("InfluxDB")["app"];
            string env = Configuration.GetSection("InfluxDB")["env"];
            string username = Configuration.GetSection("InfluxDB")["username"];
            string password = Configuration.GetSection("InfluxDB")["password"];

            var uri = new Uri(InfluxDBConStr);

            var metrics = AppMetrics.CreateDefaultBuilder()
            .Configuration.Configure(
            options =>
            {
                options.AddAppTag(app);
                options.AddEnvTag(env);
            })
            .Report.ToInfluxDb(
            options =>
            {
                options.InfluxDb.BaseUri = uri;
                options.InfluxDb.Database = database;
                options.InfluxDb.UserName = username;
                options.InfluxDb.Password = password;
                options.HttpPolicy.BackoffPeriod = TimeSpan.FromSeconds(30);
                options.HttpPolicy.FailuresBeforeBackoff = 5;
                options.HttpPolicy.Timeout = TimeSpan.FromSeconds(10);
                options.FlushInterval = TimeSpan.FromSeconds(5);
            })
            .Build();

            services.AddMetrics(metrics);
            services.AddMetricsReportScheduler();
            services.AddMetricsTrackingMiddleware();
            services.AddMetricsEndpoints();

        }
        #endregion

        services.AddMvc().AddJsonOptions(op =>op.SerializerSettings.ContractResolver= new Newtonsoft.Json.Serialization.DefaultContractResolver());
    }

    // This method gets called by the runtime. Use this method to configure the HTTP request pipeline.
    public void Configure(IApplicationBuilder app, IHostingEnvironment env, ILoggerFactory loggerFactory)
    {
        if (env.IsDevelopment())
        {
            app.UseDeveloperExceptionPage();
            loggerFactory.AddConsole(Configuration.GetSection("Logging"));
            loggerFactory.AddDebug();
        }

        loggerFactory.AddNLog();
        env.ConfigureNLog("nlog.config");
        app.AddNLogWeb();

        #region 注入Metrics
        string IsOpen = Configuration.GetSection("InfluxDB")["IsOpen"].ToLower();
        if (IsOpen == "true")
        {
            app.UseMetricsAllMiddleware();
            // Or to cherry-pick the tracking of interest
            app.UseMetricsActiveRequestMiddleware();
            app.UseMetricsErrorTrackingMiddleware();
            app.UseMetricsPostAndPutSizeTrackingMiddleware();
            app.UseMetricsRequestTrackingMiddleware();
            app.UseMetricsOAuth2TrackingMiddleware();
            app.UseMetricsApdexTrackingMiddleware();

            app.UseMetricsAllEndpoints();
            // Or to cherry-pick endpoint of interest
            app.UseMetricsEndpoint();
            app.UseMetricsTextEndpoint();
            app.UseEnvInfoEndpoint();
        }
        #endregion

        app.UseCors(builder =>
        {
            builder.AllowAnyOrigin();
            builder.AllowAnyHeader();
            builder.AllowAnyMethod();
        });

        app.UseMvc();
    }
}

} `

alhardy commented 6 years ago

Looks like your application can't connect to InfluxDB? i.e. Getting a task cancelled exception in DefaultLineProtocolClient

landonzeng commented 6 years ago

I run the application, query the Influxdb database, found that the application can be inserted into the Influxdb, after running for a period of time, received the wrong information, will be the Influxdb lock table lock database causes? Is it related to the configuration of Influxdb?

landonzeng commented 6 years ago

My InfluxDB Config Code: ##`# Welcome to the InfluxDB configuration file.

# The values in this file override the default values used by the system if # a config option is not specified. The commented out lines are the configuration # field and the default value used. Uncommenting a line and changing the value # will change the value used at runtime when the process is restarted.

# Once every 24 hours InfluxDB will report usage data to usage.influxdata.com # The data includes a random ID, os, arch, version, the number of series and other # usage data. No data from user databases is ever transmitted. # Change this option to true to disable reporting. `# reporting-disabled = false

`# Bind address to use for the RPC service for backup and restore. bind-address = "127.0.0.1:9088"

### ### [meta] ### ### Controls the parameters for the Raft consensus group that stores metadata ### about the InfluxDB cluster. ###

[meta] `# Where the metadata/raft database is stored dir = "influxdb/meta"

# Automatically create a default retention policy when creating a database. # retention-autocreate = true

# If log messages are printed for the meta service # logging-enabled = true

### ### [data] ### ### Controls where the actual shard data for InfluxDB lives and how it is ### flushed from the WAL. "dir" may need to be changed to a suitable place ### for your system, but the WAL settings are an advanced configuration. The ### defaults should work for most systems. ###

[data] `# The directory where the TSM storage engine stores TSM files. dir = "influxdb/data"

`# The directory where the TSM storage engine stores WAL files. wal-dir = "influxdb/wal"

# The amount of time that a write will wait before fsyncing. A duration # greater than 0 can be used to batch up multiple fsync calls. This is useful for slower # disks or when WAL write contention is seen. A value of 0s fsyncs every write to the WAL. # Values in the range of 0-100ms are recommended for non-SSD disks. `# wal-fsync-delay = "0s"

# The type of shard index to use for new shards. The default is an in-memory index that is # recreated at startup. A value of "tsi1" will use a disk based index that supports higher # cardinality datasets. # index-version = "inmem"

# Trace logging provides more verbose output around the tsm engine. Turning # this on can provide more useful output for debugging tsm engine issues. `# trace-logging-enabled = false

# Whether queries should be logged before execution. Very useful for troubleshooting, but will # log any sensitive data contained within a query. `# query-log-enabled = true

`# Settings for the TSM engine

# CacheMaxMemorySize is the maximum size a shard's cache can # reach before it starts rejecting writes. `# cache-max-memory-size = 1048576000

# CacheSnapshotMemorySize is the size at which the engine will # snapshot the cache and write it to a TSM file, freeing up memory `# cache-snapshot-memory-size = 26214400

# CacheSnapshotWriteColdDuration is the length of time at # which the engine will snapshot the cache and write it to # a new TSM file if the shard hasn't received writes or deletes # cache-snapshot-write-cold-duration = "10m"

# CompactFullWriteColdDuration is the duration at which the engine # will compact all TSM files in a shard if it hasn't received a # write or delete # compact-full-write-cold-duration = "4h"

# The maximum number of concurrent full and level compactions that can run at one time. A # value of 0 results in runtime.GOMAXPROCS(0) used at runtime. This setting does not apply # to cache snapshotting. # max-concurrent-compactions = 0

# The maximum series allowed per database before writes are dropped. This limit can prevent # high cardinality issues at the database level. This limit can be disabled by setting it to # 0. # max-series-per-database = 1000000

# The maximum number of tag values per tag that are allowed before writes are dropped. This limit # can prevent high cardinality tag values from being written to a measurement. This limit can be # disabled by setting it to 0. # max-values-per-tag = 100000

### ### [coordinator] ### ### Controls the clustering service configuration. ##`#

[coordinator] # The default time a write request will wait until a "timeout" error is returned to the caller. # write-timeout = "10s"

# The maximum number of concurrent queries allowed to be executing at one time. If a query is # executed and exceeds this limit, an error is returned to the caller. This limit can be disabled # by setting it to 0. # max-concurrent-queries = 0

# The maximum time a query will is allowed to execute before being killed by the system. This limit # can help prevent run away queries. Setting the value to 0 disables the limit. `# query-timeout = "0s"

# The time threshold when a query will be logged as a slow query. This limit can be set to help # discover slow or resource intensive queries. Setting the value to 0 disables the slow query logging. `# log-queries-after = "0s"

# The maximum number of points a SELECT can process. A value of 0 will make # the maximum point count unlimited. This will only be checked every 10 seconds so queries will not # be aborted immediately when hitting the limit. # max-select-point = 0

# The maximum number of series a SELECT can run. A value of 0 will make the maximum series # count unlimited. `# max-select-series = 0

# The maxium number of group by time bucket a SELECT can create. A value of zero will max the maximum # number of buckets unlimited. `# max-select-buckets = 0

### ### [retention] ### ### Controls the enforcement of retention policies for evicting old data. ##`#

[retention] # Determines whether retention policy enforcement enabled. # enabled = true

# The interval of time when retention policy enforcement checks run. # check-interval = "30m"

### ### [shard-precreation] ### ### Controls the precreation of shards, so they are available before data arrives. ### Only shards that, after creation, will have both a start- and end-time in the ### future, will ever be created. Shards are never precreated that would be wholly ##`# or partially in the past.

[shard-precreation] # Determines whether shard pre-creation service is enabled. # enabled = true

# The interval of time when the check to pre-create new shards runs. # check-interval = "10m"

# The default period ahead of the endtime of a shard group that its successor # group is created. `# advance-period = "30m"

### ### Controls the system self-monitoring, statistics and diagnostics. ### ### The internal database for monitoring data is created automatically if ### if it does not already exist. The target retention within this database ### is called 'monitor' and is also created with a retention period of 7 days ### and a replication factor of 1, if it does not exist. In all cases the ### this retention policy is configured as the default for the database.

[monitor] # Whether to record statistics internally. # store-enabled = true

# The destination database for recorded statistics # store-database = "_internal"

# The interval at which to record statistics # store-interval = "10s"

### ### [http] ### ### Controls how the HTTP endpoints are configured. These are the primary ### mechanism for getting data into and out of InfluxDB. ###

[http] `# Determines whether HTTP endpoint is enabled. enabled = true

`# The bind address used by the HTTP service. bind-address = ":9086"

`# Determines whether user authentication is enabled over HTTP/HTTPS. auth-enabled = true

# The default realm sent back when issuing a basic auth challenge. # realm = "InfluxDB"

# Determines whether HTTP request logging is enabled. # log-enabled = true

# Determines whether detailed write logging is enabled. # write-tracing = false

# Determines whether the pprof endpoint is enabled. This endpoint is used for # troubleshooting and monitoring. `# pprof-enabled = true

# Determines whether HTTPS is enabled. # https-enabled = false

# The SSL certificate to use when HTTPS is enabled. # https-certificate = "/etc/ssl/influxdb.pem"

# Use a separate private key location. # https-private-key = ""

# The JWT auth shared secret to validate requests using JSON web tokens. # shared-secret = ""

# The default chunk size for result sets that should be chunked. # max-row-limit = 0

# The maximum number of HTTP connections that may be open at once. New connections that # would exceed this limit are dropped. Setting this value to 0 disables the limit. `# max-connection-limit = 0

# Enable http service over unix domain socket # unix-socket-enabled = false

# The path of the unix domain socket. # bind-socket = "/var/run/influxdb.sock"

### ### [subscriber] ### ### Controls the subscriptions, which can be used to fork a copy of all data ### received by the InfluxDB host. ###

[subscriber] # Determines whether the subscriber service is enabled. # enabled = true

# The default timeout for HTTP writes to subscribers. # http-timeout = "30s"

# Allows insecure HTTPS connections to subscribers. This is useful when testing with self- # signed certificates. `# insecure-skip-verify = false

# The path to the PEM encoded CA certs file. If the empty string, the default system certs will be used # ca-certs = ""

# The number of writer goroutines processing the write channel. # write-concurrency = 40

# The number of in-flight writes buffered in the write channel. # write-buffer-size = 1000

### ### [[graphite]] ### ### Controls one or many listeners for Graphite data. ##`#

[[graphite]] # Determines whether the graphite endpoint is enabled. # enabled = false # database = "graphite" # retention-policy = "" # bind-address = ":2003" # protocol = "tcp" `# consistency-level = "one"

# These next lines control how batching works. You should have this enabled # otherwise you could get dropped metrics or poor performance. Batching `# will buffer points in memory if you have many coming in.

# Flush if this many points get buffered # batch-size = 5000

# number of batches that may be pending in memory # batch-pending = 10

# Flush at least this often even if we haven't hit buffer limit # batch-timeout = "1s"

# UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max. # udp-read-buffer = 0

### This string joins multiple matching 'measurement' values providing more control over the final measurement name. # separator = "."

### Default tags that will be added to all metrics. These can be overridden at the template level ### or by tags extracted from metric `# tags = ["region=us-east", "zone=1c"]

### Each template line requires a template pattern. It can have an optional ### filter before the template and separated by spaces. It can also have optional extra ### tags following the template. Multiple tags should be separated by commas and no spaces ### similar to the line protocol format. There can be only one default template. # templates = [ # ".app env.service.resource.measurement", ## Default template `# "server.", `# ]

### ### [collectd] ### ### Controls one or many listeners for collectd data. ##`#

[[collectd]] # enabled = false # bind-address = ":25826" # database = "collectd" # retention-policy = "" # # The collectd service supports either scanning a directory for multiple types # db files, or specifying a single db file. # typesdb = "/usr/local/share/collectd" # # security-level = "none" `# auth-file = "/etc/collectd/auth_file"

# These next lines control how batching works. You should have this enabled # otherwise you could get dropped metrics or poor performance. Batching `# will buffer points in memory if you have many coming in.

# Flush if this many points get buffered # batch-size = 5000

# Number of batches that may be pending in memory # batch-pending = 10

# Flush at least this often even if we haven't hit buffer limit # batch-timeout = "10s"

# UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max. # read-buffer = 0

### ### [opentsdb] ### ### Controls one or many listeners for OpenTSDB data. ##`#

[[opentsdb]] # enabled = false # bind-address = ":4242" # database = "opentsdb" # retention-policy = "" # consistency-level = "one" # tls-enabled = false `# certificate= "/etc/ssl/influxdb.pem"

# Log an error for every malformed point. # log-point-errors = true

# These next lines control how batching works. You should have this enabled # otherwise you could get dropped metrics or poor performance. Only points `# metrics received over the telnet protocol undergo batching.

# Flush if this many points get buffered # batch-size = 1000

# Number of batches that may be pending in memory # batch-pending = 5

# Flush at least this often even if we haven't hit buffer limit # batch-timeout = "1s"

### ### [[udp]] ### ### Controls the listeners for InfluxDB line protocol data via UDP. ##`#

[[udp]] # enabled = false bind-address = ":9089" # database = "udp" `# retention-policy = ""

# These next lines control how batching works. You should have this enabled # otherwise you could get dropped metrics or poor performance. Batching `# will buffer points in memory if you have many coming in.

# Flush if this many points get buffered # batch-size = 5000

# Number of batches that may be pending in memory # batch-pending = 10

# Will flush at least this often even if we haven't hit buffer limit # batch-timeout = "1s"

# UDP Read buffer size, 0 means OS default. UDP listener will fail if set above OS max. # read-buffer = 0

### ### [continuous_queries] ### ### Controls how continuous queries are run within InfluxDB. ##`#

[continuous_queries] # Determines whether the continuous query service is enabled. # enabled = true

# Controls whether queries are logged when executed by the CQ service. # log-enabled = true

# interval for how often continuous queries will be checked if they need to run # run-interval = "1s"

alhardy commented 6 years ago

What is "received the wrong information" referring too? Incorrect data?

Are these the default InfluxDB settings?

landonzeng commented 6 years ago

wrong information is: System.Threading.Tasks.TaskCanceledException: A task was canceled. at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1.ConfiguredTaskAwaiter.GetResult() at System.Net.Http.HttpClient.d58.MoveNext() --- End of stack trace from previous location where exception was thrown --- at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at App.Metrics.Reporting.InfluxDB.Client.DefaultLineProtocolClient.d8.MoveNext()

InfluxDB settings is default Config

landonzeng commented 6 years ago

Influxdb and .NetCore WebAPi running on Windows Server 2008 R2

alhardy commented 6 years ago

Unfortunately I cannot replicate your issue, I've ran the 2.0 InfluxDB sample app for 4 hours without issues and also I am using it for a production app, attempting to publish a metrics snapshot to snapshot.raintank.io but it's taking a while.

Are testing on your local or in a production environment, if you're in a prod environment how are you hosting InfluxDB?

landonzeng commented 6 years ago

This is my application's error log: http://studyw.cn/Content/20171106.html

landonzeng commented 6 years ago

At the beginning, I deployed InfluxDB on the Windows system, and now I'm deploying InfluxDB in Linux, but it's the same error

alhardy commented 6 years ago

Can you try cloning the sample project and pointing it to your influxdb instance without any auth?

landonzeng commented 6 years ago

ok, I'll try it,thanks!

landonzeng commented 6 years ago

I ran the sample project and didn't report error message, but the project was.Net core 1.2, and my project was no problem at.Net core 1.2 version, and then when I upgraded to core 2, the error message came out

alhardy commented 6 years ago

Interesting because the sample project is dotnet core 2....

alhardy commented 6 years ago

Sandbox project here is dotnet core 2.0 and runs fine.

Without seeing your project not sure where else to assist on this one unfortunately

RouR commented 6 years ago

https://github.com/AppMetrics/InfluxDB/blob/a0364f98d2d7d86db9494a00f5b8e49e0619f210/sandbox/MetricsInfluxDBSandboxMvc/Startup.cs#L32

Compilation error. .AddMetrics() is unknown extension.

alhardy commented 6 years ago

@RouR that change has not yet been released, I'm going to guess you've run the solution pulling packages from nuget rather than the latest Dev packages on myget?

alhardy commented 6 years ago