Memory rising to 29Gb in one hour run

[niranjanhm-bh-git]

Steps to reproduce: List the minimal actions needed to reproduce the behavior.

1. Singleton InfluxDBClient instance used for all writes. Single writeApi used for all calls throughout

   client = new InfluxDBClient(url, Token);
   var writeOptions = new WriteOptions
           {
               BatchSize = 5000, 
               FlushInterval = 500, 
               JitterInterval = 0, 
           };
   converter = new DomainEntityConverter();
   writeApiConv = client.GetWriteApi(writeOptions, converter);

   // Below set data is called with other test client with load like: // 200 parellel calls with each call having 500 samples, this is equal to 1lakh tags (or point data). // Above load is called at every 1 second interval. // Observation: Good news (happy with) writes are pretty fast in ~100ms, 200 parallel calls finishes easily. // My main concern is memory rise. Below details are for 10 min run memory going to ~15GB

public async Task SetData(SetDataRequest dataRequest, string bucket = bucketTagAsMeasurementName_Conv)
        {
           var data = new List<StaticTagAsMeasurement2>();
            foreach (var sampleSet in dataRequest.SampleSets)
            {
                for (int sampleIndex = 0; sampleIndex < sampleSet.Samples.Count; sampleIndex++)
                {
                    var sample = sampleSet.Samples[sampleIndex];
                    data.Add(new StaticTagAsMeasurement2
                    {
                        Id = Guid.Parse(sampleSet.TagId),
                        HashId = sampleSet.TagId.GetHashCode(),
                        Value = double.Parse(sample.Value.ToString()),
                        DataStatus = sample.DataStatus,
                        NodeStatus = sample.NodeStatus,
                        IsValid = sample.DataStatus == 0,
                        Time = sample.Timestamp
                    });
                }
            }
            writeApiConv.WriteMeasurements(data, WritePrecision.Ns, bucket, orgName);
}

// I tried first with commented PointData approach & then with PointData.Builder approach - same memory rise with both.

DomainEntityConverter{
 public PointData ConvertToPointData<T>(T entity, WritePrecision precision)
        {
            if (entity is StaticTagAsMeasurement2 tag2)
            {
                var pointBuilder = PointData.Builder.Measurement(tag2.Id.ToString());
                pointBuilder = pointBuilder.Field("hashId", tag2.HashId);
                pointBuilder = pointBuilder.Field("value", tag2.Value);
                pointBuilder = pointBuilder.Field("isValid", tag2.IsValid);
                pointBuilder = pointBuilder.Field("dataStatus", tag2.DataStatus);
                pointBuilder = pointBuilder.Field("nodeStatus", tag2.NodeStatus);
                pointBuilder = pointBuilder.Timestamp(new DateTime(tag2.Time, DateTimeKind.Utc), precision);
                return pointBuilder.ToPointData();

               // var pointData = PointData
               //.Measurement(tag2.Id.ToString())
               //.Field("hashId", tag2.HashId);
                //.Field("value", tag2.Value)
                //.Field("isValid", tag2.IsValid)
                //.Field("dataStatus", tag2.DataStatus)
                //.Field("nodeStatus", tag2.NodeStatus)
                //.Timestamp(new DateTime(tag2.Time, DateTimeKind.Utc), precision);

                // return pointData;
            }
        }
}

Expected behavior: The memory should be stable say around 1GB so that long runs can be achieved

Actual behavior: Memory is rising very fast

Specifications:

• Client Version:
• InfluxDB Version: v2.6
• Platform: Intel(R) Core(TM) i7-9850H CPU @ 2.60GHz   2.59 GHz 64.0 GB (63.7 GB usable) 64-bit operating system, x64-based processor

[niranjanhm-bh-git]

Hi any update on the issue, we are doing a POC with Influx, without this support we wont be able to proceed

[niranjanhm-bh-git]

Kindly have a look at findings & if a quick update on planing to consider this as bug or not would help.

[bednar]

Hi @niranjanhm-bh-git,

It should be cause by Windows Defender Network Filter. For more info see https://github.com/influxdata/influxdb-client-csharp/issues/164#issuecomment-788752458.

Regards