influxdata / influxdb-client-csharp

InfluxDB 2.x C# Client
https://influxdata.github.io/influxdb-client-csharp/api/InfluxDB.Client.html
MIT License
355 stars 95 forks source link

C# InfluxDBQueryable problem with AggregateWindow #614

Closed ManuzXo closed 7 months ago

ManuzXo commented 7 months ago

AggregateWindow position bug Linq to Flux: Then I'm having a bug most likely with the output of the flux query via linq, essentially constructing the expression dynamically, I don't know why the expression with the Aggregate condition is put before the filter of the columns, and this leads to do the aggregation with all the columns in the table. I also have a link to stack overflow [https://stackoverflow.com/questions/77954064/influx-db-c-sharp-linq-problem-influxdbqueryable]()

C#

 var _queryable = InfluxDBQueryable<Record>.Queryable(bucketName, Organization, queryApi == null ? queryApiTest : queryApi, memberResolver, queryableOptimizerSettings: new QueryableOptimizerSettings
                {
                    AlignFieldsWithPivot = false,
                    DropMeasurementColumn = false,
                    DropStartColumn = false,
                    DropStopColumn = false,

                });
                if (start != null && end != null)
                    _queryable =  _queryable.Where(x => x.Timestamp >= start && x.Timestamp <= end ).ToInfluxQueryable();

                _queryable  = _queryable.Where(x => x.Measurament == Measurament).ToInfluxQueryable();
                if (Node != null)
                    _queryable = _queryable.Where(x => x.Node == Node).ToInfluxQueryable();
                if(Tag != null )
                    _queryable = _queryable.Where(x => x.Tag == Tag).ToInfluxQueryable();
                if(SubTag != null)
                    _queryable = _queryable.Where(x => x.SubTag == SubTag).ToInfluxQueryable();

                if (columnValue != null && columnValue.Count > 0)
                {
                    Expression<Func<Record, bool>> _combinedExpression = null;
                    foreach (var _col in columnValue)
                    {
                        if (_col.Value == null)
                        {
                            if (_combinedExpression == null)
                                _combinedExpression = x => x._field == _col.Key;
                            else
                                _combinedExpression = _combinedExpression.OR(x => x._field == _col.Key);// x.Attributes.Any(b => b.Name == _col.Key && b.Value != null));

                        }
                        else
                        {
                            if (_combinedExpression == null)
                                _combinedExpression = x => x._field == _col.Key && x._value == _col.Value;
                            else
                                _combinedExpression = _combinedExpression.OR(x => x._field == _col.Key && x._value == _col.Value);
                        }
                    }
                    _queryable = _queryable.Where(_combinedExpression).ToInfluxQueryable();
                }
                if (aggregateWindow != TimeSpan.MinValue)
                    _queryable = _queryable.Where(x => x.Timestamp.AggregateWindow(aggregateWindow, null, aggregateMethod)).ToInfluxQueryable();

                var _querys = _queryable.ToDebugQuery();
                Debug.WriteLine(_querys.ToJson());
                Debug.WriteLine(string.Join("\n|>", _querys._Query.Split(new string[] { "|>" }, StringSplitOptions.RemoveEmptyEntries)));

Flux Query Output

from(bucket: p1) |>range(start: time(v: start_shifted), stop: time(v: stop_shifted)) |>filter(fn: (r) => (r["_measurement"] == p5) and (r["node"] == p6) and (r["tag"] == p7) and (r["SubTag"] == p8)) |>aggregateWindow(every: p11, fn: p12) |>filter(fn: (r) =>(r["_field"] == p9) or (r["_field"] == p10))

bednar commented 7 months ago

Hello @ManuzXo,

Thank you for using our client and for bringing this to our attention.

You've correctly observed that our LINQ expression processor is designed to append the aggregator window before applying field filters. This design decision supports the primary use case involving the pivot function, which is a common pattern in data transformations by LINQ. Noticing from your example, you have AlignFieldsWithPivot = false, which deviates from this typical use case.

We understand that this behavior might not suit all scenarios, including yours. Given this, we’re very open to exploring adjustments that could make our client more versatile and accommodating of different use cases like yours.

Would you be interested in contributing to this effort? We warmly welcome Pull Requests (PRs) from our user community and would be delighted to review your submission. Your input could greatly benefit the project, making it more flexible and useful for a broader audience.

If you decide to contribute or have any questions about the process, please don't hesitate to reach out. We’re here to support you every step of the way.

Best Regards

ManuzXo commented 7 months ago

@bednar Thank you very much for the answer, in summary at the moment I have fixed the problem by writing the queries with flux, more than anything else for working times, in this case when I will have the opportunity to get my hands on the client I will do it with pleasure!