influxdata / influxdb-java

Java client for InfluxDB
MIT License
1.18k stars 477 forks source link

Does influxdb-java has limitation for the # of select queries in one query() call? #930

Open strNewBee opened 1 year ago

strNewBee commented 1 year ago

Environment

java 1.8
SpringBoot 2.4.3
SpringCloud 3.0.5
Influxdb-java 2.20
Influxdb 1.8

Platform

client app: windows 10
influxdb server: centos 7

Question

My influxdb database stores the data from May 2nd to May 5th, and the query string is as below, which contains 5 select:

SELECT metricsA FROM measurementA WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002' AND sign=~/.*/) OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a' AND sign=~/1.1|1.2/)) GROUP BY uniqId,sign tz('Etc/GMT-8');
SELECT metricsB FROM measurementB WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002' AND sign=~/.*/) OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a' AND sign=~/1.1|1.2/)) GROUP BY uniqId,sign tz('Etc/GMT-8');
SELECT metricsC,metricsD FROM measurementC WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002' AND sign=~/.*/) OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a' AND sign=~/1.1|1.2/)) GROUP BY uniqId,sign tz('Etc/GMT-8');
SELECT metricsE FROM measurementD WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002') OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a')) GROUP BY uniqId tz('Etc/GMT-8');
SELECT metricsF FROM measurementE WHERE time >= '2023-05-02T10:10:30.78+08:00' AND time < '2023-05-20T08:20:30.78+08:00' AND ((uniqId = 'cadd3c08082f07740c459d4366b91002' AND sign=~/.*/) OR (uniqId = 'c0c0250e096735acc6b6dc341b47504a' AND sign=~/1.1|1.2/)) GROUP BY uniqId,sign tz('Etc/GMT-8');

I'm calling InfluxDB.query() to execute it, but only the first 3 select query returns QueryResult. The Json format of QueryResult is:

"result":[
    {
        "series": [
                    {
                        "name": "measurementA",
                        "tags": {
                            "sign": "1.1",
                            "uniqId": "c0c0250e096735acc6b6dc341b47504a",
                            "IP": "4.4.4.4"
                        },
                        "columns": [
                            "time",
                            "PSUIn"
                        ],
                        "values": [
                            [
                                "2023-05-03T08:04:00+08:00",
                                900.0
                            ],
                            ...
                        ]
                    }
          ]
    },
    {...},
    {...}
]

When I execute each select query seperately, they can all return QueryResults. And there is more, which is when I narrowed down the time range to { 2023-05-02T10:10:30.78+08:00 ---- 2023-05-04T10:10:30.78+08:00 } instead of { 2023-05-02T10:10:30.78+08:00 ---- 2023-05-20T08:20:30.78+08:00 }, it works just fine and gives all the results for 5 queries.

I had no clue why this happened. Is there any constrants for the number of select query in one query() call?

strNewBee commented 1 year ago

Now i can only query 3 select in a batch, and I'm worrying about the extra I/O may slow it down. When I set batch up to more than 3, it still gets this problem by missing some queries.

public static QueryResult queryMetricsByIpRegexV2(String start, String end,
                                                    Map<QueryMapEntity, List<String>> tableMap,
                                                    Map<String, String> rs2ip,
                                                    Map<String, String> ip2regex)
    {
        /* basic where */
        StringBuilder whereBase = new StringBuilder();
        StringBuilder objBase = new StringBuilder();
        StringBuilder signBase = new StringBuilder();
        createTimeBaseRFC(start, end, whereBase);

        /* create a filter */
        objBase.append(" AND (").append(createSignFilterRegex(rs2ip, ip2regex, false)).append(")");
        signBase.append(" AND (").append(createSignFilterRegex(rs2ip, ip2regex, true)).append(")");

        /* generate queries */
        StringBuilder queryCmd = new StringBuilder();
        long queryCount = 0L;
        List<String> queryCmdList = new ArrayList<>();
        for(Map.Entry<QueryMapEntity, List<String>> pair: tableMap.entrySet()) {
            QueryMapEntity target = pair.getKey();
            Boolean is_sign = target.getIs_sign();
            SelectQueryImpl query = QueryBuilder.select(pair.getValue().toArray())
                    .from(target.getDbName(), target.getTableName())
                    .where(whereBase.toString() + (is_sign ? signBase.toString() : objBase.toString()))
                    .groupBy((Object[]) (is_sign ? groupBySign : groupByUniq))
                    .tz("Etc/GMT-8");
            queryCmd.append(query.getCommand());
            queryCount += 1;
            if(queryCount == CommonConstant.queryBatch){  // 3 in a batch by default
                queryCmdList.add(queryCmd.toString());
                queryCmd.setLength(0);
                queryCount = 0L;
            }
        }
        if(queryCount != 0)
            queryCmdList.add(queryCmd.toString());

        /* perform queries */
        QueryResult result = null;
        for(String queryStr: queryCmdList){
            log.info("influxdb query: {}", queryStr);
            if(result == null)
                result = InfluxDBUtil.query(queryStr);
            else{
                for(QueryResult.Result res: InfluxDBUtil.query(queryStr).getResults()){
                    result.getResults().add(res);
                }
            }
        }
        return result;
    }
strNewBee commented 1 year ago

Grateful for any help. (T T)