Range query bug on pilosa cluster

[yangfan1224]

For bugs, please provide the following:

Expected behavior

I create a time type field, TimeQuantum is TimeQuantum.DAY, then write some simple data with different date, when I query data using Range API with specific date param, the query reuslt should be I write on the specific date.

Actual behavior

after writing data into pilosa, I can get correct result for some minutes, but if i go on querying, no matter the date param is , the result is all data write in different date views .

Steps to reproduce the behavior

to reproduce the behavior , i write a simple test code as follows, you can run it untill the bug produce.

public class PilosaClientExample {

    public static Long[] union(Long[] arr1, Long[] arr2) {
        Set<Long> set = new HashSet<Long>();

        for (Long str : arr1) {
            set.add(str);
        }

        for (Long str : arr2) {
            set.add(str);
        }

        Long[] result = {  };

        return set.toArray(result);
    }

    public static boolean RangeQueryValidation(PilosaClient client, Field field, Calendar start, Calendar end, Long [] expectedColumns){
        final boolean[] result = {true};
        PqlRowQuery rawQuery = field.range(2, start.getTime(), end.getTime());
        QueryResponse response = client.query(rawQuery, QueryOptions.builder().setExcludeColumns(false).build());
        response.getResults().forEach(new Consumer<QueryResult>() {
            @Override
            public void accept(QueryResult queryResult) {
                List<Long> columnsList = queryResult.getRow().getColumns();
                boolean isValid = Arrays.equals(expectedColumns, columnsList.toArray(new Long[0]));
                if (!isValid){
                    String outstr = String.format("range query expected columns result is %s \n" +
                                    "query columns result is                %s",
                            Arrays.toString(expectedColumns), columnsList );
                    System.out.println(outstr);
                }
                result[0] = isValid;
            }
        });
        return result[0];
    }

    public static void main(String [] args){
        Cluster cluster = Cluster.withHost(
                URI.address("servicenode06:10101"),
                URI.address("servicenode07:10101"),
                URI.address("servicenode08:10111"),
                URI.address("servicenode01:10111"),
                URI.address("servicenode05:10111")
        );

        ClientOptions options = ClientOptions.builder()
                .setConnectTimeout(1000)  // if can't connect in  a second, close the connection
                .setSocketTimeout(10000)  // if no response received in 10 seconds, close the connection
                .setConnectionPoolSizePerRoute(3)  // number of connections in the pool per host
                .setConnectionPoolTotalSize(10)  // number of total connections in the pool
                .setRetryCount(5)  // number of retries before failing the request
                .build();

        PilosaClient client = PilosaClient.withCluster(cluster, options);
        Schema schema = client.readSchema();
        Index index = schema.index("test01");
        FieldOptions cateOptions = FieldOptions.builder()
                .fieldTime(TimeQuantum.DAY)
                .build();
        Field cate1 = index.field("cate1",cateOptions);
        client.syncSchema(schema);
        PqlBatchQuery query = index.batchQuery();
        int count = 500;

        Calendar cal1 = Calendar.getInstance();
        cal1.set(2018,7,1);
        System.out.println(cal1.getTime());
        Long [] expected20180801 = new Long[count];
        for (int i = 0; i < count; i++) {
            expected20180801[i] = Long.valueOf(i);
            query.add(cate1.set(2, i, cal1.getTime()));
        }

        cal1.set(2018,7,5);
        System.out.println(cal1.getTime());
        Long [] expected20180805 = new Long[count];
        int arrayIndex = 0;
        for (int i = 300; i < 800; i++) {
            expected20180805[arrayIndex++] = Long.valueOf(i);
            query.add(cate1.set(2, i, cal1.getTime()));
        }

        cal1.set(2018,7,10);
        System.out.println(cal1.getTime());
        Long [] expected20180810 = {Long.valueOf(99999)};
        query.add(cate1.set(2,99999, cal1.getTime()));
        client.query(query);

        Long [] expected2018080110 = union(union(expected20180801,expected20180805),expected20180810);
        Arrays.sort(expected2018080110);
        System.out.println(Arrays.toString(expected2018080110));

        System.out.println("validation start at:" +Calendar.getInstance().getTime());

        while(true){
            // range query validation
            Calendar start = Calendar.getInstance();
            Calendar end = Calendar.getInstance();

            //query for data on 2018-08-05
            start.set(2018,7,5,0,0);
            end.set(2018,7,6,0,0);
            if(!RangeQueryValidation(client,cate1,start,end,expected20180805))break;

            //query for data on 2018-08-01
            start.set(2018,7,1,0,0);
            end.set(2018,7,2,0,0);
            if(!RangeQueryValidation(client,cate1,start,end,expected20180801))break;

            //query for data on 2018-08-10
            start.set(2018,7,10,0,0);
            end.set(2018,7,11,0,0);
            if(!RangeQueryValidation(client,cate1,start,end,expected20180810))break;

            //query for data on 2018-08-01 ~ 2018-08-10
            start.set(2018,7,1,0,0);
            end.set(2018,7,11,0,0);
            if(!RangeQueryValidation(client,cate1,start,end,expected2018080110))break;

            try {
                Thread.sleep(1000);
            } catch (InterruptedException e) {
                e.printStackTrace();
            }
        }
        System.out.println("validation finish at:" +Calendar.getInstance().getTime());

    }
}

Information about your environment (OS/architecture, CPU, RAM, cluster/solo, configuration, etc.)

OS/architecture CentOS7.4 linux-amd64 pilosa: 5 nodes cluster pilosa version: v1.0.2 or mater code until 13 Aug. configuration: data-dir = "/data01/pilosa/data" bind = "servicenode08:10101" verbose = true [gossip] port = 22000 seeds = "servicenode08:22000"

[cluster] replicas = 2 coordinator = true

For feature requests, please provide the following:

Description

during search on this issue, I foud that this bug only show on pilosa cluster, not a single node, and it can show correct results for about five to ten minutes, so i guess the bug is caused by data sync between cluster nodes.

Success criteria (What criteria will consider this ticket closeable?)

[yuce]

Hi @yangfan1224 . Thanks for your issue report. I will try to re-create the issue here. What version of the Pilosa Java client are you using? A released one, or from the master branch or one of the PRs ?

[yangfan1224]

I use java client 1.0.0 which build from the master code by myself. if you want to reproduce the bugs, besides the java client version, you should better set the replica param of cluster more than 1.

Yuce Tekol notifications@github.com 于 2018年8月15日周三 23:14写道：

Hi @yangfan1224 https://github.com/yangfan1224 . Thanks for your issue report. I will try to re-create the issue here. What version of the Pilosa Java client are you using? A released one, or from the master branch or one of the PRs ?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/pilosa/pilosa/issues/1598#issuecomment-413229780, or mute the thread https://github.com/notifications/unsubscribe-auth/ABwuvOGiauvRfCCA2lJWXk9_51AXYdcLks5uRDrvgaJpZM4V74U0 .

[yuce]

Thanks. I was able to reproduce the problem with both the Java client and a shell script. We are investigating the problem.

[travisturner]

@yangfan1224 the latest code on master addresses this bug. Thanks for the bug report, and let us know if you still have problems with anything.