Wrong result on new tests

[rasrov]

I'm trying to add some code to test new functionality that I want to add to redis repository, but I'm having some troubles about.

First of all, I want to add filter function to Group class and this need tests to reques a PR.

I have this test code:

AggregationTest.java
...
@Test
public void testFilterAtGroupLevel() {
  client.ftCreate(index,
      NumericField.of("id").sortable(),
      TextField.of("incidentType").sortable(),
      TextField.of("category").sortable(),
      NumericField.of("section"),
      TextField.of("road").sortable(),
      TextField.of("description").sortable(),
      NumericField.of("createdTime").sortable()
  );

  addDocument(new Document("event1").set("incidentType", "Weather Incident").set("category", "Is raining").set("section", "section1").set("road", 1).set("description", "description1").set("createdTime", 1683713590));
  addDocument(new Document("event2").set("incidentType", "Weather Incident").set("category", "Is raining").set("section", "section1").set("road", 1).set("description", "description2").set("createdTime", 1683713590));
  addDocument(new Document("event3").set("incidentType", "Weather Incident").set("category", "Light snow").set("section", "section2").set("road", 1).set("description", "description3").set("createdTime", 1683714100));
  addDocument(new Document("event4").set("incidentType", "Slippery Road").set("category", "1").set("section", "section2").set("road", 1).set("description", "description4").set("createdTime", 1683713590));
  addDocument(new Document("event5").set("incidentType", "Weather Incident").set("category", "Moderate snow").set("section", "section2").set("road", 1).set("description", "description5").set("createdTime", 1683713590));

  addDocument(new Document("event6").set("incidentType", "Weather Incident").set("category", "Is raining").set("section1", "section1").set("road", 2).set("description", "description6").set("createdTime", 1683713590));
  addDocument(new Document("event7").set("incidentType", "Weather Incident").set("category", "Is raining").set("section2", "section1").set("road", 2).set("description", "description7").set("createdTime", 1683713590));
  addDocument(new Document("event8").set("incidentType", "Slippery Road").set("category", "1").set("section", "section2").set("road", 2).set("description", "description8").set("createdTime", 1683713590));
  addDocument(new Document("event9").set("incidentType", "Slippery Road").set("category", "1").set("section", "section2").set("road", 2).set("description", "description9").set("createdTime", 1683713590));
  addDocument(new Document("event10").set("incidentType", "Slippery Road").set("category", "1").set("section", "section2").set("road", 2).set("description", "description10").set("createdTime", 1683714100));

  Group firstGroup = new Group("@incidentType", "@category", "@section", "@road", "@description", "@createdTime");
  firstGroup.filter("@createdTime < 1683714000");

  Group secondGroup = new Group("@incidentType", "@category", "@section", "@road", "@description");
  secondGroup.reduce(Reducers.count().setAlias("numIncidents"));
  secondGroup.reduce(Reducers.first_value("@description").setAlias("description"));

  Group thirdGroup = new Group("@incidentType", "@category", "@section", "@road", "@description", "numIncidents");

  AggregationBuilder r = new AggregationBuilder("*")
      .groupBy(firstGroup)
      .groupBy(secondGroup)
      .groupBy(thirdGroup);

  AggregationResult res = client.ftAggregate(index, r);
  assertEquals(4, res.totalResults);
}

The result of index and documents creation is this:

"FT.CREATE" "aggbindex" "SCHEMA" "id" "NUMERIC" "SORTABLE" "incidentType" "TEXT" "SORTABLE" "category" "TEXT" "SORTABLE" "section" "NUMERIC" "road" "TEXT" "SORTABLE" "description" "TEXT" "SORTABLE" "createdTime" "NUMERIC" "SORTABLE"
"HSET" "event1" "road" "1" "incidentType" "Weather Incident" "description" "description1" "createdTime" "1683713590" "section" "section1" "category" "Is raining"
"HSET" "event2" "road" "1" "incidentType" "Weather Incident" "description" "description2" "createdTime" "1683713590" "section" "section1" "category" "Is raining"
"HSET" "event3" "road" "1" "incidentType" "Weather Incident" "description" "description3" "createdTime" "1683714100" "section" "section2" "category" "Light snow"
"HSET" "event4" "road" "1" "incidentType" "Slippery Road" "description" "description4" "createdTime" "1683713590" "section" "section2" "category" "1"
"HSET" "event5" "road" "1" "incidentType" "Weather Incident" "description" "description5" "createdTime" "1683713590" "section" "section2" "category" "Moderate snow"
"HSET" "event6" "road" "2" "incidentType" "Weather Incident" "description" "description6" "createdTime" "1683713590" "category" "Is raining" "section1" "section1"
"HSET" "event7" "road" "2" "incidentType" "Weather Incident" "description" "description7" "createdTime" "1683713590" "category" "Is raining" "section2" "section1"
"HSET" "event8" "road" "2" "incidentType" "Slippery Road" "description" "description8" "createdTime" "1683713590" "section" "section2" "category" "1"
"HSET" "event9" "road" "2" "incidentType" "Slippery Road" "description" "description9" "createdTime" "1683713590" "section" "section2" "category" "1"
"HSET" "event10" "road" "2" "incidentType" "Slippery Road" "description" "description10" "createdTime" "1683714100" "section" "section2" "category" "1"

The test are failing with this query:

"FT.AGGREGATE" "aggbindex" "*" 
    "GROUPBY" "6" "@incidentType" "@category" "@section" "@road" "@description" "@createdTime" 
        "FILTER" "@createdTime < 1683714000" 
    "GROUPBY" "5" "@incidentType" "@category" "@section" "@road" "@description" 
        "REDUCE" "COUNT" "0" "AS" "numIncidents" 
        "REDUCE" "FIRST_VALUE" "1" "@description" "AS" "description" 
    "GROUPBY" "6" "@incidentType" "@category" "@section" "@road" "@description" "numIncidents"

Query is not grouping correctly and I don't know why.

I tryed to do a simple aggregate query, to try to get all descriptions. Like this:

"FT.AGGREGATE" "aggbindex" "*" 
    "GROUPBY" "1" "@description"

But the result is not as expected. Is returning only two results, insted ot the ten that there are.

1) "2"
2) 1) "description"
   2) "description6"
3) 1) "description"
   2) "description7"

Index info:

[sazzad16]

@rasrov Do you get similar result if you execute the commands from redis-cli?

[rasrov]

Hii @sazzad16

Yes, those results are from redis-cli.

I tried the same aggregation query with data set from another test and it works well. I don't see any difference between my data set and other one.

Mentioned test:

AggregationTest.java
...
@Test
public void aggregateIterationCollect() {
  client.ftCreate(index, TextField.of("name").sortable(), NumericField.of("count"));

  addDocument(new Document("data1").set("name", "abc").set("count", 10));
  addDocument(new Document("data2").set("name", "def").set("count", 5));
  addDocument(new Document("data3").set("name", "def").set("count", 25));
  addDocument(new Document("data4").set("name", "ghi").set("count", 15));
  addDocument(new Document("data5").set("name", "jkl").set("count", 20));

  AggregationBuilder agg = new AggregationBuilder()
      .groupBy("@name", new ArrayList<>(), Reducers.sum("@count").as("sum"))
      .sortBy(10, SortedField.desc("@sum"))
      .cursor(2, 10000);

  assertEquals(4, client.ftAggregateIteration(index, agg).collect(new ArrayList<>()).size());
}

Index and documents creation:

"FT.CREATE" "aggbindex" "SCHEMA" "name" "TEXT" "SORTABLE" "count" "NUMERIC"
"HSET" "data1" "name" "abc" "count" "10"
"HSET" "data2" "name" "def" "count" "5"
"HSET" "data3" "name" "def" "count" "25"
"HSET" "data4" "name" "ghi" "count" "15"
"HSET" "data5" "name" "jkl" "count" "20"

Query:

"FT.AGGREGATE" "aggbindex" "*"
    "GROUPBY" "1" "@count"

Result:

1) "5"
2) 1) "count"
   2) "25"
3) 1) "count"
   2) "10"
4) 1) "count"
   2) "5"
5) 1) "count"
   2) "15"
6) 1) "count"
   2) "20"

The query has returned the 5 records that exist, as the one I created should do.

[sazzad16]

@rasrov Please contact RediSearch

We may help you with executing a query from Java (Jedis) but we may not help you with the actual query.

Note: Instead of Java code, you should use raw/captured commands there.

[rasrov]

Solved at https://github.com/RediSearch/RediSearch/issues/3560