Using MongoRepository

[glodepa]

Can this library be adapted for use with mongo repository?

[torshid]

I currently have no idea if MongoRepository is compatible with JPA as I have no experience using it. But you may easily extend this library in order to achieve your goal as this library is mainly a compiler which does not depend on Spring or JPA. I am going to document this in the coming days.

You have two ways:

• Clone the repository, and add a custom method to the IExpression interface, which you are going to implement in all expression classes (e.g. Condition, Operation). You are then going to call this method which is going to recursively build your query. You may look into the current generate method returning a string, it is pretty straightforward.

• If you do not want to clone the repository, you may call the generate(Filter ast, Function<IExpression, T> func) method of the FilterCompiler class and provide it the func argument which will basically do the same thing as in the first way but will have to check the type of the input expressions:

  
  FilterCompiler.generate(filter, new Function<IExpression, T>() {
  
  @Override
  public T apply(IExpression exp) {
  
  if (exp instanceof Filter) {
    return apply(((Filter) exp).getBody());
  }
  
  if (exp instanceof OperationInfix) {
    OperationInfix op = (OperationInfix) exp;
    switch (op.getOperator()) {
      case AND:
        // do something with op.getLeft() and op.getRight()
    }
  }
  
  // other expressions
  
  throw new UnsupportedOperationException("Unsupported expression " + exp);
  
  }

});

As said previously, you should take the current implementations as a guide in order to fully support the syntax described in the readme. You may also create your own syntax/expressions if you wish to do so.

If you have experience with MongoRepository you can also open a PR adding its support using the first way, that would be great. I can also do it if you can assist me but it may take time.

Thanks.

[torshid]

I made some research and I was able to compile the queries to Bson filters, for example:

@Override
public Bson generate(Object payload) {

  switch (getOperator()) {

    case AND:
      return com.mongodb.client.model.Filters.and(getLeft().generate(payload),
          getRight().generate(payload));

    case OR:
      return com.mongodb.client.model.Filters.or(getLeft().generate(payload),
          getRight().generate(payload));

    default:
      throw new InvalidQueryException("Unsupported infix operator " + getOperator().getLiteral());

  }

}

But I am not sure how can this be used with MongoRepository, do you have any idea? I don't see any method of that interface which can take some kind of filter.

[marcopag90]

Hi torshid, i'm working with @glodepa at the REST language implementation with MongoRepository, following the same logic of the EntityFilter you've adopted. We are making use of the com.turkraft.springfilter.FilterParse to parse the expressions and then performing a conversion to a org.springframework.data.mongodb.core.query.Criteria. Then, we registered the MongoFilterArgumentResolver, whose condition is applied:

@Override
public boolean supportsParameter(MethodParameter methodParameter) {
  return methodParameter.getParameterType().equals(Criteria.class)
        && methodParameter.hasParameterAnnotation(EntityFilter.class);
  }

and of course, we started the implementation of the method to build the Criteria.

@Override
public Object resolveArgument(
      MethodParameter methodParameter,
      ModelAndViewContainer modelAndViewContainer,
      NativeWebRequest nativeWebRequest,
      WebDataBinderFactory webDataBinderFactory) throws Exception {

    EntityFilter entityFilter = methodParameter.getParameterAnnotation(EntityFilter.class);

    return getCriteria(methodParameter.getGenericParameterType().getClass(),
        !entityFilter.parameterName().isEmpty()
            ? nativeWebRequest.getParameterValues(entityFilter.parameterName())
            : null);

  }

  private Criteria getCriteria(Class<?> specificationClass, String[] inputs) {
    Criteria result = null;
    if (inputs != null && inputs.length > 0) {
      Collection<IExpression> filters = new ArrayList<>();
      for (String input : inputs) {
        if (input.trim().length() > 0) {
          filters.add(FilterParser.parse(input.trim()));
        }
      }
      result = processFilter(FilterQueryBuilder.and(filters));
    }
    return result;
  }

  private <T> T processFilter(IExpression expression) {
    if (expression instanceof ConditionInfix) {
      ConditionInfix condition = (ConditionInfix) expression;
      switch (((ConditionInfix) expression).getComparator()) {
        case EQUAL:
          return (T) Criteria.where(processFilter(condition.getLeft()))
              .is(processFilter(condition.getRight())
              );
      }
    } else if (expression instanceof Field) {
      return (T) ((Field) expression).getName();
    } else if (expression instanceof Input) {
      IInput value = ((Input) expression).getValue();
      return (T) (value == null ? null : value.getValue()); //this method is missing!
    } else if (expression instanceof Arguments) {
      Object[] values = new Object[((Arguments) expression).getValues().size()];
      int count = 0;
      for (IExpression value : ((Arguments) expression).getValues()) {
        values[count++] = processFilter(value);
      }
      return (T) values;
    }
    return null;
  }

After that, it's all about creating our own MongoRepository implementation (since there's no direct usage of Criteria):

@NoRepositoryBean
public interface GenericMongoRepository<T, I> extends MongoRepository<T, I> {

  Page<T> findAll(Criteria criteria, Pageable pageable);
}

and taking advantage of our own implementation inside the MongoRepository:

public class CriteriaMongoRepository<T, I>
    extends SimpleMongoRepository<T, I> implements GenericMongoRepository<T, I> {

  private final MongoOperations mongoOperations;
  private final MongoEntityInformation<T,I> metadata;

  public CriteriaMongoRepository(MongoEntityInformation<T, I> metadata,
      MongoOperations mongoOperations) {
    super(metadata, mongoOperations);
    this.mongoOperations = mongoOperations;
    this.metadata = metadata;
  }

  @Override
  public Page<T> findAll(Criteria criteria, Pageable pageable) {
    Query query = new Query().with(pageable);
    if (criteria != null) {
      query.addCriteria(criteria);
    }
    return new PageImpl<>(mongoOperations.find(query, metadata.getJavaType()));
  }
}

So far we created a simple example with the equal operator and of course code will require some refactor to make the method private <T> T processFilter(IExpression expression) human readable 👍 . What we are missing to make it work is the method getValue() inside the com.turkraft.springfilter.token.IToken.IInput interface. I noticed you created the method in the current branch but you haven't published it as a Maven jar yet. Would you be so kind to release it? Or can we achieve the same result in another way? Thanks for your help, best regards :)

[torshid]

Hello @marcopag90,

I am not sure if you checked the mongorepository branch (PR #60), but I initially did the implementation as you did with Criteria and everything worked fine besides the not operation, therefore I explored other things and I found that it was possible to make queries with com.mongodb.client.model.Filters (but it uses Bson instead of Criteria).

I think that it may be enough to create a GenericMongoRepository as you did, but which takes a Bson instead of Criteria. Again I am not experienced with MongoDB so I could be on the wrong path.

If you are able to filter with Bson, then creating the MongoFilterArgumentResolver is what is left at this point. Most of the filtering features are currently implemented in that PR. If that's not possible then it would be great if you have a solution for the problem above.

Are you able to create a Criteria equivalent to not a : 'b' (meaning a is not equal to b) for example? I tried a few things but It didn't work.

Criteria.where("a").not().is("b");
// Invalid query: 'not' can't be used with 'is' - use 'ne' instead

(new Criteria()).not().where("a").is("b");
// no exception but 'not' is skipped

This doesn't work too not (a : 'b' or c : 'd'):

(new Criteria()).not().orOperator(Criteria.where("a").is("b"), Criteria.where("c").is("d"))
// operator $not is not allowed around criteria chain element: Document{{$or=[Document{{a=b}}, Document{{a=b}}]}}

That's the only real problem actually.

I am basically not able to reproduce the queries given here https://docs.mongodb.com/manual/reference/operator/query/not/

You may also check this page in order to see com.mongodb.client.model.Filters usages.

I will try to make a new release tomorrow but if you can clone and run the project in your local environment it would be better (I can always help if you encounter a problem).

[marcopag90]

Oh, you are right. I'm not that familiar with MongoDb as well, I started studying it a few days ago. I'll investigate if there's a proper syntax for the not operator using Criteria. Tomorrow i'll clone the mongorepository branch and try to find if there's a way to build a Criteria from a bson. Looking at the Criteria documentation, there's a method to get the bson Document getCriteriaObject so i think there has to be away to achieve the reverse operation. Criteria object seems excellent to integrate with other interface, like Pageable to get sort and pagination out of the box.

[marcopag90]

Ok, i made it. Maybe the syntax could be more concise, for now it's just a raw test. Anyway, I used a bottom up approach, starting from the class org.springframework.data.mongodb.core.query.Query, which allows to create different types of queries based on other objects. After some research, I found that it's not that easy to convert a Bson Document to Criteria API, see here: https://stackoverflow.com/questions/60454970/spring-data-mongo-db-convert-from-document-to-criteria

If i can avoid using java reflections, i'm happier (better performance and so on...). So i looked for another way: the Query class can accept a org.bson.Document object, that is one of the Bson interface implementation, here it's explained how to convert it: https://stackoverflow.com/questions/49262903/mongodb-java-driver-convert-bsondocument-to-document-and-back

Then I used your generateBson() method from the IExpression interface and wrote the argument resolver.

public class DocumentFilterArgumentResolver implements HandlerMethodArgumentResolver {

  @Override
  public boolean supportsParameter(MethodParameter parameter) {

    return parameter.getParameterType().equals(Document.class)
        && parameter.hasParameterAnnotation(EntityFilter.class);
  }

  @Override
  public Object resolveArgument(MethodParameter parameter, ModelAndViewContainer mavContainer,
      NativeWebRequest webRequest, WebDataBinderFactory binderFactory) {

    EntityFilter entityFilter = parameter.getParameterAnnotation(EntityFilter.class);
    Objects.requireNonNull(entityFilter);
    Bson bson = getBson(!entityFilter.parameterName().isEmpty() ?
        webRequest.getParameterValues(entityFilter.parameterName()) : null
    );
    return getDocument(bson);
  }

  private Bson getBson(String[] inputs) {
    if (inputs != null && inputs.length > 0) {
      Collection<IExpression> filters = new ArrayList<>();
      for (String input : inputs) {
        if (input.trim().length() > 0) {
          filters.add(FilterParser.parse(input.trim()));
        }
      }
      return FilterQueryBuilder.and(filters).generateBson();
    }
    return null;
  }

  private Document getDocument(Bson bson) {
    if (bson == null) {
      return null;
    }
    BsonDocument bsonDocument =
        bson.toBsonDocument(BsonDocument.class, MongoClientSettings.getDefaultCodecRegistry());
    DocumentCodec codec = new DocumentCodec();
    DecoderContext decoderContext = DecoderContext.builder().build();
    return codec.decode(new BsonDocumentReader(bsonDocument), decoderContext);
  }
}

and the repository implementation as follows:

public class MongoDocumentRepository<T, I>
    extends SimpleMongoRepository<T, I> implements MongoRepository<T, I> {

  private final MongoEntityInformation<T, I> metadata;
  private final MongoOperations mongoOperations;

  public MongoDocumentRepository(
      MongoEntityInformation<T, I> metadata,
      MongoOperations mongoOperations
  ) {
    super(metadata, mongoOperations);
    this.metadata = metadata;
    this.mongoOperations = mongoOperations;
  }

  @Override
  public Page<T> findAll(Document bson, Pageable pageable) {
    Query query;
    if (bson != null) {
      query = new BasicQuery(bson);
    } else {
      query = new Query();
    }
    query = query.with(pageable);
    return new PageImpl<>(mongoOperations.find(query, metadata.getJavaType()));
  }
}

This works like a charm. Now, what's left is just to test all the operators to check if everything is fine. Oh, the not operator, generated from the Document object, works 👍

[torshid]

That's so great, many thanks for your contribution. :)

I already wrote some tests here but they are too simple maybe.

I am going to add your resolver to the PR and release a new version as soon as possible.

There is a actually one question left in my mind, I would like to know what you think. Should we add the GenericMongoRepository interface only? Adding MongoDocumentRepository as a class would be wrong I think since it may break the design of potential users in case they already extend other classes in their repositories. But the thing is that only exposing the interface will require implementation and it may be too complex for some users.

Maybe we can add both the interface and the class, and let the user choose one. Do you have any other idea?

[marcopag90]

Yea, that's the same question i had left. The problem is that the MongoRepository interface inside Spring data mongo has only the SimpleMongoRepository implementation that is the one I extended. That interface has no direct method for Criteria or Bson objects. The SimpleMongoRepository makes use of the org.springframework.data.mongodb.core.MongoOperations interface, see here: https://docs.spring.io/spring-data/mongodb/docs/current/api/org/springframework/data/mongodb/core/MongoOperations.html.

In Spring, you can autowire the org.springframework.data.mongodb.core.MongoTemplate which is the implementation that allows to perform queries over MongoDB: I assume that's the counter part of the JdbcTemplate in pure sql style. So a developer has two choices:

1. Use the MongoTemplate inside a RestController where the filter is defined, but requires some more code to write. I mean, you get the Document object but without a query object you can't do much.
2. Extending the GenericMongoRepository as I did seems the easiest way, providing the default implementation i created. That's the easiest way, without any other boilerplate code to write, but as you said, it could "potentially" break the design.

So I think what you suggest could be the solution: add both repositories and let the user decide if going with the default implementation, providing another one or injecting in their contexts the MongoTemplate to make use of the Document object resolved from the DocumentFilterArgumentResolver.

If you want, i can commit my changes on the mongorepository branch, no need for you to copy-paste everything again...it's up to you! Let me know what you think.

[torshid]

I decided to provide the interface DocumentExecutor (name similar to JpaSpecificationExecutor), with the DocumentExecuterImpl interface which provides default implementations: https://github.com/turkraft/spring-filter/blob/mongorepository/src/main/java/com/turkraft/springfilter/DocumentExecutorImpl.java

I think that it's quite flexible like that. You may have something like:

class MyRepo extends SimpleMongoRepository<T, I> implements DocumentExecutorImpl<T, I> {
  // need to provide getMetadata() and getMongoOperations() and you are ready to go
}

Is it possible for you to check if everything works fine in the mongorepository branch and then I am going to merge the PR.

If you want, i can commit my changes on the mongorepository branch, no need for you to copy-paste everything again...it's up to you!

I had already committed what you shared but I am going to add a contributors section in the readme with your name.

[marcopag90]

Ok, tomorrow I'm going to check everything from the branch 👍
Nice work so far!

[marcopag90]

Ok so, i made some tests and so far operators not working are:

• in

  @Override
  public Bson generateBson(Object payload) {
  
  ...
  
  if (!(getRight() instanceof Input)) { //in operator is an argument
    throw new InvalidQueryException("Right side of infix conditions should be an input");
  }
  
  ...
  }

  and so all other argument operators are not yet supported (but this seems reasonable, mongo has different aggregate functions from jpa). Maybe for now it could be enough to make the in operator to work properly. All the other operators are working as expected, with an exception for all java.time.* fields.

This is a useful link for all the supported mapping https://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mapping-conversion.

To understand how to build the proper query from a bson Document, I used the Criteria object and then generated the valid Document from it.

Criteria dateCriteria = Criteria.where("lastModified").lt(LocalDate.of(2021-04-25));
Document document =  dateCriteria.getCriteriaObject();
Query query = FilterUtils.getQueryFromDocument(document); // Query: { "lastModified" : { "$lt" : { "$java" : 2021-04-25 } } }, Fields: {}, Sort: {}
 return new PageImpl<>(getMongoOperations()
    .find(query.with(pageable), getMetadata().getJavaType()));

This produces the following valid query, here's a piece of log: o.s.data.mongodb.core.MongoTemplate : find using query: { "lastModified" : { "$lt" : { "$date" : "2021-04-24T22:00:00Z"}}} fields: Document{{}}

So there's clearly something going on in the MongoTemplate to resolve the $java placeholder to that nested object "$date". This makes me think to go back using the Criteria object instead and build the query with it maybe...

I made some research about the use of the not() operator in the Criteria object, and that's a logical operator, so the correct Criteria can be built with ne() : https://stackoverflow.com/questions/19353955/mongodb-spring-data-criteria-not-operator And here how to combine a Criteria.andOperator() https://stackoverflow.com/questions/33546581/mongotemplate-criteria-query.

Or if you think there might be a way to parse the query into a valid Document using java date fields, maybe it's all about adjusting the syntax, but i'm not that familiar with parsers so no idea for now.

[torshid]

Thanks again for your valuable research. I have pushed the in and ~ (which is actually the regex operator) implementations.

The problem with the mapping of java.time.* or everything else besides numbers, bools and strings is a serious one here.

The thing is that as opposed to JPA, we don't have the type of the field we are accessing. With JPA, I just had to call the getValueAs(Class<?> klass) method of IInput which was converting the present value to the given type (check the Text class). For example if the field we are accessing is a date, then the string input is correctly converted to a date object.

It is possible to get the type with reflection but it may cost a lot. I don't have any other idea currently.

I think that the problem would remain is we use Criteria, since the problem of knowing when to map/convert one type to another (e.g. string to date) is still here.

An alternative solution I don't really appreciate is to add mapping functions such as: some_date_field : date('25-04-2021'). We are explicity asking for the string to be converted to a date here. But it adds too much complexity to the initially easy to use syntax, it would also require changes to the query builder.

I am going to merge the PR and we may create another issue to discuss this not so easy mapping problem.

Edit: https://github.com/turkraft/spring-filter/releases/tag/0.9.6

[marcopag90]

Ok, I agree. If we can manage to make it works with date fields, I think it's a big jackpot.