MappingMongoConverter should convert and excape dollar prefix [DATAMONGO-1554]

[spring-projects-issues]

schoster barak opened DATAMONGO-1554 and commented

MappingMongoConverter should convert and escape dollar prefix as they are not supported as Map key in mongoDb. in other words, the key can not start in "$"

---

Affects: 2.0 M1 (Kay), 1.9.5 (Hopper SR5)

Reference URL: https://github.com/spring-projects/spring-data-mongodb/blob/master/spring-data-mongodb/src/main/java/org/springframework/data/mongodb/core/convert/MappingMongoConverter.java#L704

Referenced from: pull request https://github.com/spring-projects/spring-data-mongodb/pull/427

[spring-projects-issues]

schoster barak commented

i think it should be resolved in a similar way to the dot converter. if you'd like i will make a pull request

[spring-projects-issues]

Mark Paluch commented

Barak, thanks for your pull request. Working with generic structures such as Maps or directly with DBObject always comes with the possibility to use key names that are not allowed by MongoDB. Extending MappingMongoConverter is not the right direction. potentiallyEscapeMapKey and potentiallyUnescapeMapKey would grow otherwise by attracting additional conversion rules.

From your description, I understand you want to store a subdocument which contains keys not allowed by MongoDB. Rather than introducing changes to MappingMongoConverter, you should use Spring Data MongoDB Lifecycle events to post-process DBObject before storing/before conversion.

BeforeSaveEvent and BeforeConvertEvent provide mutable access to the DBObject. A registered listener allows post-processing of key names in your code. Care to give the post-processing approach a try?

[spring-projects-issues]

schoster barak commented

Thank you for responding. in my use case, i do not have a single POJO i would like to convert it's MAPs keys. i have multiple document in multiple collections containing map keys that might start with $. though it is rare among the data, it may occur.

Implementing BeforeSaveEvent and BeforeConvertEvent for each POJO does not seem to me like the right approach. Implementing BeforeSaveEvent and BeforeConvertEvent for DBObject will require scanning it, looking for Maps with keys containing "$". this scan might take some execution time.

how about creating a Map with conversion rules at MappingMongoConverter? potentiallyEscapeMapKey and potentiallyUnescapeMapKey will convert key to value by the conversion rules map

[spring-projects-issues]

Mark Paluch commented

The way to go would be a map key escaping strategy that is configurable at MappingMongoConverter. The default implementation would exactly do what potentiallyEscapeMapKey and potentiallyUnescapeMapKey do at the moment.

Please note that fields starting with $ cannot be updated individually/used as predicates – Mongo's built-in operators start with $ and our query/update handling identifies keywords by checking the dollar prefix. Using the escaped field name for such operations would be the way to go

[christophstrobl]

With MongoDB 5.0 we saw some movement regarding the usage of $ prefixed keys in both the root as well as nested document structures.

Generally speaking it is possible to store $ prefixed fields in the root level document.

class With$FieldName {

    @Field("$myValue")
    String value;
}

{
    "_id" : "...",
    "$myValue" : "v1"
}

The culprit there is the query part which only allows known keywords and requires a rewrite form { '$myValue' : 'v1' } to something like the snippet below, which seems not feasible with the current query mapping.

{ "$expr" : { "$eq" : [ { "$getField" : { "$literal" : "$myValue"}}, "v1"]}}

Regarding the usage of $ in nested documents or similar Map like structures it is possible to insert but not upsert those.

class With$MapKey {
    Map<String,String> map;
}

With$MapKey source = new With$MapKey();
source.map = Map.of("k1", "v1", "$k2", "v2");

// works
template.insert(source);

// fails with: Field names in a replacement document can not start with '$' but '$k2' does
template.save(source); 

There's a lot of restrictions that come with the usage of $ for field names though it is generally possible using the current API. The hooks already mentioned in previous comments provide a sufficient enough workaround and we do not intend to change the current mapping process but ask to respect store specific limitations which may require some data preprocessing prior to storage.