Open lucas-napse opened 1 year ago
@puneetbehl I have the changes of MongoEntityPersister in my ide. Do you want me to upload it as a pull request?
The change: OLD
if (embeddedInstances instanceof Collection) { // true
Collection coll = (Collection)embeddedInstances;
for (Object dbo : coll) { // for each embedded element...
if (dbo instanceof Document) { // <------ this condition is problematic: is a Map, not a Document
Document nativeEntry = (Document)dbo;
NEW
if (embeddedInstances instanceof Collection) {
Collection coll = (Collection)embeddedInstances;
for (Object dbo : coll) {
if (dbo instanceof Map) { // Map, instead Document, in this section
Document nativeEntry;
if(dbo instanceof Document){ // then, it is a Document? then the same thing
nativeEntry = (Document)dbo;
}
else{
nativeEntry = new Document((Map)dbo); // if it is a Map, a new Document instance is created based on that Map.
}
@puneetbehl I cant upload this PR, the server returns 403 "no permission" when i try to push to branch 73x.
This is not the solution. The solution is found below, in the modification of the getValue() method
@puneetbehl The same occurrs in grails 4.1.2, with 'org.grails.plugins:mongodb:7.1.0', and "org.grails:gorm-mongodb-spring-boot:7.1.0".
@puneetbehl Hi. We can solve this problem. It is due to lack of conversion to Document, for the mechanism to work correctly. Using our system (a large, very complex system) we have been able to verify that it works correctly. Since I don't have pull request permissions, I leave the change to be made here:
class MongoEntityPersister
FROM
public static final ValueRetrievalStrategy<Document> VALUE_RETRIEVAL_STRATEGY = new ValueRetrievalStrategy<Document>() {
@Override
public Object getValue(Document document, String name) {
return document.get(name);
}
TO
public static final ValueRetrievalStrategy<Document> VALUE_RETRIEVAL_STRATEGY = new ValueRetrievalStrategy<Document>() {
@Override
public Object getValue(Document document, String name) {
Object object = document.get(name);
if(object instanceof Document){
return (Document) object;
}
else if(object instanceof Map){
return new Document((Map) object);
}
return object;
}
I believe I'm running into the same issue. We're also upgrading to Grails 5.3.2 (from 3.3.10) and GORM for MongoDB 7.2.1 (from 6.1.7).
We're finding that when you're dealing with two levels or more of hasMany
relationships, the second level doesn't come back from the database in a GORM call.
For example, given the following data structure:
Country {
...
static hasMany = [regions: Region]
}
Region {
...
static hasMany = [states: State]
}
State {
...
}
If I were to do a GORM call Country.findByName('USA')
, accessing regions
off of that would return a list of Region objects, and if you were to loop through those Regions, their properties would be populated, but the states
property would be a list of null ([null]
).
@lucas-napse are you just running with a local fork of this repo with your changes included in your dependencies?
@seanhastings Exactly. I downloaded this project, and made the change in MongoEntityPersister, only in the getValue() method, as shown in the last comment. With gradle I could compile the jar, and using that jar in our project (replacing the existing one in the gradle cache [intellij: find MongoEntityPersister class, then: Select opened files - circle in top of project tree, and in the tree right click, copy path/reference path -> absolut path]) we were able to verify that it was what solved the problem.
@puneetbehl Please add this patch because it is an important feature which presents the bug. In addition, the solution is available !!! Thank you very much.
@lucas-napse Unfortunately it seems like the fix for your issue didn't fix mine, I'll need to keep digging.
please, @puneetbehl , review this change. We need to have this solution in our system so that it can work without problems. Very thankful.
I'm attempting to recreate the issue through testing with GORM MongoDB. Could you kindly generate a sample application that mimics the problem?
Furthermore, I believe employing the subsequent approach would offer a superior solution:
Within org/grails/datastore/mapping/mongo/engine/MongoEntityPersister.java file:
dbo.getClass().isAssignableFrom(Document.class)
In relation to the pull request, my understanding is that the procedure involves initially forking the repository, implementing your modifications, and subsequently initiating a new pull request from the forked repository. Please inform me if this method is ineffective for you.
We also urgently need this fixed.
If I generate a sample app for this problem, can I hope for at least a patch release relatively soon?
I'm also concerned that the assumption seems wide-spread that the driver returns a Document when it actually returns a LinkedHashMap.
+1 - we faced the same issue after upgrade
So for us issue is also caused in MongoEntityPersister but in different place. Maybe @seanhastings you can check it for yourself too:
Issue is in loadEmbeddedCollection, if embedded collection is not a map, this method will create collection and try to map instances, but it also checks only for "Document" and ignores any other values.
It happens on this line:
Collection coll = (Collection)embeddedInstances;
for (Object dbo : coll) {
if (dbo instanceof Document) { // existing entries are ignored on this line```
But I had issue to reproduce this on fresh project and I did more debugging. The difference I found is that our migrated project did not use the codec engine. After I added grails.mongodb.engine=codec
the issue seems to be gone now.
What I was not able to find for now is why codec is used for freshly generated project but is not used for my legacy one.
I could see if codec is used in MongoQuery
constructor which is setting isCodecPersister
Using the grails.mongodb.engine property (my fresh project had none) fixed for me. Thanks!
I'm running into 2 more incompatible errors.
parent.embeddeds.size()
gives an NPE and needs to be changed to parent.embedddeds?.size() :? 0
@practical-programmer forgot to get back to this here with my own solution, but I also was able to fix our problem by switching to the "codec" engine, we were on "mapping".
Hi! Thanks for the solution grails.mongodb.engine=codec This change of value from mapping to codec fix our problem too.
@puneetbehl
In this case, should a pull request be raised to add to the manual in the version update section, that it should be changed from "mapping" to "codec" if there is a problem with embedded entities?
This clarification would really help a lot.
Thank you very much.
Is there any documentation on grails.mongodb.engine and differencens between codec v.s. mapping?
@lucas-napse I think it would make sense to add this to upgrade notes under documentation.
I had that problem, I couldn't find the solution, in my case I couldn't update embedded documents and the solution I had found was adding engine:mapping in the .yml. But I had the problem with this post that when I performed the get it did not return the elements of the list. The solution I found was moving my embedded class to the domain folder.
But I had issue to reproduce this on fresh project and I did more debugging. The difference I found is that our migrated project did not use the codec engine. After I added
grails.mongodb.engine=codec
the issue seems to be gone now.What I was not able to find for now is why codec is used for freshly generated project but is not used for my legacy one.
I could see if codec is used in
MongoQuery
constructor which is settingisCodecPersister
in wich file do you add this setting? thanks
Hi @puneetbehl !
In our system we have a large number of domain classes. In our recent upgrade to grails 5 (5.3.2) that makes use of GORM 7.3.0 of our system, which was working fine with the Grails 2.5.6, we noticed that the embedded entities were not loaded in the object that contained it: For example,
In the above
Country
has embeddedregions
, these regions are initialized as anempty list
when loaded through GORM. Whereas, in the database collectionCountry
hasregions
. Even, when using the MongoDB API, these regions are loaded correctly as list ofDocument
.This does not work when using either
Country.findAll()
, orCountry.createCriteria().list{}
. It maybe breaking with other GORM queries as well.Upon, debugging, we noticed that the problem might be caused because the database returns a
Map
instead of aDocument
. The Groovy classMongoEntityPersister
exclusively checks for the classDocument
which breaks certain conditions. For example:MongoEntityPersister.loadEmbeddedCollection()
.When the method is executed, I noticed that in our case it goes through the
else
branch because the following condition evaluates tofalse
The above results in embedded
regions
to return an emptyList
because it is aMap
.By modifying the above to following, might fixes the problem:
The above change solves the our problem where the
regions
are correctly loaded via the GORMCountry.findAll
method.Doing more tests I notice that there is a second problem: if this embedded entity uses composition, some related attributes are loaded and others are not. For example: my country has regions, and a region has a region type. This type of region is a Domain. The regions are embedded in the country, and a region has only one type of region, as an associated instance (composition).
This instance, the region type, is not loaded.
Performing more debugging, I find that the code goes by: NativeEntityPersister#refreshObjectStateFromNativeEntry(PersistentEntity, Object, Serializable, T, boolean)
... continuing debugging leads to this method ...
... continuing debugging leads to this method ...
...clearly this method gives true when it is a Document, but my region type appears as Map, which gives false. This, by giving false, the caller (AbstractMongoObectEntityPersister#getEmbedded(T, String)) determines that it is null, so it does not load my region type.
Making this change should fix it:
The problem now is that internally we get various errors related to the Map and Document type. And these are related to the getValue() and setValue() methods of the MongoEntityPersister class:
This is the error you get after making all the changes I mentioned above:
As a solution we will try to use an older version of gorm, but if this problem can be solved it would be great since it is always necessary to have the latest. If you need more information, please comment and I will respond as soon as possible.
Some details: | Grails Version: 5.3.2 | JVM Version: 11.0.16.1 | Gorm version: 7.3.0 | Windows 11, with last updates.
(build.gradle) implementation 'org.grails.plugins:mongodb:7.3.0' implementation "org.grails:gorm-mongodb-spring-boot:7.3.0"