Mapping with reference fails when using more than one db (>= 0.110)

[BairDev]

If you fetch one and the same object from your DB with Morphia (e.g. an user), but fetch a user from a different DB (different dbName, different Datastore) meanwhile, the mapping of references fails. Assume that all need objects do exist in the db(s).

The error goes like:

ERROR [2015-04-22 11:45:28,244] io.dropwizard.jersey.errors.LoggingExceptionMapper: Error handling a request: {id}}
! org.mongodb.morphia.mapping.MappingException: The reference({ "$ref" : "commAddresses", "$id" : "{valid id}" }) could not be fetched for com.greylogix.smartgy.essentials.entities.User.commAddresses
! at org.mongodb.morphia.mapping.ReferenceMapper.resolveObject(ReferenceMapper.java:332) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.mapping.ReferenceMapper$1.eval(ReferenceMapper.java:260) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.utils.IterHelper.loopOrSingle(IterHelper.java:70) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.mapping.ReferenceMapper.readCollection(ReferenceMapper.java:257) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.mapping.ReferenceMapper.fromDBObject(ReferenceMapper.java:174) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.mapping.Mapper.readMappedField(Mapper.java:608) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.mapping.Mapper.fromDb(Mapper.java:585) ~[{our-service}-1.33.1.jar:1.33.1]
! ... 70 common frames omitted
! Causing: org.mongodb.morphia.mapping.MappingException: Could not map com.greylogix.smartgy.essentials.entities.User with ID: 54f59b6a0cf2e31ac406bcec
! at org.mongodb.morphia.mapping.Mapper.fromDb(Mapper.java:590) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.mapping.Mapper.fromDBObject(Mapper.java:299) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.query.MorphiaIterator.convertItem(MorphiaIterator.java:79) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.query.MorphiaIterator.processItem(MorphiaIterator.java:65) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.query.MorphiaIterator.next(MorphiaIterator.java:60) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.mongodb.morphia.query.QueryImpl.get(QueryImpl.java:402) ~[{our-service}-1.33.1.jar:1.33.1]
! at com.greylogix.smartgyportal.resources.LoginResource.login(LoginResource.java:104) ~[{our-service}-1.33.1.jar:1.33.1]
! at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:1.7.0_72]
! at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[na:1.7.0_72]
! at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:1.7.0_72]
! at java.lang.reflect.Method.invoke(Method.java:606) ~[na:1.7.0_72]
! at org.glassfish.jersey.server.model.internal.ResourceMethodInvocationHandlerFactory$1.invoke(ResourceMethodInvocationHandlerFactory.java:81) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher$1.run(AbstractJavaResourceMethodDispatcher.java:164) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.invoke(AbstractJavaResourceMethodDispatcher.java:181) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.server.model.internal.JavaResourceMethodDispatcherProvider$ResponseOutInvoker.doDispatch(JavaResourceMethodDispatcherProvider.java:158) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.server.model.internal.AbstractJavaResourceMethodDispatcher.dispatch(AbstractJavaResourceMethodDispatcher.java:101) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.invoke(ResourceMethodInvoker.java:389) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:347) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.server.model.ResourceMethodInvoker.apply(ResourceMethodInvoker.java:102) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.server.ServerRuntime$2.run(ServerRuntime.java:305) ~[{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:271) [{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.internal.Errors$1.call(Errors.java:267) [{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.internal.Errors.process(Errors.java:315) [{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.internal.Errors.process(Errors.java:297) [{our-service}-1.33.1.jar:1.33.1]
! at org.glassfish.jersey.internal.Errors.process(Errors.java:267) [{our-service}-1.33.1.jar:1.33.1]
[snip ...]

I just can guess: is might be a caching issue in resolveObject or in a connected method. This works with Morphia 0.109 but not with 0.110 or higher.

Java code snippet:

/*
 * We have a config db and customer dbs. Therefore we use different datastores.
 * First we look for a user in the config db, then we try to fetch the user object from our 
 * customer db. The name of the customer db is stored in the user object of the config db.
 * This construction might not be ideally, but it usually works. But when you try to fetch users from  
 * different customer dbs, the mapping of some references in this very user object fails.
 */
Datastore datastoreConfig = morphiaDatastoreManager.getDatastoreConfig();
Query<PortalUserIdentifier> identifyQuery = datastoreConfig.createQuery(PortalUserIdentifier.class);
identifyQuery.filter(PortalUserIdentifier.FIELD_EMAIL, user.getEmail());
PortalUserIdentifier storedIdentifier = identifyQuery.get();

    if (storedIdentifier != null) {
        User storedUser;
        String tenantId = storedIdentifier.getTenant(); //name of the customer db
        Datastore dsTenant = morphiaDatastoreManager.getDatastore(tenantId);

       storedUser = dsTenant.find(User.class, PortalUserIdentifier.FIELD_EMAIL, user.getEmail()).get(); //here it breaks!
      // and so on ...

[evanchooly]

This was fixed with #230. You can try with a -SNAPSHOT dependency or build morphia locally if you'd like to try it out.

[BairDev]

Wow, this is interesting. Thanks for the fix!

I did not understand this: This problem has been reported two years ago, Morphia worked in a similar scenario with 0.109, but not with 0.108 and >=0.110? Thanks again.

[evanchooly]

I don't know that it ever really worked. It's not a scenario we test though I suppose it might be time to add some testing in this area since it seems it's increasingly common.

[ozguraydinli]

I think this bug still exists. We are getting a MappingException, when we are querying two databases. It can easily be seen that the datastore instance in ReferenceMapper has the wrong database name when you debug your application.

[evanchooly]

I see I let that PR through without an explicit test. That was my failure. I'll add one now to confirm.

[evanchooly]

Can you take a look at (very) simple test I added and see if it matches your usage? The test passes so i'm curious how your usage is different such that it doesnt work for you.

https://github.com/mongodb/morphia/blob/master/morphia/src/test/java/org/mongodb/morphia/TestDatastore.java#L476-476

[ozguraydinli]

I think test is ok. Is this fix is in rc0? or in rc1?

[evanchooly]

this works for rc0. I didn't make any changes to make this test work.

[ozguraydinli]

I just figured out the difference, the FacebookUser does not have @Reference field. Can you please try something like below:

public class FacebookUser {

  @Reference
  private ArrayList<MyPojo>  references;

}

[evanchooly]

Got it. I'm looking in to how extensive a fix this will be. If it's too much it'll have to wait for post 1.0 but I think I can do this fairly cleanly.

[ozguraydinli]

Please do it, half of our application does not work now :) Thanks.

[evanchooly]

Take a look at the test now. This uses references, threads, and ThreadLocalDatastoreProvider to manage the different databases.

On a given thread, however, cross-database references still are not possible because the Java driver's DBRef type does not support the $db field.

[evanchooly]

It might help to know what your use case is so we can see what possible solutions there might be.

[ozguraydinli]

In our use case, the application is running in a servlet container so the test with ThreadLocal is not exactly the same scenario. Can you try to implement something like: In a single thread,

• read from db1
• read from db2
• read from db1 again (this is where the mapping error happens)

somehow datastore instance keeps the db2 as its db instance for the second read operation on db1

[evanchooly]

Could you attach a test case recreating this? Unless you're doing something with the DatastoreProvider, Morphia's only ever going to load from the one database. Datastore.save() is an explicit call to a Datastore where a load eventually makes its way in to a Mapper which gets the Datastore to use from Mapper.getDatastore() which just delegates to the DatastoreProvider.

[ozguraydinli]

Can you try to run this test please? You can paste it inside TestDatastore.java

 @Entity(noClassnameStored = true)
    public static class Foo {
        @Id
        private long id;

        private String fooField;

        @Reference
        private List<Bar> barReferences;

        public Foo() {

        }

        public Foo(String fooField) {
            this.fooField = fooField;
        }

        public Foo(long id, String fooField) {
            this.id = id;
            this.fooField = fooField;
        }

        public long getId() {
            return id;
        }

        public void setId(long id) {
            this.id = id;
        }

        public String getFooField() {
            return fooField;
        }

        public void setFooField(String fooField) {
            this.fooField = fooField;
        }

        public List<Bar> getBarReferences() {
            return barReferences;
        }

        public void setBarReferences(List<Bar> barReferences) {
            this.barReferences = barReferences;
        }
    }

    @Entity(noClassnameStored = true)
    public static class Bar {
        @Id
        private long id;

        private String barField;

        public Bar() {

        }

        public Bar(long id, String barField) {
            this.id = id;
            this.barField = barField;
        }

        public Bar(String barField) {
            this.barField = barField;
        }

        public long getId() {
            return id;
        }

        public void setId(long id) {
            this.id = id;
        }

        public String getBarField() {
            return barField;
        }

        public void setBarField(String barField) {
            this.barField = barField;
        }
    }

    private void saveDataFor_testMultipleDatabases2() {
      Datastore db1 = getMorphia().createDatastore(getMongoClient(), "db1");
      Datastore db2 = getMorphia().createDatastore(getMongoClient(), "db2");

      final FacebookUser db1FacebookUser = new FacebookUser(4, "DB1 FaceBook Friend");
      db1.save(db1FacebookUser);

      Bar bar1 = new Bar(1, "bar 1");
      db2.save(bar1);
      Bar bar2 = new Bar(2, "bar 2");
      db2.save(bar2);

      ArrayList<Bar> bars = new ArrayList<Bar>();
      bars.add(bar1);
      bars.add(bar2);

      Foo foo1 = new Foo(3, "Foo 1");
      foo1.setBarReferences(bars);

      db2.save(foo1);
    }

    @Test
    public void testMultipleDatabases2() throws InterruptedException, TimeoutException, ExecutionException {

      Datastore db1 = null;
      Datastore db2 = null;

      try {
          saveDataFor_testMultipleDatabases2();

          // below is almost the same use case we are trying.
          // the order is important

          db2 = getMorphia().createDatastore(getMongoClient(), "db2");

          List<Foo> foos = db2.find(Foo.class).asList();
          Assert.assertEquals(1, foos.size());

          Foo foo = foos.get(0);
          Assert.assertEquals(2, foo.getBarReferences().size());

          db1 = getMorphia().createDatastore(getMongoClient(), "db1");
          List<FacebookUser> facebookUsers = db1.find(FacebookUser.class).asList();
          Assert.assertEquals(1, facebookUsers.size());

          List<Foo> foos2 = db2.find(Foo.class).asList();
          Assert.assertEquals(1, foos2.size());

          Foo foo2 = foos2.get(0);
          Assert.assertEquals(2, foo2.getBarReferences().size());

      } finally {
          if(db1 != null){
            db1.getDB().dropDatabase();
          }

          if(db2 != null) {
            db2.getDB().dropDatabase();
          }
        }
    }

[evanchooly]

Datastore and Mapper are bound in some awkward ways at the moment. The difference between my test case and yours is that with the ThreadLocalDatastore, I could provide a contextual distinction between which db I was needing to use. In your test, you simply bounce between DBs. Everything is fine up until you try to find foos2. Up to that point, you'd create a new Datastore each time would register that Datastore with the DatastoreProvider in Mapper. When you try to fetch foos2, you don't do that so the Mapper uses the Datastore that was last registered which is the one that points at db1. If you want to switch dbs like that, you'll have to explicitly call mapper.getDatastoreProvider().register(db2).

Admittedly this is a bit of a wart in Morphia's architecture but one that can't change this close to 1.0.

[heruan]

:+1: Referencing from one datastore to another is vital in large environments and applications; since Mongo DBRefs support the $db field, it would be awesome to have Morphia behave seamlessly with them.

[evanchooly]

I'm not 100% sure this will make it to 1.1 but we'll see. The Java driver doesn't support the $db field in DBRef so i'll have to munge it in Morphia. This may or may not work but I'll give it my best shot.

[heruan]

I agree it should be handled natively by the Java driver and let Morphia act just as interface. Here's the Java driver issue: https://jira.mongodb.org/browse/JAVA-855

[fixl]

We're running up against this issue inside an OSGI container. As @ozguraydinli mentioned, ThreadLocal is not workable for us because we can't guarantee which thread we're on.

We've worked around this issue for now:

public final class MorphiaSupport {

    private MorphiaSupport() {}

    public static Datastore createDatastore(MongoClient mongoClient, String dbName, Class... classesToMap) {
        return createDatastore(new Morphia(), mongoClient, dbName, classesToMap);
    }

    public static Datastore createDatastore(Morphia morphia, MongoClient mongoClient, String dbName, Class... classesToMap) {
        Datastore ds = morphia.map(classesToMap).createDatastore(mongoClient, dbName);

        DatastoreProvider provider = new SingleDatastoreProvider();
        provider.register(ds);

        morphia.getMapper().getOptions().setDatastoreProvider(provider);

        return ds;
    }
}

}

EDIT: On second thought, the problem we're facing isn't that we're referencing documents in different databases. The problem is that we create Datastores in different bundles referring to different databases. However, when using @Reference in a pojo, Morphia refers to the wrong Datastore (sometimes, depending on the order in which bundles are started).

[evanchooly]

The fix I have in mind should obviate the whole concept of a DatastoreProvider and should be much more friendly in such scenarios.

[dklotz]

This may be a bit of a "me too", but please keep in mind that this bug is not just about references from one database to another, but can can also break (and does, in our case) references in the same db, as long as you query multiple databases in your application. I can confirm what @ozguraydinli spotted: ReferenceMapper.resolveObject will use a Datastore with a wrong database name in that case.

[evanchooly]

Yeah, the problem is the DatastoreProvider which seems to be a vestigial component in Morphia. The best fix would be to remove it as it's not really necessary but I'm afraid it'd break the few who have explicitly relied on it. I'm attempting to prototype a solution that fixes this while not breaking any uses of the interface.

[relferreira]

It seems that this topic is closed, but I could not found how to make this work anywhere. Is this functionality really available? Thanks in advance!

[evanchooly]

Yes. It was included in the 1.1.0 version (as indicated by the milestone). If you're having issues, please post a test case to the mailing list.