DatastoreTemplate or Repository does not fetch SubChild Descendants (Multiple parent keys not supported)

[sanveer-osahan]

I have a Parent/Child/SubChild relationship modeled as follows using @Descendants

@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity(name = "parent")
public class Parent {
    @Id
    Long id;

    @Descendants
    List<Child> children;
}

@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity(name = "child")
public class Child {
    @Id
    Key id;

    @Descendants
    List<SubChild> subChildren;
}

@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity(name = "subchild")
public class SubChild {
    @Id
    Key id;

    String someValue;
}

When you try to retrieve the parent object(s) using a DatastoreTemplate or Repository, they contain the child records but the child records do not contain subchild records. Even if you try to recieve only the child object(s), they do not retrieve their subchild records.

Steps to Reproduce

I have created the following JUnit test to explain the issue:

@Test
void datastoreTest() {
    Key parentKey = Key.newBuilder(gcpDataStoreProjectId, "parent", 1).setNamespace(gcpDataStoreNameSpace).build();
    Key childKey = Key.newBuilder(parentKey, "child", 2).build();
    Key subChildKey = Key.newBuilder(childKey, "subchild", 3).build();

    SubChild subChild = new SubChild(subChildKey, "someValue");
    Child child = new Child(childKey, Arrays.asList(subChild));
    Parent parent = new Parent(1L, Arrays.asList(child));

    datastoreTemplate.save(parent);
    Assertions.assertEquals(parent, datastoreTemplate.findById(parent.getId(), Parent.class));
}

Expected Result

The assertion should pass as we are trying to fetch the same entity object which was saved.

Actual Result

The assertion fails as the fetched entity doesn't retrieve the subchild record(s).

Additional Steps

System.out.println(datastoreTemplate.findById(parent.getId(), Parent.class));
System.out.println(datastoreTemplate.findById(child.getId(), Child.class));
System.out.println(datastoreTemplate.findById(subChild.getId(), SubChild.class));

Parent(id=1, children=[Child(id=Key{projectId=projectId, namespace=default, path=[PathElement{kind=parent, id=1, name=null}, PathElement{kind=child, id=2, name=null}]}, subChildren=[])])

Child(id=Key{projectId=projectId, namespace=default, path=[PathElement{kind=parent, id=1, name=null}, PathElement{kind=child, id=2, name=null}]}, subChildren=[])

SubChild(id=Key{projectId=projectId, namespace=default, path=[PathElement{kind=parent, id=1, name=null}, PathElement{kind=child, id=2, name=null}, PathElement{kind=subchild, id=3, name=null}]}, someValue=someValue)

Current Workaround

For now, I'm using the following alternative:

Key parentKey = Key.newBuilder(gcpDataStoreProjectId, "parent", 1).setNamespace(gcpDataStoreNameSpace).build();
Key childKey = Key.newBuilder(parentKey, "child", "1#2").build();
Key childKeyForSubChild = Key.newBuilder(gcpDataStoreProjectId, "child", "1#2").setNamespace(gcpDataStoreNameSpace).build();
Key subChildKey = Key.newBuilder(childKeyForSubChild, "subchild", 3).build();

SubChild subChild = new SubChild(subChildKey, "someValue");
Child child = new Child(childKey, Arrays.asList(subChild));
Parent parent = new Parent(1L, Arrays.asList(child));
datastoreTemplate.save(parent);
System.out.println(datastoreTemplate.findById(parent.getId(), Parent.class));

Parent(id=1, children=[Child(id=Key{projectId=costoptimizationproject, namespace=test, path=[PathElement{kind=parent, id=1, name=null}, PathElement{kind=child, id=null, name=1#2}]}, subChildren=[SubChild(id=Key{projectId=costoptimizationproject, namespace=test, path=[PathElement{kind=child, id=null, name=1#2}, PathElement{kind=subchild, id=3, name=null}]}, someValue=someValue)])])

This solution fetches the children along with their subchildren because now subchild entity has only single parent key. But I'm compelled to use parentId#childId as combination for child entity id.

Can there be a support for retrieving sub-descendants with multiple parent keys?

[dmitry-s]

@sanveer-osahan Recursive save and retrieval works automatically for @Descendant properties. You need to leave the id property of children and subchildren to be null. It does not work when id fields hold values.

It is expected that fields of type Key are generated automatically, so manual use of Key.newBuilder should be avoided.

Please reopen if you have further questions.

[sanveer-osahan]

@dmitry-s My requirement is to store and retrieve child objects based on their parent, and subchild objects based on their parent and child. I have to organize my data in the following manner (sequential ids):

    P1            P2          
    /\            /\
  C1 C2         C1 C2
  /\               /\
S1 S2             S1 S2

Even if I use autogenerated keys, I will have to store sequential ids separately and for child and subchild, this will be an overhead because the parent field is used in datastore to track their ancestors and I am not able to take advantage of this functionality.

[dmitry-s]

@sanveer-osahan I'm not sure I understand what exactly you are trying to achieve. Do you need your child entities to be ordered? In that case, would embedded list work for you?

Also, could you describe your use case in more details? It is not clear at this point why you need to store the sequential ids.

Thanks!

[sanveer-osahan]

@dmitry-s I'm developing the following rest services:

GET /parents - Get all parents GET /parents/{id} - Get parent by id POST /parents - Add new parent

GET /parents/{pid}/children - Get all children for parent pid GET /parents/{pid}/children/{id} - Get child by id for parent pid POST /parents/{pid}/children - Add new child for parent pid

GET /parents/{pid}/children/{cid}/subchildren - Get all subchildren for parent pid and child cid GET /parents/{pid}/children/{cid}/subchildren{id} - Get subchildchild by id for parent pid and child cid POST /parents/{pid}/children/{cid}/subchildren - Add new subchild for parent pid and child cid

I have to use datastore for performing CRUD operations(Update and Delete are not a priority as of now). When GET /parents is called, the response should show data in sequence. For e.g.

[
  {
    "id": 1,
    "name": "parent1",
    "children": [
      {
        "id": 1,
        "name": "parent1_child1",
        "subchildren": [
          {
            "id": 1,
            "name": "parent1_child1_subchild1"
          },
          {
            "id": 2,
            "name": "parent1_child1_subchild2"
          }
        ]
      },
      {
        "id": 2,
        "name": "parent1_child2",
        "subchildren": [
          {
            "id": 1,
            "name": "parent1_child2_subchild1"
          },
          {
            "id": 2,
            "name": "parent1_child2_subchild2"
          },
          {
            "id": 3,
            "name": "parent1_child2_subchild3"
          }
        ]
      }
    ]
  },
  {
    "id": 2,
    "name": "parent2",
    "children": [
      {
        "id": 1,
        "name": "parent2_child1",
        "subchildren": [
          {
            "id": 1,
            "name": "parent2_child1_subchild1"
          }
        ]
      },
      {
        "id": 2,
        "name": "parent2_child2",
        "subchildren": [
          {
            "id": 1,
            "name": "parent2_child2_subchild1"
          },
          {
            "id": 2,
            "name": "parent2_child2_subchild2"
          },
          {
            "id": 3,
            "name": "parent2_child2_subchild3"
          },
          {
            "id": 4,
            "name": "parent2_child2_subchild4"
          }
        ]
      }
    ]
  }
]

[sanveer-osahan]

Update: The current workaround which I have mentioned is not working for spring cloud version Greenwich but working for Hoxton.

[dmitry-s]

@sanveer-osahan ok, I think I understand: you want to retrieve the descendants in order. This feature is not currently supported, but what we can do is to add an orderBy parameter to @Descendants annotation, so you could pass a field name and a direction.

You would have to introduce a new field in the child entity class that would be used for sorting (which is a good thing - generally, it is not recommended to use keys for sorting because if you need to change ordering you would have to change keys).

So your code would look something like this:

@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity(name = "parent")
public class Parent {
    @Id
    Long id;

    @Descendants(orderBy = "name ASC")
    List<Child> children;
}

@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity(name = "child")
public class Child {
    @Id
    Key id;

    @Descendants(orderBy = "position DESC")
    List<SubChild> subChildren;

    String name;
}

@Data
@NoArgsConstructor
@AllArgsConstructor
@Entity(name = "subchild")
public class SubChild {
    @Id
    Key id;

    String someValue;

    int position;
}

Would that work for you?

Also, have you tried using embedded lists? They are stored and retrieved in order. Would that work for your case or is there some functionality that you can only achieve with using descendants?

[sanveer-osahan]

@dmitry-s This will address one of the issues that I'm facing. The other issue is about performing CRUD operations. Suppose I want to retrieve just a subchild based on its ancestor's ids (parent and child), it would be much feasible if I can query using a key combination of parent id and child id. I'm not sure how to approach this using embedded lists. I'll look into it and get back to you.