search

grpc / grpc-dotnet

gRPC for .NET
Apache License 2.0
4.19k stars 769 forks source link

gRPC / EF Core - Many to Many Reference Loop? #1177

Closed JeepNL closed 3 years ago

JeepNL commented 3 years ago

Context:

I've created a simple (Kestrel Hosted Blazor WASM) 'Blog CMS' prototype for this question with a many to many relationship between 'Posts' and 'Tags'. When I uncomment the .Include line in BlogService.cs (see below), it results in a stack overflow error as you can see in the screenshot at the end of this post. (The error message repeats 1200+ times)

To me this looks like a reference loop error, like what you get with JSON if you don't include this code: with NewtonSoft.Json (JSON.NET) .AddNewtonsoftJson(x => x.SerializerSettings.ReferenceLoopHandling = Newtonsoft.Json.ReferenceLoopHandling.Ignore) or with System.Text.Json ReferenceLoopHandling = ReferenceLoopHandling.Ignore

Question

FYI: I created a question at SO also but haven't received any comments so far (Jan. 20th).

Code:

(part of) /Server/Services/BlogService.cs

var posts = new Posts();
var allPosts = await dbContext.Posts
    .Where(ps => ps.PostStat == PostStatus.Published)
    .Include(pa => pa.PostAuthor)
    .Include(pe => pe.PostExt)
    //.Include(tipd => tipd.TagsInPostData) // TODO: [ERROR] / doesn't work: results in a stack overflow.
    .OrderByDescending(dc => dc.DateCreated)
    .ToListAsync();
posts.PostsData.AddRange(allPosts);
return posts;

EF Core generates the join table PostsTags 'automagically' from the protobuf contract:

(part of) /Shared/Protos/blog.proto

message Post {
    int32 post_id = 1;
    int32 author_id = 2;
    string title = 3;
    string date_created = 4; // DateTime (UTC) string because of SQLite
    PostStatus post_stat = 5; // enum
    PostExtended post_ext = 6; // one to one
    Author post_author = 7; // Post with one author, one to one
    repeated Tag tags_in_post_data = 8; // Post with many Tags
}
message Posts {
    repeated Post posts_data = 1;
}

message Tag {
    string tag_id = 1; // Tag itself: string
    //int32 tag_id = 1;
    //string name = 2;
    repeated Post posts_in_tag_data = 2; // Tag with many Posts
}
message Tags {
    repeated Tag tags_data = 1;
}

I've asked a question in the EF Core repo on how to add data to this PostsTags join table, and this is possible by adding the 2 lines of code below to /Server/Data/ApplicationDbContext.cs

modelBuilder.Entity<Post>().Navigation(e => e.TagsInPostData).HasField("tagsInPostData_");
modelBuilder.Entity<Tag>().Navigation(e => e.PostsInTagData).HasField("postsInTagData_");

Console Error:

Screenshot 2021-01-20 144817

Quick Links to files mentioned here (+ SeedData.cs)

JamesNK commented 3 years ago

The Protobuf serializer doesn't support reference loops. You'll need to remove the loop, or change your code so that when you create the Protobuf objects, the loop is never created.

JeepNL commented 3 years ago

[EDIT] UPDATE (2.10 PM GMT + 1)

Please wait with a reply: I think I've found a solution! I don't know exactly what I'm doing, and I'm probably doing it wrong, but it looks like it works.

I've to do some more testing (and learning) but I'll post my solution here some time later today.

[OLD] (1.07 PM GMT + 1)

Thank you for your reply, your answer helps actually to narrow down my search for a solution (Serializer doesn't support reference loops). I hope you don't mind but I've a couple of short questions about this & gRPC related only, so I know better what to look for.

Do you mean I've to remove the 'loop' in the protobuf definition file between the protobuf messages Post, Posts and Tag, Tags (see .proto definition example above) or do you mean I need to remove the loop (.Include) in the LINQ query? The protobuf definition Post(s)/Tag(s) autogenerates a many-to-many 'structure' ie: from this definition .NET gRPC Tooling autogenerates the .NET types and classes and EF Core autogenerates the dbContext code & tables in the database (even the Join Table which doesn't have protobuf definition). You wrote

or change your code so that when you create the Protobuf objects, the loop is never created

but all of this code is autogenerated, which is great but I can't change this I think?

In short: I hope I can solve this by learning more about EF Core/LINQ (somewhere else than here 😉) and write a better LINQ query which give me the result I want (all tags in each post) but without the loop back from Tag to Post. I still have to learn a lot about C#, EF Core and gRPC also so I hope this isn't a 'stupid question', I don't want to take up your time if this isn't related to gRPC and/or I just need to learn to write better code.

JeepNL commented 3 years ago

Again, thank you for your reply because because it made it clear to me that I didn't have to think about solving the reference loop in gRPC anymore but just try to modify my queries.

As I said, I've got it working now with the code below. I do not know it this can be simplified, I need to learn much more about mapping and flatten (if that's the correct phrase) objects.

If you want you can close this issue/question.

Screenshot 2021-01-21 165020

public override async Task<Posts> GetPosts(Empty request, ServerCallContext context)
{
    var postsQuery = await dbContext.Posts.AsSplitQuery() // trying/testing ".AsSplitQuery()"
    //var postsQuery = await dbContext.Posts
        .Where(ps => ps.PostStat == PostStatus.Published)
        .Include(pa => pa.PostAuthor)
        .Include(pe => pe.PostExtended)
        .Include(tipd => tipd.TagsInPostData)
        .OrderByDescending(dc => dc.DateCreated)
        .AsNoTracking().ToListAsync();

    // The Protobuf serializer doesn't support reference loops
    // see: https://github.com/grpc/grpc-dotnet/issues/1177#issuecomment-763910215
    //var posts = new Posts();
    //posts.PostsData.AddRange(allPosts); // so this doesn't work
    //return posts

    var posts = new Posts();
    foreach (var p in postsQuery)
    {
        var post = new Post();

        post.PostId = p.PostId;
        post.Title = p.Title;
        post.DateCreated = p.DateCreated;
        post.PostStat = p.PostStat;

        post.PostAuthor = p.PostAuthor;
        post.PostExtended = p.PostExtended;

        // Just add all the tags to each post, this isn't a reference loop.
        foreach (var t in p.TagsInPostData)
        {
            var tag = new Tag();
            tag.TagId = t.TagId;
            post.TagsInPostData.Add(tag);
        }
        posts.PostsData.Add(post);
    }
    return posts;
}
JunTaoLuo commented 3 years ago

Glad you were able to get it working!