googleapis / google-cloud-dotnet

Google Cloud Client Libraries for .NET
https://cloud.google.com/dotnet/docs/reference/
Apache License 2.0
942 stars 367 forks source link

Extremely large inner DocumentReferences #12959

Closed Mayco-Anderson closed 6 months ago

Mayco-Anderson commented 6 months ago

I am successfully recovering documents from firestore using the this package. Yet, one thing that I find really strange, is that my data models have some inner references to other DocumentReferences and while I would expect those references to be very light, like a string with the path to the document, they in reality are very large objects.

As an example, I have a data mode like "exam". The exam has documentReferences to another User document and a patient reference.
If I use documentSnapshot.ToDictionary(), the resulting dictionary is so large, that if I try to parse into a json and print it into the screen, I run out of memory. Yet, those documents should be very very light, less than 0.1 mb of data. From this dictionary, If I capture a DocumentReference alone, and save the resulting as a Json string, the resulting text file is more than 5mb of data.

Am I doing something wrong, or is there another way to force those DocumenReferences have less information? Following is my code:

   public async Task<List<T>> GetDocumentListAsync<T>(string collectionName, Func<Query, Query>? queryBuilder = null) where T : BaseFirestoreDocument<T>, new()
   {
       var collectionRef = _firestoreDb.Collection(collectionName);
       Query query = collectionRef;

       if (queryBuilder != null)
       {
           query = queryBuilder(query);
       }

       var snapshot = await query.GetSnapshotAsync();

       List<T> documents = new List<T>();
       Parallel.ForEach(snapshot.Documents, documentSnapshot =>
       {
           if (!documentSnapshot.Exists) return;

           var document = documentSnapshot.ConvertTo<T>();
           document.Reference = documentSnapshot.Reference;
           documents.Add(document);
       });

       return documents;
   }
jskeet commented 6 months ago

My guess is that it's due to the way you're converting the result to JSON, and I don't know what you're doing there.

Please could you post a short but complete program which demonstrates the problem (ideally including populating a small data set)? A DocumentReference itself is very small, but if you're using some form of JSON serialization which recursively walks all properties (which would include a reference to the FirestoreDb) then that could cause problems. (You probably just want to serialize the Path property.)

jskeet commented 6 months ago

Please provide more information - there's nothing more we can do without more details. I'll close this issue early next week if we don't hear anything more.

jskeet commented 6 months ago

Closing due to a lack of information.