Emulator dashboard timeout when accessing exported data

[MorenoMdz]

[REQUIRED] Environment info

Tools v11.1.0 firebase-tools:

Platform: MacOS

[REQUIRED] Test case

Loading any kind of production seed makes the emulator UI timeout.

[REQUIRED] Steps to reproduce

Load an export from Firestore, at around 50mb is enough o make the UI not work.

[REQUIRED] Expected behavior

Be able to access all documents via the emulator UI

[REQUIRED] Actual behavior

Emulator loads, the data is there as our apps can access it normally via their SDK, but the emulator UI can't.

https://user-images.githubusercontent.com/29472480/174684112-53fea825-a0db-4df3-8c74-7553f456e46c.mov

firebase-debug.log firestore-debug.log

[Spitzbua]

We have had the same problem for many months. If collections contain too many documents, they cannot be displayed or edited. When the productive dumps became larger and larger, these problems occur. In our case, it is also no longer possible to export the data...

[MorenoMdz]

We have had the same problem for many months. If collections contain too many documents, they cannot be displayed or edited. When the productive dumps became larger and larger, these problems occur. In our case, it is also no longer possible to export the data...

Is there an actual limit of what the data size can be? Its been very very useful to be able to dump our prod stage and run some local flows in it to simulate real-world QA. Also, one weird thing is that other colleagues machines still run the dumps, I wonder it if it is related to the version of Java or something

[christhompsongoogle]

Right now we don't have a defined limit on the emulator suite, however these aren't (generally) intended for a full dump of prod data. I'll take the feedback that we could have a more explicit upper limit on the capacity of the emulator suite.

There's a stackoverflow question here too that 50MB of data doesn't import: https://stackoverflow.com/questions/73364863/firebase-firestore-emulator-not-importing-all-documents?noredirect=1#comment129568493_73364863

[MorenoMdz]

Right now we don't have a defined limit on the emulator suite, however these aren't (generally) intended for a full dump of prod data. I'll take the feedback that we could have a more explicit upper limit on the capacity of the emulator suite.

There's a stackoverflow question here too that 50MB of data doesn't import: https://stackoverflow.com/questions/73364863/firebase-firestore-emulator-not-importing-all-documents?noredirect=1#comment129568493_73364863

I have a feeling that the data loading might (also) be related to the data structure itself and not just the size of the dataset, for example, we have only a max of 3 nested sub-collection levels, but some of our root collections have thousands of documents. The emulator loads our prod dumps just fine, but the emulator UI timeout when trying to access it and doesn't display the documents, only the collections.

[MorenoMdz]

It looks like FS emulator is killing Java with OOM when trying to access the prod data:

Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "server-timer"
 {"metadata":{"emulator":{"name":"firestore"},"message":"\nException: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread \"server-timer\"\n"}}
[debug] [2022-12-04T18:35:35.455Z] 
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "HTTP-Dispatcher"
 {"metadata":{"emulator":{"name":"firestore"},"message":"\nException: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread \"HTTP-Dispatcher\"\n"}}
[debug] [2022-12-04T18:35:36.291Z] Dec 04, 2022 1:35:36 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError
INFO: operation failed: Java heap space: failed reallocation of scalar replaced objects
java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects

 {"metadata":{"emulator":{"name":"firestore"},"message":"Dec 04, 2022 1:35:36 PM com.google.cloud.datastore.emulator.impl.util.WrappedStreamObserver onError\nINFO: operation failed: Java heap space: failed reallocation of scalar replaced objects\njava.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects\n\n"}}