search

realm / realm-swift

Realm is a mobile database: a replacement for Core Data & SQLite
https://realm.io
Apache License 2.0
16.28k stars 2.15k forks source link

Does @ObservedResults async-open Realm? #7622

Closed ksiwei closed 2 years ago

ksiwei commented 2 years ago

How frequently does the bug occur?

Sometimes

Description

During the initial sync after user logins, the app sometimes crashes after 10~20 seconds due to terminating with uncaught exception of type realm::AddressSpaceExhausted: mmap() failed: Cannot allocate memory size: 26279936 offset: 134217728

I removed all the occurrences of realm() that are opened synchronously after seeing this issue: https://github.com/realm/realm-swift/issues/7040 3 but still couldn’t avoid it.

I use @ObservableResults in many places, I wonder if they are causing the crash.

Stacktrace & log output

No response

Can you reproduce the bug?

Yes, sometimes

Reproduction Steps

No response

Version

10.16.0

What SDK flavour are you using?

MongoDB Realm (i.e. Sync, auth, functions)

Are you using encryption?

No, not using encryption

Platform OS and version(s)

iOS14

Build environment

Xcode version: ... Dependency manager and version: ...

leemaguire commented 2 years ago

Hi @ksiwei. @ObservedResults does not open the Realm asynchronously. How much storage is available on the test device you are using? Could you also post some code showing how you are using Realm?

ksiwei commented 2 years ago

Sorry about the delay. Just to give you a little more context. My app is a photo app that adopts an image handling strategy that is similar to this: https://www.mongodb.com/developer/how-to/realm-data-architecture-ofish-app/#handling-images. We save image blobs to Realm, and an Atlas trigger actively uploads the blob to s3 and replaces it with an image URL.

The crash happens to people with a lot of images. When I set logLevel = .all I can see Realm was trying to apply every changeset throughout the sync history. Some of the changesets contain image blob. I wonder whether Realm is keeping references to a lot of those changesets with blobs in memory that eventually grew too large?

Also, does realm always apply all the changesets? is there a way to limit the number of changesets to be applied or squash them?

Sample log before the crash:

InternStrings   0="imageData", 1="P", 2="cards", 3="9CCA9FB6-81CB-49ED-BE64-C7E9812491B3"
Update          path=P["9CCA9FB6-81CB-49ED-BE64-C7E9812491B3"].cards[0].imageData, value=Binary(...), default=0
2022-02-01 17:43:29.614621+0000 XXX[739:46540] Sync: Connection[1]: Received: DOWNLOAD CHANGESET(server_version=2090, client_version=0, origin_timestamp=221730758462, origin_file_ident=56, original_changeset_size=70204, changeset_size=70204)
2022-02-01 17:43:29.623764+0000 XXX[739:46540] Sync: Connection[1]: Changeset(comp): 70204
2022-02-01 17:43:29.630874+0000 XXX[739:46540] Sync: Connection[1]: Changeset (parsed):
InternStrings   0="imageData", 1="P", 2="cards", 3="1166D3B3-7F69-4998-8BF6-A921CDB53B4F"
Update          path=P["1166D3B3-7F69-4998-8BF6-A921CDB53B4F"].cards[0].imageData, value=Binary(...), default=0
2022-02-01 17:43:29.632512+0000 PicoJar[739:46540] Sync: Connection[1]: Session[1]: Received: DOWNLOAD(download_server_version=2090, download_client_version=0, latest_server_version=4029, latest_server_version_salt=2166913025483924558, upload_client_version=0, upload_server_version=0, downloadable_bytes=114628511, num_changesets=12, ...)
2022-02-01 17:43:29.639893+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [1/12] (1 instructions)
2022-02-01 17:43:29.640032+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [2/12] (1 instructions)
2022-02-01 17:43:29.640086+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [3/12] (1 instructions)
2022-02-01 17:43:29.640133+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [4/12] (1 instructions)
2022-02-01 17:43:29.640241+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [5/12] (1 instructions)
2022-02-01 17:43:29.640295+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [6/12] (1 instructions)
2022-02-01 17:43:29.640350+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [7/12] (1 instructions)
2022-02-01 17:43:29.640394+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [8/12] (1 instructions)
2022-02-01 17:43:29.641121+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [9/12] (1 instructions)
2022-02-01 17:43:29.641493+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [10/12] (6 instructions)
2022-02-01 17:43:29.641588+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [11/12] (1 instructions)
2022-02-01 17:43:29.642191+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning incoming changeset [12/12] (1 instructions)
2022-02-01 17:43:29.642265+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning local changeset [1/3] (0 instructions)
2022-02-01 17:43:29.642322+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning local changeset [2/3] (4 instructions)
2022-02-01 17:43:29.642377+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Scanning local changeset [3/3] (8 instructions)
2022-02-01 17:43:29.642445+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [1/12] (1 instructions)
2022-02-01 17:43:29.642512+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [2/12] (1 instructions)
2022-02-01 17:43:29.642553+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [3/12] (1 instructions)
2022-02-01 17:43:29.642594+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [4/12] (1 instructions)
2022-02-01 17:43:29.643289+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [5/12] (1 instructions)
2022-02-01 17:43:29.643359+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [6/12] (1 instructions)
2022-02-01 17:43:29.643404+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [7/12] (1 instructions)
2022-02-01 17:43:29.643463+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [8/12] (1 instructions)
2022-02-01 17:43:29.643505+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [9/12] (1 instructions)
2022-02-01 17:43:29.643553+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [10/12] (6 instructions)
2022-02-01 17:43:29.643646+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [11/12] (1 instructions)
2022-02-01 17:43:29.644006+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Indexing incoming changeset [12/12] (1 instructions)
2022-02-01 17:43:29.644611+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Finished changeset indexing (incoming: 12 changeset(s) / 17 instructions, local: 3 changeset(s) / 12 instructions, conflict group(s): 16)
2022-02-01 17:43:29.644682+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Transforming local changeset [1/3] through 12 incoming changeset(s) with 16 conflict group(s)
2022-02-01 17:43:29.644720+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Transforming local changeset [2/3] through 12 incoming changeset(s) with 16 conflict group(s)
2022-02-01 17:43:29.645367+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Transforming local changeset [3/3] through 12 incoming changeset(s) with 16 conflict group(s)
2022-02-01 17:43:29.645726+0000 XXX[739:46540] Sync: Connection[1]: Session[1]: Finished transforming 3 local changesets through 12 incoming changesets (12 vs 17 instructions, in 16 conflict groups)
libc++abi: terminating with uncaught exception of type realm::AddressSpaceExhausted: mmap() failed: Cannot allocate memory size: 24657920 offset: 134217728
pavel-ship-it commented 2 years ago

Hi @ksiwei Is it possible to run the app with Instruments utility to see how many memory takes the app for the images? Depending on what and how the app does to image data it could explain the lack of free space. Let's say image file will take only several MBytes as NSData but the same image can take tenths of MBytes as UIImage. The solution can be to sync the images one by one and/or to check the free space before opening next photo. Any way you can get more info from Instruments Allocations tool.

ksiwei commented 2 years ago

Thanks @pavel-ship-it I will try to play around with the instrument allocation tool - haven't really properly used it.

Weirdly it only runs out of memory during the initial sync - I will double-check if that still happens if I comment out all the UI code.

We do have some chunking logic to make sure every write is under 15MB. 

        let fifteenMB = 15 * 1024 * 1024
        let imageChunks = images.chunked(into: 5)
        for chunk in imageChunks {
            let totalBytesForImages = chunk.reduce(0) { acc, p in
                return acc + (p.imageData?.count ?? 0)
            }
            if totalBytesForImages < fifteenMB {
                try! realm.write {
                    realm.add(chunk)
                }
            } else {
                for image in imageChunk {
                    try! realm.write {
                        realm.add(image)
                    }
                }
            }
        }
leemaguire commented 2 years ago

@ksiwei the memory bloating you are encountering looks related to the use of writes inside the for loops. I would suggest moving the write block outside of the for loop like so:

     let fifteenMB = 15 * 1024 * 1024
     let imageChunks = images.chunked(into: 5)
     try ! realm.write {
        for chunk in imageChunks {
            let totalBytesForImages = chunk.reduce(0) { acc, p in
                return acc + (p.imageData?.count ?? 0)
            }
            if totalBytesForImages < fifteenMB {
                    realm.add(chunk)
            } else {
                for image in imageChunk {
                     realm.add(image)
                }
            }
        }
    }