[Bug] App Crashes if other FIrebase-using app is open at the same time.

JudahMantell commented 2 years ago

[REQUIRED] Please fill in the following fields:

Unity editor version: _2021.3.2
Firebase Unity SDK version: 9.1.0
Source you installed the SDK: UPM
Problematic Firebase Component: All (Auth, Database, etc.)
Other Firebase Components in use: Auth, Firestore, Storage(Auth, Database, etc.)
Additional SDKs you are using: None (Facebook, AdMob, etc.)
Platform you are using the Unity editor on: Mac/PC (Mac, Windows, or Linux)
Platform you are targeting: Mac/PC (iOS, Android, and/or desktop)
Scripting Runtime: Mono (Mono, and/or IL2CPP)
Pre-built SDK from the website or open-source from this repo: pre-built

[REQUIRED] Please describe the issue here:

(Please list the full steps to reproduce the issue. Include device logs, Unity logs, and stack traces if available.) This one is pretty cut-and-dry. If I have my project open in the editor, then open up a build, when it tries to run any firebase code, the build will crash immediately. I thought this was because it's two instances of the same firebase app (which makes vague sense to me...?), but then I got a bug report from one of my users saying the build crashes immediately.

I did some digging and lo and behold, their AVG antivirus uses Firebase Analytics!. That's even mention in the player.logs (attached)

Steps to reproduce:

The repo rate is 100% for me. Simply try to open a build of your firebase app at the same time as the editor (and sign in, or call some firebase code), and either the build or the editor should crash.

If you have a downloadable sample project that reproduces the bug you're reporting, you will likely receive a faster response on your issue.

Relevant Code:

Because this is so generic, I don't have any specific code that would cause it.

Player.log Player-prev.log

Has anyone else encountered this? Any resolution? Thanks!

Edit with some more info: It looks like Auth is working properly, but when I need to query from FireStore (which I do right after a successful login), it crashes.

paulinon commented 2 years ago

Hi @SceneForgeStudio,

Thanks for reporting this. Could you provide a minimal, reproducible example of your implementation along with the security rules used for Cloud Firestore so we could analyze what's causing this behavior?

JudahMantell commented 2 years ago

I can't provide an example because the firebase api I'm using is a paid product, but the developer of it said that all bugs like this should be present with the base API as well.

But the user flow is this:

Sign in with email + password
Query a Firestore document
Crash

My rules are as follows:

rules_version = '2';
service cloud.firestore {
  match /databases/{database}/documents {
    match /{document=**} {
      allow read, write: if request.auth!=null;
    }
    match /customers/{uid} {
      allow read: if request.auth.uid == uid;

      match /checkout_sessions/{id} {
        allow read, write: if request.auth.uid == uid;
      }
      match /subscriptions/{id} {
        allow read: if request.auth.uid == uid;
      }
    }

    match /products/{id} {
      allow read: if true;

      match /prices/{id} {
        allow read: if true;
      }
    }

  }
}

paulinon commented 2 years ago

Hi @SceneForgeStudio,

Thanks for the additional information. I created a sample project that performs the flow you described, and I was able to observe the issue, even with the latest SDK version (9.2.0). On my end, it's the editor that's crashing.

I'll relay my observations to the team, and you may refer to this thread for updates. For now, this will be marked as a bug.

JudahMantell commented 2 years ago

Hi @SceneForgeStudio,

Thanks for the additional information. I created a sample project that performs the flow you described, and I was able to observe the issue, even with the latest SDK version (9.2.0). On my end, it's the editor that's crashing.

I'll relay my observations to the team, and you may refer to this thread for updates. For now, this will be marked as a bug.

Ah, I'm glad I'm not crazy lol! I see that you built for MacOS. It's worth noting that this happens on Windows machines as well. For me personally, the crash with the editor open isn't that big of a deal. It's the fact that it crashes because of conflicts with other firebase-using software that's a big issue. It prevents users from using my app entirely.
Thanks!

dconeybe commented 2 years ago

@paulinon Could you attach your player.log so we can see the stack trace?

@SceneForgeStudio Can you try disabling Firestore persistence to see if it fixes the crash? If it does fix the crash, then this is a known issue that only one application at a time can access a specific Firestore with persistence enabled. This is because persistence is backed by leveldb which does not support simultaneous access by multiple processes.

paulinon commented 2 years ago

Hi @dconeybe,

Here are the latest logs I gathered: Playerlog.txt

JudahMantell commented 2 years ago

@paulinon Could you attach your player.log so we can see the stack trace?

@SceneForgeStudio Can you try disabling Firestore persistence to see if it fixes the crash? If it does fix the crash, then this is a known issue that only one application at a time can access a specific Firestore with persistence enabled. This is because persistence is backed by leveldb which does not support simultaneous access by multiple processes.

Thanks for the suggestion! Looking into disabling persistence using the API wrapper I'm using. I'll report back when I hear from the dev!

dconeybe commented 2 years ago

@paulinon Yes, you are experiencing the known issue:

ERROR: FIRESTORE INTERNAL ASSERTION FAILED: firestore_client.cc(198) void firebase::firestore::core::FirestoreClient::Initialize(const firebase::firestore::credentials::User &, const firebase::firestore::api::Settings &): Failed to open DB: Internal: Failed to open LevelDB database at [redacted]: Resource temporarily unavailable (expected created.ok())

@SceneForgeStudio If disabling persistence indeed fixes the problem, one workaround I've seen people use is to put a call to disable persistence inside of a #if UNITY_EDITOR block, so that in the Unity Editor it disables persistence to avoid conflicting with other instances of the application.

JudahMantell commented 2 years ago

@paulinon Yes, you are experiencing the known issue:
ERROR: FIRESTORE INTERNAL ASSERTION FAILED: firestore_client.cc(198) void firebase::firestore::core::FirestoreClient::Initialize(const firebase::firestore::credentials::User &, const firebase::firestore::api::Settings &): Failed to open DB: Internal: Failed to open LevelDB database at [redacted]: Resource temporarily unavailable (expected created.ok())
@SceneForgeStudio If disabling persistence indeed fixes the problem, one workaround I've seen people use is to put a call to disable persistence inside of a #if UNITY_EDITOR block, so that in the Unity Editor it disables persistence to avoid conflicting with other instances of the application.

I will definitely look into that! But it potentially doesn't fix interference with other apps though, right?

DellaBitta commented 2 years ago

@SceneForgeStudio Evidence points to crashing only on the same device at this point, but do you have evidence otherwise?

Thanks!

JudahMantell commented 2 years ago

@DellaBitta Sorry if I was unclear. Yes, I mean on the same device. But my main issue is beyond the Unity editor. From my experience, if the user has another firebase-using app open at the same time (say AVG Antivirus), that will interfere as well and cause a crash of the Unity firebase app. You can see that happening in these crash logs: Player.log Player-prev.log

Thanks!

DellaBitta commented 2 years ago

Ah, I see your point. Thanks for following up so quickly.

DellaBitta commented 2 years ago

If you could please let us know if the disabling persistence in Firestore fixes the problem for you, too. It would help us better understand the root cause. Thank you!

JudahMantell commented 2 years ago

Yup, I'll do my best to get right on this and report back here!

JudahMantell commented 2 years ago

So far disabling persistence fixes the build/editor crash!
Is there any practical changes in doing so though? Now I gotta give it a test in production with one of my users and I'll report back.

DellaBitta commented 2 years ago

In this case you should only need to disable persistence in your editor, not in production builds.

I wanted to circle back to the concern about other products than yours which use Firebase potentially causing a crash in your application. The reason the crash occurs is because the two runtimes are attempting to use the same directory on your computer at the same time. Disabling persistence makes the application skip disk storage, so that's why it fixes the issue in your editor.

The storage path is mangled by the app name. Therefore other apps (such as AVG) shouldn't cause a resource lock conflict and shouldn't cause a crash to occur.

I'm going to mark this issue as a duplicate since Issue #1284 in our Unity Quickstart SDK is tracking our fix for the problem. I'll leave this open for now in case any of my assertions above aren't true. :)

JudahMantell commented 2 years ago

@DellaBitta Interesting, thank you for the info! In the crash logs I attached above, it specifically mentions AVG though, and those were from a production build. You're saying that has nothing to do with the problem? Or are you saying that disabling persistence wouldn't fix that? Thanks!

dconeybe commented 2 years ago

@SceneForgeStudio Looking at Player.log attached to the OP, it looks like AVG anti-virus "injects" DLLs into your application (i.e. C:\Program Files (x86)\AVG\Antivirus\aswhook.dll and C:\Program Files (x86)\AVG\Antivirus\aswAMSI.dll). If they have statically-compiled Firebase into those DLLs then it's possible that this DLL injection injects an incompatible version of Firebase into your app, which sounds very bad and I could see this leading to crashes.

At the bottom of your Player.log, it also says

A crash has been intercepted by the crash handler. For call stack and other details, see the latest crash report generated in:
C:/Users/ISHMEL~1/AppData/Local/Temp/SceneForge Studio Inc_/SceneForge Studio/Crashes

Could you attach any relevant info from those crash logs?

To summarize, it looks like there are two distinct crashes occurring, which both manifest as the application appearing to "crash immediately" upon launch:

The "FIRESTORE INTERNAL ASSERTION FAILED" crash, which is fixed by disabling persistence when #UNITY_EDITOR is defined.
The crash caused by AVG Anti-Virus, which shows up as uWS::HttpSocket<0>::upgrade in the crash stack trace.

Since crash 1 is a known issue, this ticket should focus on crash 2.

JudahMantell commented 2 years ago

Working on getting the crash logs from the user now. Thanks for the info!

google-oss-bot commented 2 years ago

Hey @SceneForgeStudio. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

JudahMantell commented 2 years ago

@SceneForgeStudio Looking at Player.log attached to the OP, it looks like AVG anti-virus "injects" DLLs into your application (i.e. C:\Program Files (x86)\AVG\Antivirus\aswhook.dll and C:\Program Files (x86)\AVG\Antivirus\aswAMSI.dll). If they have statically-compiled Firebase into those DLLs then it's possible that this DLL injection injects an incompatible version of Firebase into your app, which sounds very bad and I could see this leading to crashes.

At the bottom of your Player.log, it also says
A crash has been intercepted by the crash handler. For call stack and other details, see the latest crash report generated in:
C:/Users/ISHMEL~1/AppData/Local/Temp/SceneForge Studio Inc_/SceneForge Studio/Crashes
Could you attach any relevant info from those crash logs?

To summarize, it looks like there are two distinct crashes occurring, which both manifest as the application appearing to "crash immediately" upon launch:

The "FIRESTORE INTERNAL ASSERTION FAILED" crash, which is fixed by disabling persistence when #UNITY_EDITOR is defined.

The crash caused by AVG Anti-Virus, which shows up as uWS::HttpSocket<0>::upgrade in the crash stack trace.

Since crash 1 is a known issue, this ticket should focus on crash 2.

Ah okay, I got more logs and crash dumps, sorry for the delay. Tracking down users isn't as easy as I first thought! Using WinDbg I can see that crash.dmp says Illegal Instruction... I have no idea what that means, hopefully someone here can shed some light on that?

If there's a better way of learning more about the crash dumps, let me know!
Three of them are attached in a zipped folder, as GitHub doesn't accept .dmp attachments.

Thanks!

FILE_IN_CAB:  crash.dmp

CONTEXT:  (.ecxr)
rax=0000000000000008 rbx=0000029abbd90003 rcx=0000000000290040
rdx=0000000000000001 rsi=00000299d658bb00 rdi=0000009ccd49ed40
rip=00007ffb9067a3bd rsp=0000009ccd49ec70 rbp=000000000000003d
 r8=0000029abbd9003f  r9=00000000000000ff r10=0000029abbd90003
r11=0000000000000000 r12=0000000000000004 r13=0000000000000310
r14=00000000000000f0 r15=0000009ccd49ed18
iopl=0         nv up ei pl nz na pe nc
cs=0033  ss=002b  ds=002b  es=002b  fs=0053  gs=002b             efl=00010202
FirebaseCppApp_9_0_0+0x31a3bd:
00007ffb`9067a3bd c4e278f5e9      bzhi    ebp,ecx,eax
Resetting default scope

EXCEPTION_RECORD:  (.exr -1)
ExceptionAddress: 00007ffb9067a3bd (FirebaseCppApp_9_0_0+0x000000000031a3bd)
   ExceptionCode: c000001d (Illegal instruction)
  ExceptionFlags: 00000000
NumberParameters: 0

PROCESS_NAME:  SceneForge Studio.exe

ERROR_CODE: (NTSTATUS) 0xc000001d - {EXCEPTION}  Illegal Instruction  An attempt was made to execute an illegal instruction.

EXCEPTION_CODE_STR:  c000001d

IP_ON_STACK: 
+0
0000009c`cd49eca0 0200            add     al,byte ptr [rax]

FRAME_ONE_INVALID: 1

STACK_TEXT:  
0000009c`cd49ec70 0000009c`cd49eca0     : 00000299`d658bb00 00000000`00000000 0000009c`cd49ed40 00000299`d658bb00 : FirebaseCppApp_9_0_0+0x31a3bd
0000009c`cd49ec78 00000299`d658bb00     : 00000000`00000000 0000009c`cd49ed40 00000299`d658bb00 0000029a`bbd9028f : 0x0000009c`cd49eca0
0000009c`cd49ec80 00000000`00000000     : 0000009c`cd49ed40 00000299`d658bb00 0000029a`bbd9028f 0000029a`bbd90002 : 0x00000299`d658bb00

FAILED_INSTRUCTION_ADDRESS: 
FirebaseCppApp_9_0_0+31a3bd
00007ffb`9067a3bd c4e278f5e9      bzhi    ebp,ecx,eax

SYMBOL_NAME:  FirebaseCppApp_9_0_0+31a3bd

MODULE_NAME: FirebaseCppApp_9_0_0

IMAGE_NAME:  FirebaseCppApp-9_0_0.dll

STACK_COMMAND:  ~102s; .ecxr ; kb

FAILURE_BUCKET_ID:  ILLEGAL_INSTRUCTION_c000001d_FirebaseCppApp-9_0_0.dll!Unknown

OS_VERSION:  10.0.19041.1

BUILDLAB_STR:  vb_release

OSPLATFORM_TYPE:  x64

OSNAME:  Windows 10

FAILURE_ID_HASH:  {8ade77c5-2806-959a-f2f0-38ba15fe1f1f}

crash dumps.zip

dconeybe commented 2 years ago

Hi @SceneForgeStudio. The "illegal instruction" error means that the process tried to execute a CPU instruction that is invalid. This is typically a symptom of memory corruption or the instruction pointer pointing at some arbitrary location in memory. It could also happen if you application was compiled with some newer CPU features enabled but the customer is running on an older CPU that doesn't support those new features. Basically, it's an error that has a large number of potential root causes.

Unfortunately, I don't have the bandwidth to debug this issue (although I'd love to... it sounds quite interesting). What I would recommend is to install windbg and use it to open the crash dump files to see what information you can glean from them: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/analyzing-a-user-mode-dump-file#windbg. All I can tell from the crash dump information you provided in the previous comment is that something went wrong somewhere in FirebaseCppApp-9_0_0.dll, or at least FirebaseCppApp-9_0_0.dll was the unlucky part of the application where the memory corruption manifested.

I also should point out that using the Firebase Unity SDK in desktop apps is in beta and "is intended only for workflows during the development of your game, not for publicly shipping code" (https://firebase.google.com/docs/unity/setup#desktop-workflow). So we can only offer very minimal support to issues you're experiencing. If I understand correctly, your app is "SceneForge Studio" which is a desktop app and is trying to make use of Firestore, which we don't support for production desktop applications. There's nothing technically stopping you from doing this, but since it's an unsupported setup, our team is not resourced to provide support for it.

Finally, I'd encourage you to upgrade to the latest version of the Firebase Unity SDK, which, at the moment, is 9.3.0. Keeping up with the latest release makes any support that we can provide much easier.

Sorry for the not-so-great news. I wish you well in your debugging. If you're able to get more information about the crash I can definitely take a look.

JudahMantell commented 2 years ago

Thank you @dconeybe , I appreciate it! I will definitely keep digging and see what I can find.
I understand that the desktop functionality is still in beta; it works fine for every user besides this one, that's why this is so strange to me. In any case, thank you again for the info, and I will update this thread if more is found.

google-oss-bot commented 2 years ago

Hey @SceneForgeStudio. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

google-oss-bot commented 2 years ago

Since there haven't been any recent updates here, I am going to close this issue.

@SceneForgeStudio if you're still experiencing this problem and want to continue the discussion just leave a comment here and we are happy to re-open this.

firebase / firebase-unity-sdk