FirebaseError: Document already exists

[mafergus]

[REQUIRED] Describe your environment

• Operating System version: Multiple (Windows 10, Android 10)
• Browser version: Multiple (Chrome Mobile 91.0.4472, Edge, Samsung Internet 13.0)
• Firebase SDK version: 8.4.3
• Firebase Product: Firestore

We are seeing this error when trying to add a document to a collection. I've scoured Google and Stack-Overflow, gone through the source code and also gprc source code. I can't understand the reason for this error:

firebase.firestore().collection('some-collection').add({ data: 1 })

As far as I can tell, this should never throw a "document exists" error. Is this valid behavior or a Firebase bug?

[schmidt-sebastian]

@mafergus Thanks for reporting this!

add() uses a random client-side generated identifier, so there is a small chance for a collision - but the chance for this is so small that we are willing to say that it should never occur. Are you able to provide debug logs for us, which will then show what identifier causes the collision?

You can enable debug logging by invoking setDebugLevel('debug').

[mafergus]

Yes we're happy to assist in any way possible, you guys are doing great work. Would you mind pointing me to the code that generates this identifier so I can better understand the algorithm? Perhaps we can change our code to mitigate any collisions. We've noticed this specifically during periods of peak activity.

We'll enable the debug logging, I would expect we'll hit this again over the weekend.

[schmidt-sebastian]

The code is here: https://github.com/firebase/firebase-js-sdk/blob/5ad7ff2ae955c297556223e6cb3ad9d4b897f664/packages/firestore/src/util/misc.ts#L28

[google-oss-bot]

Hey @mafergus. We need more information to resolve this issue but there hasn't been an update in 5 weekdays. I'm marking the issue as stale and if there are no new updates in the next 5 days I will close it automatically.

If you have more information that will help us get to the bottom of this, just add a comment!

[mafergus]

Hey @schmidt-sebastian, we captured a bunch of examples of this over the weekend in Sentry. As far as I'm aware, Sentry doesn't give a nice way to export breadcrumb logs. The two options I see are either pass you JSON of the breadcrumbs, or temporarily invite you to our Sentry project. Let me know how you'd like to proceed.

[schmidt-sebastian]

It depends how much logging we are talking about here. Please contact me at mrschmidt(at)google.com.

[oveddan]

I'm having this issue too

[schmidt-sebastian]

Are you able to catch and ignore these errors for now? We might need to adjust how these documents are processed internally to address this, which is major change for us. Essentially, what is happening here is that the SDK sends a write and then the connection gets dropped. The backend however executed the write. When we then try to send the write again, the backend tells us that the document already exists as our previous attempt did in fact succeed.

[oveddan]

For me this is a critical part of my application - i use the firestore document as a state record for a user that is created when they open my application, and a cloud function reads that document to determine where to route the user internally.

I've temporarily made a workaround (in the below code, store is my firestore db):

const requestToJoin = async ({
  spaceId,
  userId,
}: {
  spaceId: string;
  userId: string;
}) => {
  try {
    return await store.collection("joinRequests").add({
      spaceId,
      userId,
    });
  } catch (e) {
    console.error(e);
    try {
      console.error("failed joining, retrying");
      return await store.collection("joinRequests").add({
        spaceId,
        userId,
      });
    } catch (e) {
      console.error(e);
      return undefined;
      // notify user they cannot connect to the application
    }
  }
};

This works most of the time. Is there something I can do besides this to prevent this issue? It only started appearing a couple weeks ago.

I'm using firebase-js version 8

[oveddan]

I suppose a workaround is wrapping this add functionality in a cloud function that performs the adding, and the cloud function just returns the added element's id?

[schmidt-sebastian]

It is very likely that you can just ignore "Document already exists" error since it means that our previous attempt to write this document actually succeeded. We cannot do this from the SDK side (yet) as some other client may have interleaved a write and changed the document after the initial write attempt.

[oveddan]

I need the id of the document, so I don't think try catching and swallowing it would work? For now I will wrap it in a cloud fn.

[schmidt-sebastian]

Can you use setDoc(doc(collection), {...}) (v9) or collection.set(collection.doc(), {}) (v8)? This will use auto-generated document IDs but not verify that the document does not exist. Our document IDs are 20 bytes long, so the chance of collisions are low. SetDoc() will succeed if the SDK issues the write twice.

[vandres]

Also seeing the error very frequently in our application. We don't have that much traffic, so I also don't think, an ID collision is very likely.

[vinodgubbala]

Hope, addDoc becomes intelligent enough to mitigate this issue. The amount of errors is quite annoying. I tried setDoc solution. It is working good, but I am not really comfortable with using it on production.

[ginagr]

I am using addDoc and getting this error pretty regularly.

Code:

await addDoc(collection(this.firestore, `audit/${collectionPath}/${id}`), {   ...   });

Error:

[camr1993]

I am also running into this same issue with addDoc

[marshyon]

I am seeing the same issue with addDoc

[lukasvan3l]

I see this issue 4 times in my logs in the past weekend, and a rough guess of the succeeded amount of calls is 30K - 40K. My db structure is /users/{userId}/progress/{progressId} the 30/40K calls are distributed over 5K users, and these were new users, so the /progress collection was empty. So there are only 6 records on average for every user.

4 is a lot of occurrences with this little data, so it might be that there's something else going on than just coincidence in randomness...

[railerbailer]

I'm also experiencing this. I think it's the cause of crashing my entire application, but when authed on same user with incognito, no problemo. Any suggestions?

[MarkDuckworth]

Thanks for continuing to bring this to our attention, your feedback helps us understand how this affects you. We still believe the root cause to be as described in this Feb 14th comment. Additionally, please continue to refer to the workaround in this Feb 17th comment.

[raoneel]

I have figured out why this is happening for me. I am testing some ML libraries that use deterministic seeding (such as seed_everything). In some cases, this can cause the Firestore library to generate the exact same document ID as before. To fix this, I basically just re-seed the random libraries before calling .add(). This fixed the issue for me, hope it helps someone else.

[vojto]

We're getting this error in a transaction that looks like this:

await runTransaction(db, async (tx) => {
    const snapshot = await ref.get()
    if (snapshot.exists()) {
        throw new Error("document exists")
    }

    await ref.set(...)
})

I can't reproduce it, but we have a lot of Sentry logs with it.

The Document already exists error is really confusing here, because we're explicitly ensuring that the document doesn't exist before writing to it. Also using set method should never cause "document already exists", because that method is supposed to write whether it exists or not.

It is very likely that you can just ignore "Document already exists" error since it means that our previous attempt to write this document actually succeeded. We cannot do this from the SDK side (yet) as some other client may have interleaved a write and changed the document after the initial write attempt.

Are you telling me that the change is being saved, and I should just ignore the error? What if you decide to change things in SDK and now the error becomes something I shouldn't ignore?

Is there any chance you could at least rename that error to something like "Document maybe wasn't saved" or something more accurate?'

[vinodgubbala]

@vojto did you recently update your firbase version in your code base. We found this was expected behaviour. Earlier it used to not trigger error but trigger a warning.

So between get and set, if the document state changes the transaction gets failed.

[vojto]

So between get and set, if the document state changes the transaction gets failed.

Yep - the error started showing up only recently. And I've been able to reproduce it by running the code twice at the same time!

I thought in case of concurrent edits, the transaction would be automatically rerun:

A function calling a transaction (transaction function) might run more than once if a concurrent edit affects a document that the transaction reads. (docs)

[MarkDuckworth]

Hi @vojto, I reproduced the behavior you are talking about. I believe this is a different issue so I wrote it up as https://github.com/firebase/firebase-js-sdk/issues/6659.

Thanks for reporting

[avabhi]

Even I am also getting same issue with firebase version : 9.9.4

[dconeybe]

@vojto The "Document Already Exists" error you experienced is specific to transactions and was a bug that was accidentally introduced in v9.9.4. It will be fixed in the next release. See https://github.com/firebase/firebase-js-sdk/issues/6659 for details.

The "Document Already Exists" issue reported in the OP is not related to transactions, and still exists.

[appsgenie]

I am also seeing this with "firebase": "8.2.3" FirebaseError: [code=already-exists]: Document already exists Not using any transactions and simply doing firestore.collection('locations').add({..}) Seeing these sporadically in Sentry and no recent version updates have taken place. Wanted to see if it's ok to ignore these and if anything else might be happening.

[dconeybe]

@appsgenie That error is probably safe to ignore. What is likely happening is that the request is sent to the server to create the new document, then the client loses connection before the response is received from the server. When the client reconnects then it re-sends the "create document" request, which fails because while the document already exists because while the client was briefly offline the server completed the request to create the new document.

[LanderBeeuwsaert]

Just to add we're having this as well. We're going to implement the workaround but a fix about this would of course be appreciated.

[vutoan]

Hi, I faced this issue in version 9.6.1 as well.

I cannot ignore this issue because after addDoc success, I have a callback function which is API call, but as you guys described the connection drop before the response of first addDoc reach client, then client re-run addDoc but get fail Document already exists , so my callback was never reached.

The workarround solution now is to use setDoc instead of addDoc right?

Thank you.

[MarkDuckworth]

Hi @vutoan, yes it sounds like the workaround setDoc(doc(collection), {...}) will work for you.

[joshkautz]

I'm having this same issue. There is certainly something currently amiss with addDoc.

My workaround to ensure that I always receive the document reference of the newly created document from addDoc, despite occasional errors:

const collectionReference = collection(firestore, 'collection');
const documentReference = await addDoc(collectionReference, { data: 'data'}).catch((error) => {
  return doc(collectionReference, error.message.split('/').at(-1));
});

This gets the newly added document's identifier from the error message itself:

FirebaseError: Document already exists: projects/projectName/databases/(default)/documents/collection/<<<documentId>>>

[ydax]

Possible Problem Source: Auth Reconnections?

I think I have an insight here that hasn't been mentioned yet.

Someone earlier in this thread said that this is likely because there's some sort of server disconnection for a moment and then the attempt to add the document reconnect to the server and I'm also seeing a lot of people that are reporting that this is happening in areas where people are logging in.

So, I think may be something that could be happening here is that auth is triggering some type of reconnection because the user has a new token or something like that and so if you're trying to basically add a document during this process, there's something going on where there's a sort of "blip" in the connection because of auth, which is causing this error to occur.

[dconeybe]

@ydax (or anyone else) Please enable Firestore debug logging (by calling setLogLevel('debug')) and capture the logs when the issue occurs. I know this may be really difficult since it seems to be end-users who are experiencing this and the bug is not reproducible by developers. But, if you do happen to be able to capture debug logs when this issue occurs please post them as that will greatly assist with any investigation.

[GaurangTandon]

Thanks @dconeybe for your tip. We're seeing the same error when we try to create new documents while the app is loading (takes long time to load all the documents as there are multiple Firebase round trips). I did setLogLevel('debug') and got the logs. They are a bit sensitive to share directly. However, I do notice the following relevant log lines that might be the issue:

...
16:27:55.145 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:55.145Z]  @firebase/firestore: Firestore (10.11.0): FirestoreClient Received user= USERID
16:27:55.145 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:55.145Z]  @firebase/firestore: Firestore (10.11.0): RemoteStore RemoteStore received new credentials
16:27:55.146 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:55.146Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Listen' stream 0x709d58e5 transport closed
16:27:55.147 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:55.147Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection Creating RPC 'Listen' stream 0x709d58e6: https://firestore.googleapis.com/google.firestore.v1.Firestore/Listen/channel {"httpSessionIdParam":"gsessionid","initMessageHeaders":{"X-Goog-Api-Client":"gl-js/ fire/10.11.0","Content-Type":"text/plain","Authorization":"Bearer x"},"messageUrlParams":{"database":"projects/org-name/databases/(default)"},"sendRawJson":true,"supportsCrossDomainXhr":true,"internalChannelParams":{"forwardChannelRequestTimeoutMs":600000},"forceLongPolling":true,"detectBufferingProxy":false,"useFetchStreams":true,"encodeInitMessageHeaders":true}
...
16:27:56.210 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:56.210Z]  @firebase/firestore: Firestore (10.11.0): RemoteStore RemoteStore received new credentials
16:27:56.211 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:56.211Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Listen' stream 0x709d58e6 transport closed
16:27:56.211 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:56.211Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Write' stream 0x709d58e7 transport closed
16:27:56.211 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:56.211Z]  @firebase/firestore: Firestore (10.11.0): RemoteStore Stopping write stream with 2 pending writes
16:27:56.211 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:56.211Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
16:27:56.212 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:56.212Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
16:27:56.212 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:56.212Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
16:27:56.212 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:56.212Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection Creating RPC 'Listen' stream 0x709d58e8: https://firestore.googleapis.com/google.firestore.v1.Firestore/Listen/channel {"httpSessionIdParam":"gsessionid","initMessageHeaders":{"X-Goog-Api-Client":"gl-js/ fire/10.11.0","Content-Type":"text/plain","Authorization":"Bearer x"},"messageUrlParams":{"database":"projects/org-name/databases/(default)"},"sendRawJson":true,"supportsCrossDomainXhr":true,"internalChannelParams":{"forwardChannelRequestTimeoutMs":600000},"forceLongPolling":true,"detectBufferingProxy":false,"useFetchStreams":true,"encodeInitMessageHeaders":true}
...
16:27:57.734 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.734Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
16:27:57.734 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.734Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Write' stream 0x709d58e9 sending: {"streamToken":"EAMZEGhCAoH0tbU=","writes":[{"update":{"name":"projects/org-name/databases/(default)/documents/snippets/H7tCTg7yinGqP7vugHDS","fields":{...},"updateTransforms":[{"fieldPath":"updated_at","setToServerValue":"REQUEST_TIME"},{"fieldPath":"created_at","setToServerValue":"REQUEST_TIME"}],"currentDocument":{"exists":false}}]}
16:27:57.734 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.734Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
...
16:27:57.957 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.957Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
16:27:57.959 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.959Z]  @firebase/firestore: Firestore (10.11.0): FirestoreClient Received user= USERID
16:27:57.959 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.959Z]  @firebase/firestore: Firestore (10.11.0): RemoteStore RemoteStore received new credentials
16:27:57.959 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.959Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Listen' stream 0x709d58e8 transport closed
16:27:57.960 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.960Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Write' stream 0x709d58e9 sending: {"streamToken":"EAQZEGhCAoH0tbU=","writes":[]}
16:27:57.960 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.960Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Write' stream 0x709d58e9 transport closed
16:27:57.960 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.960Z]  @firebase/firestore: Firestore (10.11.0): RemoteStore Stopping write stream with 4 pending writes
16:27:57.960 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.960Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
16:27:57.960 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.960Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
16:27:57.960 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.960Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: Get next mutation batch
16:27:57.961 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:57.961Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection Creating RPC 'Listen' stream 0x709d58ea: https://firestore.googleapis.com/google.firestore.v1.Firestore/Listen/channel {"httpSessionIdParam":"gsessionid","initMessageHeaders":{"X-Goog-Api-Client":"gl-js/ fire/10.11.0","Content-Type":"text/plain","Authorization":"Bearer x"},"messageUrlParams":{"database":"projects/org-name/databases/(default)"},"sendRawJson":true,"supportsCrossDomainXhr":true,"internalChannelParams":{"forwardChannelRequestTimeoutMs":600000},"forceLongPolling":true,"detectBufferingProxy":false,"useFetchStreams":true,"encodeInitMessageHeaders":true}
...
16:27:58.970 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:58.970Z]  @firebase/firestore: Firestore (10.11.0): MemoryPersistence Starting transaction: notifyLocalViewChanges
16:27:58.971 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:58.971Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Write' stream 0x709d58eb received: {"streamId":"0","streamToken":"GRBoQgKB9LW1"}
16:27:58.971 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:58.971Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Write' stream 0x709d58eb sending: {"streamToken":"GRBoQgKB9LW1","writes":[{"update":{"name":"projects/org-name/databases/(default)/documents/snippets/H7tCTg7yinGqP7vugHDS","fields":{...},"updateTransforms":[{"fieldPath":"updated_at","setToServerValue":"REQUEST_TIME"},{"fieldPath":"created_at","setToServerValue":"REQUEST_TIME"}],"currentDocument":{"exists":false}}]}
...
16:27:59.129 chunk-EVSHPO2K.js?v=9dd1e9cc:1057 [2024-09-04T15:27:59.129Z]  @firebase/firestore: Firestore (10.11.0): WebChannelConnection RPC 'Write' stream 0x709d58eb received error: {"code":409,"message":"Document already exists: projects/org-name/databases/(default)/documents/snippets/H7tCTg7yinGqP7vugHDS","status":"ALREADY_EXISTS"}

It seems the write ('Write' stream) is attempted again after the token has changed (RemoteStore received new credentials) because the token change closed the previous write stream (transport closed). It would be great if Firebase SDK could handle this problem in the SDK itself as we have multiple places in the code calling addDoc and it's a bit difficult to wrap each one in try-catch and handle the error manually. Please let me know if you need any other logs.

[dconeybe]

@GaurangTandon Thank you for the logs. That is indeed quite helpful. I noticed that you redacted some information from the logs, notably this log line which occurs 2 times: FirestoreClient Received user= USERID. Could you tell me if the redacted "USERID" values are the same? Or are they different? Thank you.

[GaurangTandon]

@dconeybe It's the same user ID in both cases.

[dconeybe]

@GaurangTandon Could you tell me what auth method you are using (anonymous, email/password, google auth)? Also, are you using App Check?

I'm asking because it's odd to me that the credentials are being refreshed in 2 seconds and I'm wondering if it's related to a specific auth mechanism or App Check.

[GaurangTandon]

This is using google auth. I don't think we use app check.

Note that when our app starts, we load several Firebase documents one after the other (in waterfall style), and we call getIdToken in a few places. I checked the cases where we set forceRefresh (its second parameter) to true. I found that at each time we see RemoteStore received new credentials in the debug log (three times), we made a corresponding getIdToken(user, true) call (in our code) around ~300-400ms before that. I guess that explains why the credentials are being refreshed 🙂

Now, we don't plan to set forceRefresh=false in these cases, because it fixes specific bugs we observed in production historically. Regardless of that, it would be helpful if Firebase SDK could handle that correctly.

[dconeybe]

@GaurangTandon Ahh that makes perfect sense that calling getIdToken() with forceRefresh=true would result in Firestore getting the updated credentials and rebuilding the "listen" and "write" streams, causing this issue. I agree that Firestore should handle this case correctly too. I'll look into it more and reply back once I have something concrete.

In the meantime, the only workaround I can think of is to simply catch and ignore the "document already exists" error and treat it like a success. Alternately, you could try regenerating the document ID and creating it again if you get the "document already exists" error.

[dconeybe]

@GaurangTandon I am able to reproduce this issue following your usage pattern with the code below:

import { collection, Firestore, addDoc } from 'firebase/firestore';
import { getAuth, signInWithEmailAndPassword, getIdToken } from 'firebase/auth';

export async function issue5549(db: Firestore): Promise<void> {
  const auth = getAuth(db.app);
  const user = await signInWithEmailAndPassword(auth, "example@google.com", "password");

  const collectionRef = collection(db, 'issue5549');
  const promises: Array<Promise<unknown>> = [];
  for (let i=0; i<30; i++) {
    const documentName = `0000${i+1}`.slice(-3);
    promises.push(addDoc(collectionRef, {name: documentName}));
  }

  for (const promise of promises) {
    getIdToken(user.user, /* forceRefresh= */true)
    await promise; // fails occasionally due to "document already exists" error.
  }
}

Here are the (redacted) logs: https://gist.github.com/dconeybe/d5ea0081828523302a34fbad117a4ba2

[dconeybe]

I talked to the team and we don't have a solution for the problem yet. I don't have an ETA either. I will keep you posted.

Another workaround we came up with would be to use a transaction (i.e. runTransaction(...)) to create the document. The downside of this approach would be that a transaction will fail if the device is offline.