Sync existing user email addresses when connecting Mailchimp Extension to Firebase

[dackers86]

Moving from https://github.com/firebase/extensions/issues/415

[papmodern]

Any update???

I could understand there is a PR related to this issue, but nothing mentioned in the main doc ad repo

[barticus]

Hi @dackers86 ,

Thanks for raising this.

This might be best delivered as a function that is deployed but manually triggered by the user to read through all existing users. We'll scope this out.

Regards,

[MasterScrat]

Any news on this? This issue really limits the ease of use of the extension, as either it needs to be setup from the start of the project's lifetime, or it requires writing a custom script that has large overlap with what the extension does 🤔

[barticus]

Hi @MasterScrat , are you expecting all data to be synced (e.g. merge fields, tags, activities)? I've been looking at how to get #52 merged but it is currently limited to just auth details (email + status). Also there are some testing difficulties. However, we could extend on the pattern there. The solution would need to iterate through the indicated firestore collection paths, probably with a user defined filter or at least looking for documents with the subscriberEmail indicated field set, and then treat it as an "insert" (no before document). We'd need to also test to handle high volumes and verify any mailchimp limits.

As a workaround, so you're not recreating any functionality in the extension:

1. Create a new collection, or subcollection if you were using a subcollection
2. Update the extension to point to this new collection
3. Write a script to copy between the two collections. List all documents in the existing collection and then create a document in the new collection. It might be something like:

   import { doc, setDoc, collection, getDocs } from "firebase/firestore"; 
   const allDocs = await getDocs(collection(db, "EXISTING_COLLECTION_PATH"));
   allDocs.forEach(async (d) => {
   await setDoc(doc(db, "NEW_COLLECTION_PATH", d.id), d.data());
   });

4. Run the script
5. Update the extension to point back to the existing collection

6. Write a script to clean up the temporary collection

   import { doc, deleteDoc } from "firebase/firestore"; 
   
   const allDocs = await getDocs(collection(db, "NEW_COLLECTION_PATH"));
   allDocs.forEach(async (d) => {
   await deleteDoc(doc(db, "NEW_COLLECTION_PATH", d.id));
   });

We will look towards supporting natively syncing on install/update but as I cannot commit to a timeline the above might be help someone in the meantime.

[mklsk]

@barticus thanks for posting the workaround! Seems a bit clunky at the moment, to be entirely fair 🫠

Love your idea here:

This might be best delivered as a function that is deployed but manually triggered by the user to read through all existing users.

Would be great to see it come through!

[BilalAtique]

Hi @MasterScrat , are you expecting all data to be synced (e.g. merge fields, tags, activities)? I've been looking at how to get #52 merged but it is currently limited to just auth details (email + status). Also there are some testing difficulties. However, we could extend on the pattern there. The solution would need to iterate through the indicated firestore collection paths, probably with a user defined filter or at least looking for documents with the subscriberEmail indicated field set, and then treat it as an "insert" (no before document). We'd need to also test to handle high volumes and verify any mailchimp limits.

As a workaround, so you're not recreating any functionality in the extension:

1. Create a new collection, or subcollection if you were using a subcollection
2. Update the extension to point to this new collection
3. Write a script to copy between the two collections. List all documents in the existing collection and then create a document in the new collection. It might be something like:

import { doc, setDoc, collection, getDocs } from "firebase/firestore"; 
const allDocs = await getDocs(collection(db, "EXISTING_COLLECTION_PATH"));
allDocs.forEach(async (d) => {
  await setDoc(doc(db, "NEW_COLLECTION_PATH", d.id), d.data());
});

4. Run the script
5. Update the extension to point back to the existing collection
6. Write a script to clean up the temporary collection

import { doc, deleteDoc } from "firebase/firestore"; 

const allDocs = await getDocs(collection(db, "NEW_COLLECTION_PATH"));
allDocs.forEach(async (d) => {
  await deleteDoc(doc(db, "NEW_COLLECTION_PATH", d.id));
});

We will look towards supporting natively syncing on install/update but as I cannot commit to a timeline the above might be help someone in the meantime.

Considering this approach. I wrote a function and run it locally:

import { onRequest } from "firebase-functions/v2/https";
import * as admin from "firebase-admin";

if (!admin.apps.length) admin.initializeApp();

const migrateData = async () => {
  const db = admin.firestore();
  const existingCollection = db.collection("users");
  const newCollection = db.collection("temp-users");

  const allDocs = (await existingCollection.get()).docs;

  const result = await Promise.all(
    allDocs.map(async (doc) => {
      const docData = doc.data();
      try {
        await newCollection.doc(doc.id).set(docData);
        const successStr = `${docData.username} is migrated successfully`;
        console.log(successStr);
        return successStr;
      } catch (error) {
        return `${docData.username} migration failed`;
      }
    })
  );
  return result;
};

const copyExistingUsersToTempCollection = onRequest(async (req, res) => {
  try {
    const result = await migrateData();
    res.send(result);
  } catch (error) {
    console.log(error);
    res.send(error);
  }
});

export default copyExistingUsersToTempCollection;

Although it saves time, it misses tags and merge fields for many contacts. So, I have updated them manually.