papercups-io / papercups

Open-source live customer chat
https://app.papercups.io/demo
MIT License
5.76k stars 520 forks source link

Gmail integration v1 #637

Closed reichert621 closed 3 years ago

reichert621 commented 3 years ago

Problem Users want to sync their support@ email accounts with Papercups.

Why?

Solution TODO: spec this out some more!

Some thoughts:

Describe alternatives you've considered Need to investigate if Mailgun makes it possible to do something like this more easily 🤔 hard to imagine that would be the case though?

Additional context TBD

Testing This will probably require a fair bit of manual testing.

(TODO: list out some QA test case scenarios)

Questions, or need help getting started? Feel free to ask below, or ping us on Slack :)

(You can also check out our CONTRIBUTING.md)

a8t commented 3 years ago

Interesting. Here's some thoughts.

Poll vs notification

Gmail seems to have a Push Notification system that could be used instead of polling. The push notification POST body includes a base64-encoded string that parses to:

{"emailAddress": "user@example.com", "historyId": "9876543210"}

where historyId refers to a new History object, which represents an atomic change to the user's inbox.

What is a "new email"

History has this shape (per this link):

{
  "id": string,
  "messages": [
    {
      object (Message)
    }
  ],
  "messagesAdded": [
    {
      object (MessageAdded)
    }
  ],
  "messagesDeleted": [
    {
      object (MessageDeleted)
    }
  ],
  "labelsAdded": [
    {
      object (LabelAdded)
    }
  ],
  "labelsRemoved": [
    {
      object (LabelRemoved)
    }
  ]
}

So I think it's straightforward to get the new Message (from History.messagesAdded).

Parsing

A Message looks like this:

{
  "id": string,
  "threadId": string,
  "labelIds": [
    string
  ],
  "snippet": string,
  "historyId": string,
  "internalDate": string,
  "payload": {
    object (MessagePart)
  },
  "sizeEstimate": integer,
  "raw": string
}

We'd have to look further into what MessagePart looks like.

Replying to a thread

If you're trying to send a reply and want the email to thread, make sure that:

  • The Subject headers match
  • The References and In-Reply-To headers follow the RFC 2822 standard.

https://developers.google.com/gmail/api/guides/sending

reichert621 commented 3 years ago

thanks @a8t! super helpful :)

also going to paste in some sample response payloads from the Gmail API here...

/gmail/v1/users/me/messages

%{
  "messages" => [
    %{"id" => "1783bb273d3ca8ed", "threadId" => "1783bb273d3ca8ed"},
    %{"id" => "1783b6533a33f0f1", "threadId" => "1783b6533a33f0f1"},
    %{"id" => "1783b5705b99bde2", "threadId" => "1783b5705b99bde2"},
    %{"id" => "1783b56c46b4eef6", "threadId" => "1783b56c46b4eef6"},
    %{"id" => "1783b45f10c856ad", "threadId" => "1782884988975d0e"},
    # etc.  
  ]
}

/gmail/v1/users/me/messages/#{message_id}

%{
  "historyId" => "367404",
  "id" => "1783bdc5ee6857a2",
  "internalDate" => "1615912000000",
  "labelIds" => ["SENT"],
  "payload" => %{
    "body" => %{"size" => 0},
    "filename" => "",
    "headers" => [
      %{"name" => "MIME-Version", "value" => "1.0"},
      %{"name" => "Date", "value" => "Tue, 16 Mar 2021 12:26:40 -0400"},
      %{
        "name" => "References",
        "value" => "<CADrf=fmqwxG84EvjcE8rm-asKaQM-hZSm=kYizrvOX1kavyGtg@mail.gmail.com>"
      },
      %{
        "name" => "In-Reply-To",
        "value" => "<CADrf=fmqwxG84EvjcE8rm-asKaQM-hZSm=kYizrvOX1kavyGtg@mail.gmail.com>"
      },
      %{
        "name" => "Message-ID",
        "value" => "<CAO-SVC=6y8jKsTpVy=783mroF7v+qfX+DvWwL9SA9xL7iAa4Tw@mail.gmail.com>"
      },
      %{"name" => "Subject", "value" => "Re: Test subject for Gmail API"},
      %{"name" => "From", "value" => "Alex Reichert <alex@papercups.io>"},
      %{"name" => "To", "value" => "Alex Reichert <reichertjalex@gmail.com>"},
      %{
        "name" => "Content-Type",
        "value" => "multipart/alternative; boundary=\"000000000000d4bab205bda9d4dd\""
      }
    ],
    "mimeType" => "multipart/alternative",
    "partId" => "",
    "parts" => [
      %{
        "body" => %{
          "data" => "TmljZSEgVGhpcyBpcyBhIHRlc3QgcmVwbHkuDQoNCkxldCdzIHNlZSBob3cgaXQgZ29lcy4NCuGQpw0KDQpPbiBUdWUsIE1hciAxNiwgMjAyMSBhdCAxMjoyNiBQTSBBbGV4IFJlaWNoZXJ0IDxyZWljaGVydGphbGV4QGdtYWlsLmNvbT4NCndyb3RlOg0KDQo-IEhleSBBbGV4IQ0KPg0KPiBUaGlzIGlzIGEgdGVzdCBtZXNzYWdlIGZvciB0aGUgR21haWwgQVBJLg0KPg0K",
          "size" => 210
        },
        "filename" => "",
        "headers" => [
          %{
            "name" => "Content-Type",
            "value" => "text/plain; charset=\"UTF-8\""
          },
          %{
            "name" => "Content-Transfer-Encoding",
            "value" => "quoted-printable"
          }
        ],
        "mimeType" => "text/plain",
        "partId" => "0"
      },
      %{
        "body" => %{
          "data" => "PGRpdiBkaXI9Imx0ciI-TmljZSEgVGhpcyBpcyBhIHRlc3QgcmVwbHkuPGRpdj48YnI-PC9kaXY-PGRpdj5MZXQmIzM5O3Mgc2VlIGhvdyBpdCBnb2VzLjwvZGl2PjwvZGl2PjxkaXYgaHNwYWNlPSJzdHJlYWstcHQtbWFyayIgc3R5bGU9Im1heC1oZWlnaHQ6MXB4Ij48aW1nIGFsdD0iIiBzdHlsZT0id2lkdGg6MHB4O21heC1oZWlnaHQ6MHB4O292ZXJmbG93OmhpZGRlbiIgc3JjPSJodHRwczovL21haWxmb29nYWUuYXBwc3BvdC5jb20vdD9zZW5kZXI9YVlXeGxlRUJ3WVhCbGNtTjFjSE11YVc4JTNEJmFtcDt0eXBlPXplcm9jb250ZW50JmFtcDtndWlkPTEwN2JkMGRkLTdmODYtNGIzZS04MDM4LTdhZDMyNWUwMmYxZCI-PGZvbnQgY29sb3I9IiNmZmZmZmYiIHNpemU9IjEiPuGQpzwvZm9udD48L2Rpdj48YnI-PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9hdHRyIj5PbiBUdWUsIE1hciAxNiwgMjAyMSBhdCAxMjoyNiBQTSBBbGV4IFJlaWNoZXJ0ICZsdDs8YSBocmVmPSJtYWlsdG86cmVpY2hlcnRqYWxleEBnbWFpbC5jb20iPnJlaWNoZXJ0amFsZXhAZ21haWwuY29tPC9hPiZndDsgd3JvdGU6PGJyPjwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj5IZXkgQWxleCE8ZGl2Pjxicj48L2Rpdj48ZGl2PlRoaXMgaXMgYSB0ZXN0IG1lc3NhZ2UgZm9yIHRoZSBHbWFpbCBBUEkuPC9kaXY-PC9kaXY-DQo8L2Jsb2NrcXVvdGU-PC9kaXY-DQo=",
          "size" => 845
        },
        "filename" => "",
        "headers" => [
          %{"name" => "Content-Type", "value" => "text/html; charset=\"UTF-8\""},
          %{
            "name" => "Content-Transfer-Encoding",
            "value" => "quoted-printable"
          }
        ],
        "mimeType" => "text/html",
        "partId" => "1"
      }
    ]
  },
  "sizeEstimate" => 1927,
  "snippet" => "Nice! This is a test reply. Let&#39;s see how it goes. ᐧ On Tue, Mar 16, 2021 at 12:26 PM Alex Reichert &lt;reichertjalex@gmail.com&gt; wrote: Hey Alex! This is a test message for the Gmail API.",
  "threadId" => "1783bdc057a1f4c2"
}

/gmail/v1/users/me/threads

%{
  "nextPageToken" => "00264963116145399493",
  "resultSizeEstimate" => 250,
  "threads" => [
    %{
      "historyId" => "367407",
      "id" => "1783bdf49b1f474d",
      "snippet" => "[Webinar] Learn about global data protection and deliverability | View online version Hey there, A lot has changed since GDPR came into effect in May 2018. Have you been keeping up with all the updates"
    },
    %{
      "historyId" => "367404",
      "id" => "1783bdc057a1f4c2",
      "snippet" => "Nice! This is a test reply. Let&#39;s see how it goes. ᐧ On Tue, Mar 16, 2021 at 12:26 PM Alex Reichert &lt;reichertjalex@gmail.com&gt; wrote: Hey Alex! This is a test message for the Gmail API."
    },
    %{
      "historyId" => "367255",
      "id" => "1783bb273d3ca8ed",
      "snippet" => "Cruise to buy rival driverless car startup; ElevateBio raises $525M in VC; Sprinklr confidentially files for an IPO; Gatsby gets green light for $10M Read online | Don&#39;t want to receive these"
    },
    # etc.
  ]
}

/gmail/v1/users/me/threads/#{thread_id}?format=full

%{
  "historyId" => "367404",
  "id" => "1783bdc057a1f4c2",
  "messages" => [
    %{
      "historyId" => "367378",
      "id" => "1783bdc057a1f4c2",
      "internalDate" => "1615911941000",
      "labelIds" => ["IMPORTANT", "CATEGORY_PERSONAL", "INBOX"],
      "payload" => %{
        "body" => %{"size" => 0},
        "filename" => "",
        "headers" => [
          %{"name" => "Delivered-To", "value" => "alex@papercups.io"},
          %{
            "name" => "Received",
            "value" => "by 2002:a2e:9694:0:0:0:0:0 with SMTP id q20csp4647726lji;        Tue, 16 Mar 2021 09:26:18 -0700 (PDT)"
          },
          %{
            "name" => "X-Received",
            "value" => "by 2002:a92:540d:: with SMTP id i13mr4411823ilb.8.1615911978285;        Tue, 16 Mar 2021 09:26:18 -0700 (PDT)"
          },
          %{
            "name" => "ARC-Seal",
            "value" => "i=1; a=rsa-sha256; t=1615911978; cv=none;        d=google.com; s=arc-20160816;        b=Qr3NovZVSSOKvZFihmnEZ93EdGgNJAl+S5Rbb2KvOQPKCp5D55+RDZPP59vrIg2GxE         aWoPW+lmYxKvldQK3cZqjeJYe/2BY6WNLLGFgr1LMx8E+4V888sVZJ2e0gJvnM4pOK52         5NJBqpMS55UbB/aJAIRYs0cQEca7oTFg8Cy9GzHkOFqHsR7rPLVmzh9co0YdEOFJcF8l         wezz5in3yEtXgOCC2C5xfHIfDfkAj127+bxeipf5H9murFrEn3Ee87jS1ET4JD5XtzAk         7iWPmBoXOqcFasT2s+77wu3TF/cPMKSh0NXK3Ibqtt8osWBheZDtbVxQQHa1UnXlEUoQ         yxfA=="
          },
          %{
            "name" => "ARC-Message-Signature",
            "value" => "i=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=arc-20160816;        h=to:subject:message-id:date:from:mime-version:dkim-signature;        bh=Utx/9TlOmY4X3JEroB0TzviHX6lVHUqAFBQ7/R2R8gY=;        b=NW4WLSuqYPUXePK27OEjYN9/+oo7HhChx2ksGdIugMjYBjK1HWMLa+pObBoltCk54n         +ZkAvEpw7oJkRyG7sd33ok4IyoANQz5lw30+mWG4KayI3W0V1tZ8IZiQd8+Pxa36oxvs         kw2W9Jn4Ck5slbgWEtpfTEU81Zp3cIAon/sKk39tcWR1gceoB7dg3DRXTUYmP0F2nw1Z         /Rx4SXjVBQ+8F5wHnpH7LTm1XzfjcHDPCeulk2+ao7sdlmighY2VaQH95lV+Uu0ASc5Q         EbyufEWLFUoGOtwZamMudcIL525qm49pk7DVzcNsnJ+/i2dF9i4YdIoU3NYvhDeDoPPU         yyhA=="
          },
          %{
            "name" => "ARC-Authentication-Results",
            "value" => "i=1; mx.google.com;       dkim=pass header.i=@gmail.com header.s=20161025 header.b=gpygfwCF;       spf=pass (google.com: domain of reichertjalex@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=reichertjalex@gmail.com;       dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com"
          },
          %{"name" => "Return-Path", "value" => "<reichertjalex@gmail.com>"},
          %{
            "name" => "Received",
            "value" => "from mail-sor-f41.google.com (mail-sor-f41.google.com. [209.85.220.41])        by mx.google.com with SMTPS id p79sor8092478iod.73.2021.03.16.09.26.18        for <alex@papercups.io>        (Google Transport Security);        Tue, 16 Mar 2021 09:26:18 -0700 (PDT)"
          },
          %{
            "name" => "Received-SPF",
            "value" => "pass (google.com: domain of reichertjalex@gmail.com designates 209.85.220.41 as permitted sender) client-ip=209.85.220.41;"
          },
          %{
            "name" => "Authentication-Results",
            "value" => "mx.google.com;       dkim=pass header.i=@gmail.com header.s=20161025 header.b=gpygfwCF;       spf=pass (google.com: domain of reichertjalex@gmail.com designates 209.85.220.41 as permitted sender) smtp.mailfrom=reichertjalex@gmail.com;       dmarc=pass (p=NONE sp=QUARANTINE dis=NONE) header.from=gmail.com"
          },
          %{
            "name" => "DKIM-Signature",
            "value" => "v=1; a=rsa-sha256; c=relaxed/relaxed;        d=gmail.com; s=20161025;        h=mime-version:from:date:message-id:subject:to;        bh=Utx/9TlOmY4X3JEroB0TzviHX6lVHUqAFBQ7/R2R8gY=;        b=gpygfwCFJoVj4sS985jmGwMFJ+DafVa4QFu13DuBWWajhdV9Dp2tZ+H5Nmb7mljtuf         gjCANID4oAOVj9LxrWTTN1ieE9wZHRyCPgqll+68BAXyJ2TMudur90pz0e+hIaxuKsf1         QITbrm04eFnyZ50WTfGKeHtK9nbgxV9MOJUiMHO5nBNBdOPo3ESLwx1SZo/heJgD0PBf         z8eye+Sm8PYadgwJ229HPjjs5suqV0LBZPl3fpY4uaPioEtudl2gbWYd3NyWPAIjirny         +c9/xUzQJ1EWiHN7+G/pQprTSPJjA60xirJuAOhNC2dcvtozyE9caCUnYRIhSUupCpW8         naLA=="
          },
          %{
            "name" => "X-Google-DKIM-Signature",
            "value" => "v=1; a=rsa-sha256; c=relaxed/relaxed;        d=1e100.net; s=20161025;        h=x-gm-message-state:mime-version:from:date:message-id:subject:to;        bh=Utx/9TlOmY4X3JEroB0TzviHX6lVHUqAFBQ7/R2R8gY=;        b=Cm/H4TDPDfjZcJtbJZXHQcM8+tQWr+GJPKK9CEO/teZ+QKqs0I56/tUAGWj0MG4gEy         YrpHY/VYR0e/1+9IMjcenNov3z9dqYwuX6DckT2nfoANgrtQWwawNQJETMLmb8ks7vkY         3K66/0jSATmKnDJ60Zk3ilSeiqbQIVQFG8NTMvLjOCLyb1xKW2F9ptn9IlkIKWFN1j3C         ol9WFvyZRv5mRkhhb2XNqumvEOJpvKhohLNnV7N/aw/vK13EFafIQyRdRyVgy12R3+tW         j1z1s3IJiRRPjUBQNd+l/cPgm9UOSpvtAFToZE7RV+dOt1F5yZo8G/+g1s+99L7cCL7e         cw2Q=="
          },
          %{
            "name" => "X-Gm-Message-State",
            "value" => "AOAM530BdJj/E5PqZOZ8x0/TEiySEY2EnvUQIQdJq6gynwWFiQ0se/IN iEhSsVTu+Q5RiTd7+/EE6x4gJQX1DVkpnK1yQOYrTGsgqRM="
          },
          %{
            "name" => "X-Google-Smtp-Source",
            "value" => "ABdhPJz1jztn29kswW7Uxfe+7KS0Uapms9nz0R0QAKLlGnyv42TYnsXi9vMQfNCRrGzTNPubpXBI5vV4qbum7v/6Cb4="
          },
          %{
            "name" => "X-Received",
            "value" => "by 2002:a6b:fd07:: with SMTP id c7mr3985617ioi.198.1615911977550; Tue, 16 Mar 2021 09:26:17 -0700 (PDT)"
          },
          %{"name" => "MIME-Version", "value" => "1.0"},
          %{
            "name" => "From",
            "value" => "Alex Reichert <reichertjalex@gmail.com>"
          },
          %{"name" => "Date", "value" => "Tue, 16 Mar 2021 12:25:41 -0400"},
          %{
            "name" => "Message-ID",
            "value" => "<CADrf=fmqwxG84EvjcE8rm-asKaQM-hZSm=kYizrvOX1kavyGtg@mail.gmail.com>"
          },
          %{"name" => "Subject", "value" => "Test subject for Gmail API"},
          %{"name" => "To", "value" => "Alex Reichert <alex@papercups.io>"},
          %{
            "name" => "Content-Type",
            "value" => "multipart/alternative; boundary=\"0000000000006906a505bda9d373\""
          }
        ],
        "mimeType" => "multipart/alternative",
        "partId" => "",
        "parts" => [
          %{
            "body" => %{
              "data" => "SGV5IEFsZXghDQoNClRoaXMgaXMgYSB0ZXN0IG1lc3NhZ2UgZm9yIHRoZSBHbWFpbCBBUEkuDQo=",
              "size" => 56
            },
            "filename" => "",
            "headers" => [
              %{
                "name" => "Content-Type",
                "value" => "text/plain; charset=\"UTF-8\""
              }
            ],
            "mimeType" => "text/plain",
            "partId" => "0"
          },
          %{
            "body" => %{
              "data" => "PGRpdiBkaXI9Imx0ciI-SGV5IEFsZXghPGRpdj48YnI-PC9kaXY-PGRpdj5UaGlzIGlzIGEgdGVzdCBtZXNzYWdlIGZvciB0aGUgR21haWwgQVBJLjwvZGl2PjwvZGl2Pg0K",
              "size" => 99
            },
            "filename" => "",
            "headers" => [
              %{
                "name" => "Content-Type",
                "value" => "text/html; charset=\"UTF-8\""
              }
            ],
            "mimeType" => "text/html",
            "partId" => "1"
          }
        ]
      },
      "sizeEstimate" => 4863,
      "snippet" => "Hey Alex! This is a test message for the Gmail API.",
      "threadId" => "1783bdc057a1f4c2"
    },
    %{
      "historyId" => "367404",
      "id" => "1783bdc5ee6857a2",
      "internalDate" => "1615912000000",
      "labelIds" => ["SENT"],
      "payload" => %{
        "body" => %{"size" => 0},
        "filename" => "",
        "headers" => [
          %{"name" => "MIME-Version", "value" => "1.0"},
          %{"name" => "Date", "value" => "Tue, 16 Mar 2021 12:26:40 -0400"},
          %{
            "name" => "References",
            "value" => "<CADrf=fmqwxG84EvjcE8rm-asKaQM-hZSm=kYizrvOX1kavyGtg@mail.gmail.com>"
          },
          %{
            "name" => "In-Reply-To",
            "value" => "<CADrf=fmqwxG84EvjcE8rm-asKaQM-hZSm=kYizrvOX1kavyGtg@mail.gmail.com>"
          },
          %{
            "name" => "Message-ID",
            "value" => "<CAO-SVC=6y8jKsTpVy=783mroF7v+qfX+DvWwL9SA9xL7iAa4Tw@mail.gmail.com>"
          },
          %{"name" => "Subject", "value" => "Re: Test subject for Gmail API"},
          %{"name" => "From", "value" => "Alex Reichert <alex@papercups.io>"},
          %{
            "name" => "To",
            "value" => "Alex Reichert <reichertjalex@gmail.com>"
          },
          %{
            "name" => "Content-Type",
            "value" => "multipart/alternative; boundary=\"000000000000d4bab205bda9d4dd\""
          }
        ],
        "mimeType" => "multipart/alternative",
        "partId" => "",
        "parts" => [
          %{
            "body" => %{
              "data" => "TmljZSEgVGhpcyBpcyBhIHRlc3QgcmVwbHkuDQoNCkxldCdzIHNlZSBob3cgaXQgZ29lcy4NCuGQpw0KDQpPbiBUdWUsIE1hciAxNiwgMjAyMSBhdCAxMjoyNiBQTSBBbGV4IFJlaWNoZXJ0IDxyZWljaGVydGphbGV4QGdtYWlsLmNvbT4NCndyb3RlOg0KDQo-IEhleSBBbGV4IQ0KPg0KPiBUaGlzIGlzIGEgdGVzdCBtZXNzYWdlIGZvciB0aGUgR21haWwgQVBJLg0KPg0K",
              "size" => 210
            },
            "filename" => "",
            "headers" => [
              %{
                "name" => "Content-Type",
                "value" => "text/plain; charset=\"UTF-8\""
              },
              %{
                "name" => "Content-Transfer-Encoding",
                "value" => "quoted-printable"
              }
            ],
            "mimeType" => "text/plain",
            "partId" => "0"
          },
          %{
            "body" => %{
              "data" => "PGRpdiBkaXI9Imx0ciI-TmljZSEgVGhpcyBpcyBhIHRlc3QgcmVwbHkuPGRpdj48YnI-PC9kaXY-PGRpdj5MZXQmIzM5O3Mgc2VlIGhvdyBpdCBnb2VzLjwvZGl2PjwvZGl2PjxkaXYgaHNwYWNlPSJzdHJlYWstcHQtbWFyayIgc3R5bGU9Im1heC1oZWlnaHQ6MXB4Ij48aW1nIGFsdD0iIiBzdHlsZT0id2lkdGg6MHB4O21heC1oZWlnaHQ6MHB4O292ZXJmbG93OmhpZGRlbiIgc3JjPSJodHRwczovL21haWxmb29nYWUuYXBwc3BvdC5jb20vdD9zZW5kZXI9YVlXeGxlRUJ3WVhCbGNtTjFjSE11YVc4JTNEJmFtcDt0eXBlPXplcm9jb250ZW50JmFtcDtndWlkPTEwN2JkMGRkLTdmODYtNGIzZS04MDM4LTdhZDMyNWUwMmYxZCI-PGZvbnQgY29sb3I9IiNmZmZmZmYiIHNpemU9IjEiPuGQpzwvZm9udD48L2Rpdj48YnI-PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9hdHRyIj5PbiBUdWUsIE1hciAxNiwgMjAyMSBhdCAxMjoyNiBQTSBBbGV4IFJlaWNoZXJ0ICZsdDs8YSBocmVmPSJtYWlsdG86cmVpY2hlcnRqYWxleEBnbWFpbC5jb20iPnJlaWNoZXJ0amFsZXhAZ21haWwuY29tPC9hPiZndDsgd3JvdGU6PGJyPjwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj5IZXkgQWxleCE8ZGl2Pjxicj48L2Rpdj48ZGl2PlRoaXMgaXMgYSB0ZXN0IG1lc3NhZ2UgZm9yIHRoZSBHbWFpbCBBUEkuPC9kaXY-PC9kaXY-DQo8L2Jsb2NrcXVvdGU-PC9kaXY-DQo=",
              "size" => 845
            },
            "filename" => "",
            "headers" => [
              %{
                "name" => "Content-Type",
                "value" => "text/html; charset=\"UTF-8\""
              },
              %{
                "name" => "Content-Transfer-Encoding",
                "value" => "quoted-printable"
              }
            ],
            "mimeType" => "text/html",
            "partId" => "1"
          }
        ]
      },
      "sizeEstimate" => 1927,
      "snippet" => "Nice! This is a test reply. Let&#39;s see how it goes. ᐧ On Tue, Mar 16, 2021 at 12:26 PM Alex Reichert &lt;reichertjalex@gmail.com&gt; wrote: Hey Alex! This is a test message for the Gmail API.",
      "threadId" => "1783bdc057a1f4c2"
    }
  ]
}

Decoded message body

Encoded: "PGRpdiBkaXI9Imx0ciI-TmljZSEgVGhpcyBpcyBhIHRlc3QgcmVwbHkuPGRpdj48YnI-PC9kaXY-PGRpdj5MZXQmIzM5O3Mgc2VlIGhvdyBpdCBnb2VzLjwvZGl2PjwvZGl2PjxkaXYgaHNwYWNlPSJzdHJlYWstcHQtbWFyayIgc3R5bGU9Im1heC1oZWlnaHQ6MXB4Ij48aW1nIGFsdD0iIiBzdHlsZT0id2lkdGg6MHB4O21heC1oZWlnaHQ6MHB4O292ZXJmbG93OmhpZGRlbiIgc3JjPSJodHRwczovL21haWxmb29nYWUuYXBwc3BvdC5jb20vdD9zZW5kZXI9YVlXeGxlRUJ3WVhCbGNtTjFjSE11YVc4JTNEJmFtcDt0eXBlPXplcm9jb250ZW50JmFtcDtndWlkPTEwN2JkMGRkLTdmODYtNGIzZS04MDM4LTdhZDMyNWUwMmYxZCI-PGZvbnQgY29sb3I9IiNmZmZmZmYiIHNpemU9IjEiPuGQpzwvZm9udD48L2Rpdj48YnI-PGRpdiBjbGFzcz0iZ21haWxfcXVvdGUiPjxkaXYgZGlyPSJsdHIiIGNsYXNzPSJnbWFpbF9hdHRyIj5PbiBUdWUsIE1hciAxNiwgMjAyMSBhdCAxMjoyNiBQTSBBbGV4IFJlaWNoZXJ0ICZsdDs8YSBocmVmPSJtYWlsdG86cmVpY2hlcnRqYWxleEBnbWFpbC5jb20iPnJlaWNoZXJ0amFsZXhAZ21haWwuY29tPC9hPiZndDsgd3JvdGU6PGJyPjwvZGl2PjxibG9ja3F1b3RlIGNsYXNzPSJnbWFpbF9xdW90ZSIgc3R5bGU9Im1hcmdpbjowcHggMHB4IDBweCAwLjhleDtib3JkZXItbGVmdDoxcHggc29saWQgcmdiKDIwNCwyMDQsMjA0KTtwYWRkaW5nLWxlZnQ6MWV4Ij48ZGl2IGRpcj0ibHRyIj5IZXkgQWxleCE8ZGl2Pjxicj48L2Rpdj48ZGl2PlRoaXMgaXMgYSB0ZXN0IG1lc3NhZ2UgZm9yIHRoZSBHbWFpbCBBUEkuPC9kaXY-PC9kaXY-DQo8L2Jsb2NrcXVvdGU-PC9kaXY-DQo="

Decoded: "<div dir=\"ltr\">Nice! This is a test reply.<div><br></div><div>Let&#39;s see how it goes.</div></div><div hspace=\"streak-pt-mark\" style=\"max-height:1px\"><img alt=\"\" style=\"width:0px;max-height:0px;overflow:hidden\" src=\"https://mailfoogae.appspot.com/t?sender=aYWxleEBwYXBlcmN1cHMuaW8%3D&amp;type=zerocontent&amp;guid=107bd0dd-7f86-4b3e-8038-7ad325e02f1d\"><font color=\"#ffffff\" size=\"1\">ᐧ</font></div><br><div class=\"gmail_quote\"><div dir=\"ltr\" class=\"gmail_attr\">On Tue, Mar 16, 2021 at 12:26 PM Alex Reichert &lt;<a href=\"mailto:reichertjalex@gmail.com\">reichertjalex@gmail.com</a>&gt; wrote:<br></div><blockquote class=\"gmail_quote\" style=\"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex\"><div dir=\"ltr\">Hey Alex!<div><br></div><div>This is a test message for the Gmail API.</div></div>\r\n</blockquote></div>\r\n"

wmnnd commented 3 years ago

Here are some gerneral considerations for evaluating whether using IMAP would be a viable alternative to the Gmail API:

What is IMAP

IMAP is the standard for retrieving emails from a server and defined in RFC 3501. IMAP is supported by all email providersš.

IMAP and Gmail

Determining "new emails"

IMAP has several ways we could use to determine "new" emails

Push/pull

IMAP allows a client to remain connected and immediately receive new messages. This would make an implementation much more complex, however. So I don’t think this would make sense for an MVP.

Threading

There are at least three ways in which threading could be implemented at the IMAP client level:

Available Elixir/Erlang libraries

There are several Elixir and Erlang IMAP client libraries. None of them show any recent activity. However, since the scope of IMAP is limited, it is not to be expected that a reasonably well-functioning implementation would change frequently:

Alternatively, if we end up focusing on polling emails without long-lasting open connection, simply using Hackney et al might be a viable option.

Conclusion

Using IMAP seems like a viable alternative to Gmail’s API. We would need to evaluate whether eximap can work well within Papercups or whether we should build our own small wrapper for the IMAP protocol.

Building an IMAP integration will be a bit more work than a Gmail API integration but I don’t expect it to be significantly more difficult.


š Notable exceptions are hey.com and tutanota.com but they focus on particular aspects of email and are not suited for customer-support emails anyways.

² There is another attribute, UIDVALIDITY which we should store alongside UIDNEXT. If UIDVALIDITY has changed, then the mailbox [has been] deleted and a new mailbox with the same name [has been] created at a later date - but this is probably an edge case.

reichert621 commented 3 years ago

@wmnnd I'm thinking for rough next steps...

(we can worry about how we map these to resources in papercups later)

reichert621 commented 3 years ago

@wmnnd here are some quick thoughts, hopefully this is a bit more concrete! (I'll try to keep adding more details, but feel free to ask any questions or clarifications)

To start off we're just going to poll for new messages (every 5-10 mins?) rather than rely on a "push" notification from a webhook or something like that. This feature will only be used by <10 beta users to start, so let's not worry too much about scalability issues quite yet.

At a high level we want to:

The main operations we'll need to accomplish this:

See pseudocode below for a bit more detail...

Client.get_events_since(authorization, history_id)

The history_id here simply represents a unique ID of an event. This method should return a list of "events" since that ID.

Ideally this will give us a clear "diff" (i.e. messages added/removed) since the event ID passed in. (Similar to https://developers.google.com/gmail/api/reference/rest/v1/users.history/list)

This method should be run whenever we do a "sync" with an email account (since we'll start off by just polling for updates). We'll check for "messages added" to determine which conversations need to be created/updated in Papercups.

Client.get_thread(authorization, thread_id)

This method should retrieve the entire thread, along with its messages. The thread details should include a list of messages.

Each message should include:

Client.get_message(authorization, message_id)

This method will retrieve a single message by ID. The shape of the message will be the same as above.

Client.list_threads(authorization)

This method handles retrieving the list of email threads in an inbox. Each thread should have an ID, which can be passed in below to retrieve all the details.

This method should ideally retrieve the list sorted by most recent -> oldest threads.

Client.send_message(authorization, payload)

This method should handle sending messages as the authenticated email account. If the appropriate fields are passed into the message payload (e.g. "In-Reply-To" header, etc), then the message should be sent as a reply to its existing thread.