mikeymicrophone / commissulator

Commissulator is a set of tools that help Real Estate Agents working with Citi Pads in Manhattan streamline their paperwork.
Other
3 stars 4 forks source link

CalendarEvents can be moved, deleted, and re-synced to Google, Microsoft, and Follow Up Boss #1

Open mikeymicrophone opened 5 years ago

mikeymicrophone commented 5 years ago

Commissulator has a CalendarEvent model that is a pipeline for syncing various agent calendars from Calendly and Follow Up Boss to Microsoft and Google, respectively. Microsoft and Google are then synced by a flow from Microsoft Flow, which has its own limitations.

The Follow Up Boss connection was built with Watir to drive a Chrome browser, interact with forms and autocompletes, and either scrape or submit events. However, an API endpoint was discovered that made this system obsolete for now. The app connects to that API, the Google Calendar API, and has a partially-working OAuth client for the Microsoft Office 365 API. That client is getting auth tokens and refresh tokens, but the access tokens haven't worked for me yet to view the calendar data.

I haven't yet implemented the Calendly API and don't know if or when we'll need to, since Calendly drops events in Outlook and I have title parsing in effect that makes them compatible with our client list in Follow Up Boss. We don't currently have client lists in Microsoft or Google. The Calendly API functions through webhooks, and seems fundamentally read-only, but it will communicate with Microsoft to prevent double-booking as long as we can keep Outlook in sync.

The goal is to get Follow Up Boss in sync with Office 365 and Google. When an event is created in one it should appear in the other (this currently works), and when an event is updated, the updates should be propogated across (this is not currently built but I have primary keys for each event from both APIs, Google and Follow Up Boss, that are currently implemented).

Note, Events in the Follow Up Boss API are a different model related to client touchpoints. The events we're adding to calendars are in their undocumented endpoint called Appointments. They also add Tasks to the calendar in a similar fashion. I received the signature for attributes on this endpoint by analyzing Ajax requests made by my browser while accessing and editing the calendar on their site, which I think is React.

To summarize what I think this job will take, you will start by reviewing my OAuth code for Microsoft's Graph API and getting that to deliver data instead of auth errors. Then implement a way for events to notice when they are changed or deleted, and callbacks to send those updates to the other two calendars.

The task will involve you setting up a staging account with Microsoft, one with Follow Up Boss (they give 14 day trials), and one with Google (you'll be able to specify a calendar by id for each agent in the system, those can all live on any Google account you like). You'll also use Heroku to test the cron jobs that keep everything in sync, although that might be just pushing to an instance that I already have running for QA.

mikeymicrophone commented 5 years ago

The credentials for API access are stored by environment. I could send you a key that would give access to one of the environments we have online. You can also make your own environments and we can put those credentials into the repo since they will be encrypted.

One of the perks of this task is you get to learn the technique that I developed for storing cookies and tokens in a way that's compatible with Heroku's ephemeral file system using Active Storage. If you like, use Active Cable to stream updates down to our CalendarEvents index. The app also uses Turbolinks so there are other ways to stream updates to the UI without using JS. These are all techniques that will hone your familiarity with Rails 5 libraries. We'll be adhering to Railsy conventions including using Ruby helpers and coffeescript if you wind up doing front end modifications as part of this task. Our stack and complement of plugins is a work in progress and any sensible addition would be entertained.

mikeymicrophone commented 5 years ago

Here is the code used to communicate with the Microsoft Graph API.
https://github.com/mikeymicrophone/commissulator/blob/master/app/models/agent.rb#L59-L181

Below it there's an alternate implementation using a different library, MS-maintained, that was found in other documentation.

mikeymicrophone commented 5 years ago

Here's the code pushing events to Follow Up Boss and Google.

https://github.com/mikeymicrophone/commissulator/blob/master/app/models/calendar_event.rb

mikeymicrophone commented 5 years ago

The wiki has information about running the environment. https://github.com/mikeymicrophone/commissulator/wiki/Required-Configurations

mikeymicrophone commented 5 years ago

This is the code for driving the Follow Up Boss interface with Watir is here. https://github.com/mikeymicrophone/commissulator/blob/master/app/webdrivers/fub_calendar_event.rb

Again, this approach probably won't be the easiest way to implement updates since we now have direct REST API access, but you can also turn off headless mode and watch these macros in operation, which could help clear up any confusion potentially. Running these systems requires a couple extra buildpacks for Heroku, which are documented here. https://github.com/mikeymicrophone/commissulator/wiki/Heroku-deployment-instructions

mikeymicrophone commented 5 years ago

Also, based on the number of web portals this team uses for things like researching available apartments and posting ads, I have a feeling that Watir webdrivers are going to be a major asset to us and any developer that wants to join the team. I'm pumped that I found a way to store both cookies and the passwords needed to potentially refresh them without needing to make the repo private, and I plan to leverage that tactic heavily.

mikeymicrophone commented 5 years ago

If you notice a security loophole in the configuration you can contact me directly or, if necessary, open another issue and we can evaluate the repair options. I'm aware that credentials have touched the repo in the past week; they've been replaced with new credentials and I don't expect them to grant anyone access to Microsoft or Google accounts.

mikeymicrophone commented 5 years ago

This is a Trello board where I took notes about my work on these calendars https://trello.com/b/PD0CspNw/calendars

mikeymicrophone commented 5 years ago

These are the Rake tasks that initiate the current sync protocol. https://github.com/mikeymicrophone/commissulator/blob/master/lib/tasks/calendars.rake

One pulls from Google, saves in Rails and then pushes to Follow Up Boss. The other pulls from Follow Up Boss, saves in the Rails db and then pushes to Google. (The third one gets events by scraping the Follow Up Boss calendar page, it's not currently in use).

They use the find_or_create pattern rather than querying by filtering parameters. These will have to be morphed into a flow that incorporates a push to Microsoft, since I don't think the MS Flow system is going to be adequate for syncing Google to Outlook. However, if we account for its limitations, it might be.

mikeymicrophone commented 5 years ago

The MS Flow will copy events from Outlook to Google, and it will copy events that didn't come from MS from Google back to other Outlook calendars. We could use the API to scrub the MS signature from the event description, if that enables the multidirectional syncing to complete. That might even be an alternative to implementing the balance of the Microsoft API.

The other limitation of MS Flow is that it doesn't replicate invitees to the event in either direction. This isn't actually a factor for Calendly-originated events since Calendly doesn't add invitees anyway (they just add the guest's name to the event title). And we're not currently using Outlook as an address book, so having the contact info centralized in Follow Up Boss is adequate.

This speaks to another requirement of the task, which is that these calendars should not be sending out email invites as they each progressively receive copies of these events. For instance, an event with Franklin is created in Follow Up Boss and then copied to Google and Microsoft. None of those three services should be sending Franklin an invitation.

mikeymicrophone commented 5 years ago

The events for these calendars are mainly originating either in Calendly or Follow Up Boss.

If they come from Calendly, they land in Outlook, and the flow pushes them to a Google calendar. Then the Rake task compares them to known Google events and pulls them into the Commissulator db, and pushes them to Follow Up Boss.

If they come from Follow Up Boss, they are pulled in by the Rake task and saved to Commissulator, then pushed into Google. Google rejects them if they don't have email addresses for every invitee. Then the flow pulls them from Google into the individual agents' Outlook calendars, as well as to a aggregate calendar for all agents on the team.

If they come from Calendly, they don't end up on the aggregate calendar, (presumably) because the description contains a little tag from Microsoft that disables the sync in order to prevent duplicates. Every time I've replaced the description, the event was then synced by the flow. If that can be leveraged without creating duplicates, it would be an alternative to the Microsoft Graph.

mikeymicrophone commented 5 years ago

Microsoft offers a 1-year free developer account that you can use to access Office as well as Azure. I've never tried these types of programming myself, but I'm trying to open my mind to what is obviously a powered portfolio.

mikeymicrophone commented 5 years ago

I'll drop in some of the best resources I've found so far about this Microsoft platform. There are a couple different APIs that can access their calendars.

mikeymicrophone commented 5 years ago

https://github.com/microsoftgraph/msgraph-sdk-ruby https://apps.dev.microsoft.com

mikeymicrophone commented 5 years ago

https://docs.microsoft.com/en-us/outlook/rest/ruby-tutorial

mikeymicrophone commented 5 years ago

This tool validated the access token I have. https://jwt.io

It's still unclear why using the token is resulting in OData::AuthenticationError (401 InvalidAuthenticationToken: "Access token validation failure." from "https://graph.microsoft.com/v1.0/me")

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done


This issue now has a funding of 0.25 ETH (51.12 USD @ $204.49/ETH) attached to it.

PixelantDesign commented 5 years ago

Hey @mikeymicrophone is this issue still active?

gitcoinbot commented 3 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done


Work has been started.

These users each claimed they can complete the work by 2 years, 6 months ago. Please review their action plans below:

1) thui70 has started work.

THANK YOU VEARY MAKE IT VEARY GOOD JOB I WANT ETH

Learn more on the Gitcoin Issue Details page.