Use new APIs for exports

camphillips22 commented 3 years ago

This is significant update which migrates the existing export client from the existing APIs to the new (as of yet, undocumented) "Segment Export" APIs.

The aim is to be as backwards compatible as possible, but there are few notable changes from the existing workflow:

Exports are not "bundled"

With the existing APIs, the exports were "bundled" into a time period that was defined in the "Data Export" settings for in the FullStory application. The new APIs do not have this restriction. Instead, the new APIs allows a specifying a time period for which to export data. However, the change here means that we no longer support "listing" exports since they are now user defined. Instead, a StartTime must be specified in the config if starting fresh. However, if a "sync" exists for the provided configuration, StartTime will simply be ignored.

The second implication of this is that the export duration must be specified in the config as well. The default value (if unspecified) is now 1 hour.

New fields

With the new APIs a few new fields have been added:

UserCreated:
EventTargetSelector
SessionStart
PageStart
PagePlatform
PageScreenWidth
PageScreenHeight
PageViewportWidth
PageViewportHeight

Special cased fields

EventTargetSelectorTok: this will only be exported if a database already exists with this column.
PageUserAgent: this field used to be called PageAgent. If a database already has this column, PageUserAgent will be translated to match. Otherwise, the column will be named PageUserAgent.

camphillips22 commented 3 years ago

Ok @claygoddard I think I addressed all of the first pass.

One question: should we update the Readme with this PR, or do it after this is merged? In one sense, I think it's fine to wait since this is more or less backwards compatible. In another sense, the readme isn't accurate anymore 😬.

I think my preference is that we merge this, then follow up with the README changes and release a 1.0.0 binary. WDYT?

claygoddard commented 3 years ago

I think my preference is that we merge this, then follow up with the README changes and release a 1.0.0 binary. WDYT?

I don't feel strongly so totally up to you. I'd be fine with you including it here though for consistency with the code and config.

claygoddard commented 3 years ago

Ok @claygoddard I think I addressed all of the first pass.

I think there are a few minor ones that weren't addressed that you didn't respond to or said you would address but didn't do anything. I think they were mostly comment improvements / nits so otherwise I think this LG!

claygoddard commented 3 years ago

I think there are a few minor ones that weren't addressed that you didn't respond to or said you would address but didn't do anything.

Whoops, this was wrong on my end. I was clicking through some of the comments which was landing me on old historical views of the files 🤦. Things look good all in all!

fullstorydev / hauser