search

fullstorydev / hauser

Service for moving your Fullstory export files to a data warehouse
MIT License
49 stars 23 forks source link

Reworked fetching and saving exports into a pipeline #60

Closed camphillips22 closed 4 years ago

camphillips22 commented 5 years ago

This PR is the first part of a re-architecture that is attempting to improve the extensibility of Hauser. The tasks of downloading, decoding, grouping, and saving the FullStory event exports are separated out into separate stages of a Pipeline. The output of the Pipeline is a filename and the corresponding bundle metadata. This is all that's needed to pass to the Warehouse in order upload to the database or bucket storage.

While this PR doesn't make any significant changes to the Hauser feature set, it does enable flexibility in each of the processing steps.

camphillips22 commented 5 years ago

As a comment, ignore the Makefile changes. I have them in a separate PR, but wanted them for my own development on this.

Makefile changes integrated from master.