search

fullstorydev / hauser

Service for moving your Fullstory export files to a data warehouse
MIT License
49 stars 23 forks source link

Sam/json csv config #54

Closed sam-fs closed 5 years ago

sam-fs commented 5 years ago

SaveAsJson is now a global config variable.

Changed example-config.toml default settings to CSV format, local storage, and GCS only so that a user has more and easier paths to a valid config.

Next step is adding checks and outputing appropriate error messages to the log so that the users can fix their config, but submitting this PR now to get quick fixes published while resolving some cloud credentialing issues.

sam-fs commented 5 years ago

BQ and Red won't accept JSONs directly, so I was thinking the next step would just be adding checks for those bad configs and using log.Fatalf to explain to the user how to correct them. This is how we handle the GroupFilesByDay global config.

On Wed, May 1, 2019 at 1:28 PM Aneesh Butani notifications@github.com wrote:

@butanian commented on this pull request.

In example-config.toml https://github.com/fullstorydev/hauser/pull/54#discussion_r280145837:

@@ -33,7 +34,8 @@ DatabaseSchema = "public"

[gcs] Bucket = "" -GCSOnly = false +# upload data file only (i.e. skip load to BigQuery)?

To fix the problem, is it possible to only be sensitive to conf.Local.SaveAsJson if the selected warehouse is local?

My reasoning is that a global save as json flag implies, we create and upload json file, and then also support loading that format into bq and redshift, which we currently don't do. Thoughts?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/fullstorydev/hauser/pull/54#discussion_r280145837, or mute the thread https://github.com/notifications/unsubscribe-auth/AK67X4G5UN225VP4TOGYWALPTHHKFANCNFSM4HJQKA4Q .

-- Thanks, Sam

butanian commented 5 years ago

SGTM

the travis error was due to checkgofmt not being happy (probably spaces vs. tab on the line added in config.go). Did gofmt -s -w config/config.go to generate the commit I pushed.

sam-fs commented 5 years ago

Ah, noob oversight. :-|

Thank you for pointing that out and making the correction.

On Wed, May 1, 2019 at 2:28 PM Aneesh Butani notifications@github.com wrote:

SGTM

the travis error was due to checkgofmt not being happy (probably spaces vs. tab on the line added in config.go). Did gofmt -s -w config/config.go to generate the commit I pushed.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/fullstorydev/hauser/pull/54#issuecomment-488368904, or mute the thread https://github.com/notifications/unsubscribe-auth/AK67X4D7DG7EGU43PELRGKDPTHOLZANCNFSM4HJQKA4Q .

-- Thanks, Sam

sam-fs commented 5 years ago

This latest travis error may be due to main.go formatting. I've run gofmt -s -w on the file.