search

fullstorydev / hauser

Service for moving your Fullstory export files to a data warehouse
MIT License
48 stars 23 forks source link

CSV headers not written when using LocalStorage option #80

Closed van-fs closed 4 years ago

van-fs commented 4 years ago

When using local storage:

[local]
SaveDir = "/Users/van/Desktop/exports"
# Start time for data exports in the following format: 2018-12-27T18:30:00Z
StartTime = 2019-11-03T07:00:00Z
UseStartTime = false

The written CSV files lack headers (i.e. column names). The quick fix appears to be simply adding a Write with the existing tableColumns in ProcessFilesByDay and ProcessFilesIndividually.

    csvOut := csv.NewWriter(outfile)

    // write the columns as CSV header
    csvOut.Write(tableColumns)
    csvOut.Flush()  // this flush may not be needed as flush later gets called

Is there any reason that we should not write headers in this scenario? I haven't submitted a PR as I think we might make this configurable if headers should not be present in CSVs to work correctly with other adapters.

camphillips22 commented 4 years ago

Hey @van-fs thanks for pointing this out. We actually don't officially support CSV and local storage yet because we'd like to have consistent headers (or at least column ordering) between files. I'll be working on a change to officially support this soon.

The reason that the header is not included is because when loading data into the database (both BigQuery and Redshift), we aren't specifying that a header should be ignored. So in order to support writing headers (which we should do), we need to update the database loading to ignore the header as well.