fullstory.ExportList is throwing a 404 error for bigquery

[sonalcodeeight]

2018/12/12 14:09:05 Checking if table fullstory_data exists
2018/12/12 14:09:06 Export table exists, making sure the schema in BigQuery is compatible with the schema specified in Hauser
2018/12/12 14:09:07 Checking if table fullstory_sync_data exists
2018/12/12 14:09:08 Checking if table fullstory_data exists
2018/12/12 14:09:09 Checking for new export files since 0001-01-01 00:00:00 +0000 UTC
2018/12/12 14:09:10 Failed to fetch export list: fullstory: response error: Status:404 Not Found, StatusCode:404, RetryAfter:0

What am I missing?

Here is my config - i've set bucket name, project name, dataset name and table names:

FsApiToken = "token"
Backoff = "30s"
BackoffStepsMax = 8
CheckInterval = "30m"
TmpDir = "/tmp"
Warehouse="bigquery"
GroupFilesByDay = false

[gcs]
Bucket = "bucketname"
GCSOnly = false

[bigquery]
Project = "projectname"
Dataset = "test_fullstory"
ExportTable = "fullstory_data"
SyncTable = "fullstory_sync_data"

Thanks!

[jameremo]

Hello! It looks like hauser could could not find any data export bundles to download. It's using FullStory's data export API to download data, which requires that you have a valid API key in the FsApiToken setting. I apologize if you have that set up, it's hard to tell if it's simply been redacted in the config excerpt above. You could maybe try the data export APIs in Postman or some other test REST client just to make sure that you can access them independently of hauser.

Beyond that, it's hard to make a more definite diagnosis without knowing some details about your account. For that, it would be best if you contact support@fullstory.com, and mention this issue so that they route it back to me.

Hope that helps!

[sonalcodeeight]

@jameremo thanks for your response.

I did confirm that i can call https://export.fullstory.com/api/v1/export/list with the api token wrapped in the Authorization header as specified by the data export api and I do get a list of data via postman.

But I still get this 404 error with that same api token in the config.

Thanks.

[jameremo]

@sonalcodeeight I can't reproduce this problem with a fresh dataset; what kind of data do you have in your fullstory_sync_data table? From the following log line:

2018/12/12 14:09:09 Checking for new export files since 0001-01-01 00:00:00 +0000 UTC

It would appear that hauser somehow determined that the time from which it should start checking is a zero value for time.Time. Normally if there are no rows in the sync table, it starts from 2015 (see the Big Query implementation of Warehouse.LastSyncPoint).

It would also be helpful to me if I could correlate our server's logs with this activity, so I can see exactly what kind of requests hauser made. Could you please open an issue with support@fullstory.com so I know which account you are using?

Thanks!

[jameremo]

@sonalcodeeight Never mind, I see the ticket you opened earlier, thanks! 😄

[jameremo]

I have been working directly with @sonalcodeeight on an internal FullStory ticket, but the solution to this problem should be included here for all to see.

Hauser has a vendored dependency on https://github.com/nishanths/fullstory. Within the hauser repo, we've made a change to the base URL for data export API calls, to reflect the fact that those APIs are now served from https://export.fullstory.com, not https://fullstory.com. Unfortunately, this URL change has not been submitted to the original repo. When building with Go versions prior to 1.11 (or Go 1.11 without modules), the vendored dependency is used, so there has been no urgent need thus far to submit a PR to update URLs.

In this particular case, hauser was being built using Go 1.11, with modules turned on. (The repo was cloned directly and then go mod init run in the root of the cloned directory). The unmodified github.com/nishanths/fullstory dependency was thus fetched directly from GitHub, which caused hauser to make API calls using the wrong URL.

In the short term, this can be fixed by adding the following line to the .toml file:

ExportURL = "https://export.fullstory.com/api/v1"

In the long-term, we will submit a PR to github.com/nishanths/fullstory to update URLs as appropriate. I think we should also discuss providing better documentation (or pointers to documentation) on getting started with Go. My casual observation is that most of our customers who want to use hauser are not overly familiar with Go.

[jhump]

@jameremo, I think this could be better solved by simply providing release binaries (and updating the README to point people to download them) and also configuring a go.mod file so that it builds correctly with Go 1.11 with modules enabled.

We had a contribution to grpcurl to setup stuff like this, so we should be able to apply the same technique to hauser.

As for go.mod, it's not that hard to setup: the Go tool provides a way to auto-create it for any given package, and it is surprisingly good.

[jameremo]

Another update: with ExportURL config above, you can work around the old URLs in nishanths/fullstory, but today I found that it also doesn't set the right headers when downloading exports. The header problem cannot be circumvented with config.

41 contains support for Go modules, and will produce a build using the proper dependencies.

I agree with @jhump that we should provide release binaries and update the README appropriately. (and many thanks to him for adding the module support). PRs will be forthcoming.

[camphillips22]

I'm closing this issue since it's both stale, and the original problems are now fixed: 1) We no longer use the https://github.com/nishanths/fullstory package as a dependency and instead use an HttpClient directly. #83 2) We now are providing release binaries!.