Analytics Firehose encodes data improperly

vincekd commented 4 years ago

Describe the bug The data files in s3 produced from the firehose are no longer json text after switching to amplify analytics. There is no data transformation functions or lambda conversion set up on the firehose stream.

To Reproduce Steps to reproduce the behavior:

Set up an angular web page using the instructions on the analytics page: https://aws-amplify.github.io/docs/js/analytics#using-amazon-kinesis-firehose

Use amplify to record some data

const record = {
        data: "{json: string}\n",
        streamName: "stream-name",
};
await this.amplify.analytics().record(record, "AWSKinesisFirehose");

Expected behavior The data is stored in the s3 bucket in json format.

Environment

``` > npx envinfo --system --binaries --browsers --npmPackages --npmGlobalPackages npx: installed 1 in 0.97s System: OS: Linux 5.5 Arch Linux CPU: (8) x64 Intel(R) Core(TM) i5-8250U CPU @ 1.60GHz Memory: 4.75 GB / 15.53 GB Container: Yes Shell: 5.0.16 - /bin/bash Binaries: Node: 13.11.0 - /usr/bin/node Yarn: 1.22.4 - /usr/bin/yarn npm: 6.14.4 - /usr/bin/npm Browsers: Firefox: 74.0 npmPackages: @angular-devkit/architect: ^0.801.3 => 0.801.3 @angular-devkit/build-angular: ^0.801.3 => 0.801.3 @angular-devkit/core: ^8.1.3 => 8.3.26 @angular-devkit/schematics: ^8.1.3 => 8.3.26 @angular/cli: ^8.1.3 => 8.3.26 @angular/common: ~8.1.2 => 8.1.3 @angular/compiler: ~8.1.2 => 8.1.3 @angular/compiler-cli: ~8.1.2 => 8.1.3 @angular/core: ~8.1.2 => 8.1.3 @angular/forms: ~8.1.2 => 8.1.3 @angular/language-service: ~8.1.2 => 8.1.3 @angular/platform-browser: ~8.1.2 => 8.1.3 @angular/platform-browser-dynamic: ~8.1.2 => 8.1.3 @angular/router: ~8.1.2 => 8.1.3 @ionic-native/core: ^5.12.0 => 5.23.0 @ionic-native/splash-screen: ^5.12.0 => 5.23.0 @ionic-native/status-bar: ^5.12.0 => 5.23.0 @ionic/angular: ^4.10.2 => 4.11.10 @ionic/angular-toolkit: ~2.0.0 => 2.0.0 @sentry/browser: ^5.8.0 => 5.15.4 @types/googlemaps: ^3.37.4 => 3.39.3 @types/jasmine: ~2.8.8 => 2.8.16 @types/jasminewd2: ~2.0.3 => 2.0.8 @types/node: ~12.0.0 => 12.0.12 @types/papaparse: ^5.0.3 => 5.0.3 aws-amplify: ^3.0.6 => 3.0.6 aws-amplify-angular: ^5.0.6 => 5.0.6 codelyzer: ~4.5.0 => 4.5.0 cordova-plugin-geolocation: ^4.0.2 => 4.0.2 core-js: ^2.5.4 => 2.6.11 jasmine-core: ~2.99.1 => 2.99.1 jasmine-spec-reporter: ~4.2.1 => 4.2.1 karma: ~4.1.0 => 4.1.0 karma-chrome-launcher: ~2.2.0 => 2.2.0 karma-coverage-istanbul-reporter: ^2.0.6 => 2.1.1 karma-jasmine: ~1.1.2 => 1.1.2 karma-jasmine-html-reporter: ^0.2.2 => 0.2.2 papaparse: ^5.1.1 => 5.2.0 polylabel: ^1.0.2 => 1.0.2 protractor: ~5.4.0 => 5.4.3 rxjs: ~6.5.1 => 6.5.5 rxjs-compat: ^6.5.2 => 6.5.5 ts-node: ~8.3.0 => 8.3.0 tslib: ^1.9.0 => 1.11.1 tslint: ~5.17.0 => 5.17.0 typescript: ~3.4.3 => 3.4.5 zone.js: ~0.8.29 => 0.8.29 npmGlobalPackages: @angular/cli: 8.3.21 cordova-res: 0.6.0 cordova: 9.0.0 dom-parser: 0.1.6 express-generator: 4.16.1 generator-express-no-stress-typescript: 6.3.0 gulp: 4.0.2 ionic: 5.2.6 jszip: 3.2.2 lodash: 4.17.15 n: 6.0.1 native-run: 0.2.8 node-fetch: 2.6.0 node-sass: 4.12.0 papaparse: 5.1.1 svgpath: 2.2.3 xmldom: 0.3.0 yo: 3.1.0 ```

Additional context The logs indicate no errors. Had no issue when using @aws-sdk/client-firehose-browser/Firehose

Thanks

vincekd commented 4 years ago

Here's some more info. Looks like the data encoding done before the request is broken.

CODE:

const record = {
    data: JSON.stringify({"event": "testing"}) + '\n',
    partitionKey: "testing",
    streamName: "event-data"
};

try {
    const resp = await this.amplify.analytics().record(record, "AWSKinesisFirehose");
    console.log("resp", resp, record);
} catch (e) {
    console.error("Error recording analytics...", e);
    return false;
}

CONSOLE OUTPUT:

[DEBUG] 28:26.775 AWSKineisFirehoseProvider - putting records to kinesis 
[
  "event-data",
  "with records",
  [
    {
      "Data": "\"{\\\"event\\\":\\\"testing\\\"}\\n\""
    }
  ]
]

REQUEST:

Request URL:https://firehose.us-west-2.amazonaws.com/
Request Method:POST
{"DeliveryStreamName":"event-data","Records":[{"Data":"AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA"}]}

S3 DATA:

\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0

iartemiev commented 4 years ago

Hey @vincekd, this should be fixed now in the latest unstable version of Amplify (npm i -S aws-amplify@unstable) and will be part of the next stable release.

As a side note, when setting the value of data, you can stringify your json if you'd like, but you don't have to. You can just pass an object to data and Amplify will stringify it for you before sending it in the request.

vincekd commented 4 years ago

Thanks @iartemiev!

Regarding the stringifying, if you are trying to access the data from Athena, it needs a newline at the end of each entry for it to be properly read (unless something has changed in the last couple of months).

github-actions[bot] commented 3 years ago

This issue has been automatically locked since there hasn't been any recent activity after it was closed. Please open a new issue for related bugs.

Looking for a help forum? We recommend joining the Amplify Community Discord server *-help channels or Discussions for those types of questions.

aws-amplify / amplify-js