JSON is being garbled* - Githubissues

chanced commented 4 years ago

🐛 Bug Report

JSON data being sent from a route is being garbled*. It doesn't seem to be determined by file size or specific fields, so far as I can tell. The problem exists whether the data is returned or sent via reply.send(data) with reply.type('application/json') being set.

*garbled is definitely a technical term.

To Reproduce

Steps to reproduce the behavior:


import compress from 'fastify-compress';

export default fp(async (fastify, config?) => {
  fastify.register(blipp);
  fastify.register(cors);
  fastify.register(jwt);
  fastify.register(oauth2);
  fastify.register(compress);
  fastify.register(routes);

});

  fastify.get('/', async function(req, reply) {
    reply.type('application/json');
    reply.send([
      {
        schools: [
          '5e5978ca347b5e7e5ddf61c1',
          '5e5978ca347b5e7e5ddf61c2',
          '5e5978ca347b5e7e5ddf61c4',
          '5e5978ca347b5e7e5ddf61c5',
          '5e5978ca347b5e7e5ddf61c6',
          '5e5978ca347b5e7e5ddf61c9',
          '5e5978ca347b5e7e5ddf61ca',
          '5e5978ca347b5e7e5ddf61cb',
          '5e5978ca347b5e7e5ddf61cc',
          '5e5978ca347b5e7e5ddf61cf',
          '5e5978ca347b5e7e5ddf61d0',
          '5e5978ca347b5e7e5ddf61d1',
          '5e5978ca347b5e7e5ddf61d2',
          '5e5978ca347b5e7e5ddf61d3'
        ],
        lotSizeUnits: 'Square Feet',
        media: [
          { url: '/tmp/5380090/1.jpg', order: 1, type: 'img' },

          { url: '/tmp/5380090/9.jpg', order: 30, type: 'img' }
        ],
        propertyType: 'Residential',
        publicRemarks:
          'Customized one-story home in a desirable cul-de-sac location on an oversized 0.387 acre lot.  This open floor plan home offers 5 bedrooms/3 baths, espresso wood-look tile floors, a separate office, dining and breakfast rooms, upgraded lighting, 3 car tandem garage, a gorgeous custom steel front door, a massive master walk-in closet, and custom built-ins in living and mud room.',
        standardStatus: 'Active',
        statusChangeTimestamp: '2018-05-09T21:12:15.500Z'
      }
    ]);
  });

I've stripped down the least amount of fields to reproduce the error. Removing fields results in expected results (JSON being returned to the client). I'm not sure if the fields themselves are having an impact but so far as I can tell, they are not. It works fine on a much larger data set that needs to get sent down as one req (512kb).

Initially I was just returning the data:

fastify.get('/', async function(req, reply){ 
  const results = fastify.models.Listings.find({}).exec()
  return await results
})

but ran into this issue after installing fastify-compress so I tried setting the content-type explicitly as above

Expected behavior

JSON to be returned to the client.

Paste the results here:

<����d"�����DA-�5�rSe�^gaݪ�πʘ���շD�����=���-��g�嚊u���ú��B1�TE�(�
kB��u���V� ����
$Ђ�Izj0�����n��MT
�L�[�y]�Q��}�K��JUJ∏y�%�9�G����Auq%�嫊?������̆������F^><�g���A�y�s�*2�T�   |غ�����("�v(je��QzJ�e�����h2D�eT���;ǹWl����T�����Rf�@���x��p��f˓����w��
P�zx��Z�v�?�>�&/Rp��Nч    Z�K�f^�0zY�����En��>9��A1�wQ���U�G`!�^_o�Epԟy�@�����
��]�ᤘ~Dtĕ�"D֒�����

Your Environment

node version: v13.8.0
fastify version: >=2.12.1
os: Mac
ts-node: v8.6.2
latest stable fastify-compress

mcollina commented 4 years ago

Can you add an example that can be run with node server? How are you calling the server?

chanced commented 4 years ago

@mcollina Absolutely. I'm having to pick up TypeScript so I can hopefully find a gig but I'm not a huge fan of it anyway. This'll be a good excuse to drop it.

Fortunately I'm just getting spun up on this project so my codebase is small. Here's end-to-end on how it's called:

app.ts:

import Fastify from 'fastify';
import jwt from './plugins/jwt';
import oauth2 from './plugins/oauth2';
import routes from './routes';
import fp from 'fastify-plugin';
import mongoose from 'fastify-mongoose';
import blipp from 'fastify-blipp';
import cors from 'fastify-cors';
import { connect as connectUser } from './models/users';
import { connect as connectListings } from './models/listings';

export default fp(async (app, config?) => {
  config = config || {};
  app.register(blipp);
  app.register(cors);
  app.register(jwt);
  app.register(oauth2);
  app.register(compress);
  app.register(mongoose, { uri: process.env.MONGO_URL || 'mongodb://localhost:27017/pear' }).after(err => {
    console.error(err);
    app.decorate('models', {
      Users: connectUser(app.mongo.db),
      Listings: connectListings(app.mongo.db)
    });
  });
  app.register(routes);
});

server.ts

const startTime = Date.now();
require('dotenv').config();
import Fastify from 'fastify';
import core from './app';
import prettyms from 'pretty-ms';

const start = async () => {
  try {
    const opts = {
      logger: true
    };
    const app = Fastify(opts);
    app.register(core);
    await app.listen(+process.env.APP_PORT || 3734);
    app.ready(err => {
      if (err) {
        console.error(err);
        throw err;
      }
      const timeToLoad = prettyms(Date.now() - startTime);
      app.log.info(`App start time: ⌛ ${timeToLoad}`);
      // app.blipp();
    });
  } catch (e) {
    throw e;
  }
};

try {
  start();
} catch (e) {
  console.error(e);
}

./routes/listings.ts

import Fastify, { FastifyInstance } from 'fastify';
export default async function(fastify: FastifyInstance, opts) {

  fastify.get('/', async function(req, reply) {
    reply.type('application/json');
    reply.send([
      {
        schools: [
          '5e5978ca347b5e7e5ddf61c1',
          '5e5978ca347b5e7e5ddf61c2',
          '5e5978ca347b5e7e5ddf61c4',
          '5e5978ca347b5e7e5ddf61c5',
          '5e5978ca347b5e7e5ddf61c6',
          '5e5978ca347b5e7e5ddf61c9',
          '5e5978ca347b5e7e5ddf61ca',
          '5e5978ca347b5e7e5ddf61cb',
          '5e5978ca347b5e7e5ddf61cc',
          '5e5978ca347b5e7e5ddf61cf',
          '5e5978ca347b5e7e5ddf61d0',
          '5e5978ca347b5e7e5ddf61d1',
          '5e5978ca347b5e7e5ddf61d2',
          '5e5978ca347b5e7e5ddf61d3'
        ],
        lotSizeUnits: 'Square Feet',
        media: [
          { url: '/tmp/5380090/1.jpg', order: 1, type: 'img' },

          { url: '/tmp/5380090/9.jpg', order: 30, type: 'img' }
        ],
        propertyType: 'Residential',
        publicRemarks:
          'Customized one-story home in a desirable cul-de-sac location on an oversized 0.387 acre lot.  This open floor plan home offers 5 bedrooms/3 baths, espresso wood-look tile floors, a separate office, dining and breakfast rooms, upgraded lighting, 3 car tandem garage, a gorgeous custom steel front door, a massive master walk-in closet, and custom built-ins in living and mud room.',
        standardStatus: 'Active',
        statusChangeTimestamp: '2018-05-09T21:12:15.500Z'
      }
    ]);
  });
}

./routes/index.ts

import root from './root';
import listings from './listings';
import auth from './auth';
import { FastifyInstance } from 'fastify';
export default async (fastify: FastifyInstance, opts) => {
  fastify.register(root);
  fastify.register(listings, { prefix: '/listings' });
  fastify.register(auth, { prefix: '/auth' });
};

package.json

{
  "name": "pear",
  "version": "0.0.1",
  "description": "",
  "main": "src/server.ts",
  "directories": {
    "test": "test"
  },
  "scripts": {
    "test": "jest",
    "test:watch": "nodemon --watch tests --ext ts --watch src --ext ts --exec 'npm run test'",
    "start": "nodemon --watch src --ext ts --exec 'ts-node --files src/server.ts | pino-colada'",
    "dev": "concurrently \"npm:test\" \"npm:start\"",
    "lint": "tsc --noEmit && eslint \"**/*.{js,ts}\" --quiet"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "argon2": "^0.26.1",
    "aws-sdk": "^2.630.0",
    "axios": "^0.19.2",
    "dotenv": "^8.2.0",
    "fastify": "^2.12.1",
    "fastify-blipp": "^2.1.0",
    "fastify-cli": "^1.4.0",
    "fastify-compress": "^2.0.1",
    "fastify-cors": "^3.0.2",
    "fastify-jwt": "^1.2.1",
    "fastify-mongodb": "^2.0.0",
    "fastify-mongoose": "^0.3.0",
    "fastify-oauth2": "^3.0.0",
    "fastify-plugin": "^1.6.1",
    "mongodb": "^3.5.4",
    "pretty-ms": "^6.0.0",
    "validator": "^12.2.0"
  },
  "devDependencies": {
    "@shelf/jest-mongodb": "^1.1.3",
    "@types/dotenv": "^8.2.0",
    "@types/expect": "^24.3.0",
    "@types/jest": "^25.1.3",
    "@types/jsonwebtoken": "^8.3.8",
    "@types/mongoose": "^5.7.3",
    "@types/node": "^13.7.7",
    "@typescript-eslint/eslint-plugin": "^2.21.0",
    "@typescript-eslint/parser": "^2.21.0",
    "chalk": "^3.0.0",
    "concurrently": "^5.0.0",
    "eslint": "^6.5.1",
    "eslint-config-airbnb-typescript": "^7.0.0",
    "eslint-config-prettier": "^6.10.0",
    "eslint-plugin-import": "^2.20.1",
    "eslint-plugin-jest": "^23.8.1",
    "eslint-plugin-prettier": "^3.1.2",
    "expect": "^25.1.0",
    "jest": "^25.1.0",
    "mongodb-memory-server-core": "^6.3.1",
    "mongodb-memory-server-global": "^6.3.1",
    "nodemon": "^2.0.2",
    "nyc": "^15.0.0",
    "pino-colada": "^1.5.1",
    "prettier": "^1.18.2",
    "rimraf": "^3.0.0",
    "tap": "^12.5.3",
    "ts-jest": "^25.2.1",
    "ts-node": "^8.6.2",
    "typescript": "^3.8.3"
  }
}

tsconfig.json

{
  "compilerOptions": {
    "lib": ["ES2019"],
    "target": "ES2019", // https://github.com/microsoft/TypeScript/wiki/Node-Target-Mapping
    "module": "commonjs",
    "esModuleInterop": true,
    "allowSyntheticDefaultImports": true,
    "resolveJsonModule": true,
    "moduleResolution": "node",
    "sourceMap": true,
    "outDir": "dist",
    "baseUrl": "." // this may need to be updated to reflect sub services
  }
}

mcollina commented 4 years ago

I cannot run the above. Can you please make a runnable example? Put it in a github repo if you have multiple files.

chanced commented 4 years ago

@mcollina yes, as soon as I unwind TypeScript dependencies.

Sorry for the confusion. I should have been more clear. I didn't expect you to try and run the above. I was answering your question regarding how I'm calling the server.

chanced commented 4 years ago

@mcollina

It returned expected results with a quick fastify cli generated node-only project. When I get time, I'll do some more digging on this code-base, as is, to see if I can track down the issue. It could be a mistake on my end or one of my dependencies. I'll close this ticket as it is not clearly apparent to be a fastify-compress issue.

Thank you for your help.

JonDum commented 4 years ago

I'm also getting this issue with the latest version of fastify-compress.

There's some sort of undocumented breaking change here. Even worse, this doesn't happen in dev or test, only when NODE_ENV=production.

Will update if I can find the cause.

Eomm commented 4 years ago

There's some sort of undocumented breaking change here. Even worse, this doesn't happen in dev or test, only when NODE_ENV=production.

Please, give us a minimal reproducible example and we will happy to help; This plugin doesn't read the NODE_ENV parameter at all

fastify / fastify-compress

JSON is being garbled* #104

🐛 Bug Report

To Reproduce

Expected behavior

Your Environment

app.ts:

server.ts

./routes/listings.ts

./routes/index.ts

package.json

tsconfig.json