search

gatsbyjs / gatsby

The best React-based framework with performance, scalability and security built in.
https://www.gatsbyjs.com
MIT License
55.26k stars 10.32k forks source link

Upgrade from gatsby@1.9.123 to gatsby@1.9.127 seemingly breaks JSON parsing when there's a significant amount of JSON #3094

Closed ghost closed 6 years ago

ghost commented 6 years ago

Preamble

Gatsby version: 1.9.127 node.js version: v8.9.1 OS version: Linux version 4.13.16-35.current (Solus OS)

gatsby-config.js

module.exports = {
  siteMetadata: {
    title: `someTitle`,
    buildID: '22166834-86c7-4c26-9770-afd9b2889e09'
  },
  plugins: [
    `gatsby-plugin-react-helmet`,
    `gatsby-plugin-styled-components`,
    `gatsby-plugin-react-next`,
  ],
  pathPrefix: '/e2fd2e20-0e66-4c28-b503-e841e8f04769'
}

package.json:

{
  "name": "someName"
  "description": "some landing pages"
  "version": "1.0.0",
  "author": ""
  "dependencies": {
    "gatsby": "^1.9.73",
    "gatsby-link": "^1.6.22",
    "gatsby-plugin-react-helmet": "^1.0.8",
    "gatsby-plugin-react-next": "^1.0.4",
    "moment": "^2.19.1",
    "query-string": "^5.0.1",
    "react-redux": "^5.0.6",
    "rebass": "^1.0.4",
    "redux": "^3.7.2",
    "uuid": "^3.1.0"
  },
  "keywords": [
    "gatsby"
  ],
  "license": "MIT",
  "main": "n/a",
  "scripts": {
    "build": "gatsby build",
    "develop": "gatsby develop",
    "format": "prettier --trailing-comma es5 --no-semi --single-quote --write 'src/**/*.js'",
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "devDependencies": {
    "gatsby-plugin-styled-components": "^1.0.5",
    "prettier": "^1.7.4"
  }
}

The issue

In development mode, some pages with long JSON strings as variables throw this error before successfully recompiling (after which the pages can be successfully rendered in-browser):

$ gatsby develop
success delete html files from previous builds — 0.027 s
success open and validate gatsby-config.js — 0.004 s
success copy gatsby files — 0.013 s
success onPreBootstrap — 0.005 s
success source and transform nodes — 0.011 s
success building schema — 0.060 s
success createLayouts — 0.015 s
success createPages — 0.004 s
success createPagesStatefully — 0.146 s
success onPreExtractQueries — 0.001 s
success update schema — 0.054 s
success extract queries from components — 0.083 s
success run graphql queries — 0.034 s
success write out page data — 0.008 s
success write out redirect data — 0.000 s
success onPostBootstrap — 0.000 s
info bootstrap finished - 1.401 s
>  ERROR  Failed to compile with 2 errors                                                                                              18:40:46
 error  in ./.cache/json/district-8055.json
Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
 @ ./.cache/sync-requires.js 145:22837-22947
 error  in ./.cache/json/district-8056.json
Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)
 @ ./.cache/sync-requires.js 145:23135-23245
 WAIT  Compiling...                                                                                                                  18:40:46
 DONE  Compiled successfully in 389ms

When trying to create the production build, this error is thrown, causing the whole build to fail:

$ gatsby build
success delete html files from previous builds — 0.030 s
success open and validate gatsby-config.js — 0.004 s
success copy gatsby files — 0.013 s
success onPreBootstrap — 0.007 s
success source and transform nodes — 0.013 s
success building schema — 0.090 s
success createLayouts — 0.018 s
success createPages — 0.001 s
success createPagesStatefully — 0.150 s
success onPreExtractQueries — 0.001 s
success update schema — 0.059 s
success extract queries from components — 0.041 s
success run graphql queries — 0.021 s
success write out page data — 0.006 s
success write out redirect data — 0.000 s
success onPostBootstrap — 0.000 s
info bootstrap finished - 1.342 s
success Building CSS — 4.001 s
success Building production JavaScript bundles — 11.812 s
error Building static HTML for pages failed
See our docs page on debugging HTML builds for help https://goo.gl/yL9lND
  Error: Module build failed: SyntaxError: Unexpected end of JSON input
  - JSON.parse
  - index.js:4 Object.module.exports
    [mobytrip-landing-pages]/[json-loader]/index.js:4:49
  - @ ./.cache/sync-requires.js 144:15387-15497
error Command failed with exit code 1.

Do let me know if you need more information or an example page itself to work with. Thanks in advance.

c2pig commented 6 years ago

i have the same error

Gatsby version: 1.9.126 node.js version: v8.7.0 OS version: Darwin Kernel Version 17.0.0

success createPages — 9.367 s
success createPagesStatefully — 11.824 s
success onPreExtractQueries — 0.005 s
success update schema — 2.286 s
success extract queries from components — 0.243 s
success run graphql queries — 0.611 s
success write out page data — 0.013 s
success write out redirect data — 0.001 s
Generating image thumbnails [==============================] 720/720 26.1 secs 100%

info bootstrap finished - 107.035 s

success onPostBootstrap — 0.001 s
success Building CSS — 30.489 s
success Building production JavaScript bundles — 52.952 s

error Building static HTML for pages failed

See our docs page on debugging HTML builds for help https://goo.gl/yL9lND

  Error: Module build failed: SyntaxError: Unexpected end of JSON input

  - JSON.parse

  - index.js:4 Object.module.exports
    [xxxxxxxxx]/[json-loader]/index.js:4:49

  - @ ./.cache/sync-requires.js 26:13635-13717
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! Exit status 1
KyleAMathews commented 6 years ago

And downgrading reliably fixes this?

This actually makes sense — we saw this before where some queries hadn't finished being written out before moving to next step which meant the JSON was sometimes only partially written out. It's possible this recent PR introduced a similar bug https://github.com/gatsbyjs/gatsby/pull/2969

c2pig commented 6 years ago

Error message below does not show the root cause(deeper error stack) :- 

  Error: Module build failed: SyntaxError: Unexpected end of JSON input  
  - JSON.parse  
  - index.js:4 Object.module.exports
    [xxxxxxxxx]/[json-loader]/index.js:4:49

so i have to console.log(webpackErrors) manually. :- 

Building static HTML for pages[ './.cache/json/quantity-surveying.json\nModule build failed: SyntaxError: Unexpected end of JSON input\n    at JSON.parse (<anonymous>)\n    at Object.module.exports (/Users/c2pig/projects/postschool-fe/node_modules/json-loader/index.js:4:49)\n @ ./.cache/sync-requires.js 26:13635-13717',
  './.cache/json/veterinary-science.json\nModule build failed: SyntaxError: Unexpected end of JSON input\n    at JSON.parse (<anonymous>)\n    at Object.module.exports (/Users/c2pig/projects/postschool-fe/node_modules/json-loader/index.js:4:49)\n @ ./.cache/sync-requires.js 26:13877-13959',
  './.cache/json/real-estate-management.json\nModule build failed: SyntaxError: Unexpected end of JSON input\n    at JSON.parse (<anonymous>)\n    at Object.module.exports (/Users/c2pig/projects/postschool-fe/node_modules/json-loader/index.js:4:49)\n @ ./.cache/sync-requires.js 26:14123-14209',

I found that image links in markdown files transformed to base64 string and larger json files will be produced.

Once gatsby-remarks-images remove from gatsby-config.js, project built successfully. ie:

      resolve: `gatsby-transformer-remark`,
      options: {
        plugins: [
          /* REMOVED
          {
            resolve: `gatsby-remark-images`,
            options: {
              maxWidth: 1000,
            },
          },
          */
        ...
        ],
      },

Hope this helps.

ghost commented 6 years ago

@KyleAMathews Yep, downgrading reliably fixes this. I've run builds both on 1.9.123 and on 1.9.127 for various sites containing a few hundred pages each. 1.9.123 builds succeed every time whereas 1.9.127 cannot be relied upon to do the same.

Example of a long line in our case:

const hotels = JSON.parse(`[{"hotel_id": 240617, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Oslo Apartments - Sven Bruns Gate, Part of Forenom", "stars": 0.0, "cnt_points_in_1km": 75, "cnt_restaurants_in_1km": 85, "deeplink": "http://www.booking.com/hotel/no/oslo-apartments-as.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/376/37662599.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 6}, "review_nr": 592, "review_score": "7.8", "review_score_word": "Good", "price": 87, "price_formatted": "$87", "max_price": 124, "max_price_formatted": "$124", "p_off": 30}, {"hotel_id": 2040900, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Norwegian Hotelapartments - Bygd\u00f8y All\u00e9", "stars": 0.0, "cnt_points_in_1km": null, "cnt_restaurants_in_1km": null, "deeplink": "http://www.booking.com/hotel/no/apartment-bygdoy-alle.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/827/82744477.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 1}, "review_nr": 8, "review_score": "6.5", "review_score_word": "Pleasant", "price": 141, "price_formatted": "$141", "max_price": 195, "max_price_formatted": "$195", "p_off": 28}, {"hotel_id": 411653, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Oslo Apartments - Rosenborggate, Part of Forenom", "stars": 0.0, "cnt_points_in_1km": 20, "cnt_restaurants_in_1km": 24, "deeplink": "http://www.booking.com/hotel/no/oslo-apartments-rosenborggate-24.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/786/78664437.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 2}, "review_nr": 208, "review_score": "6.9", "review_score_word": "Pleasant", "price": 87, "price_formatted": "$87", "max_price": 113, "max_price_formatted": "$113", "p_off": 23}, {"hotel_id": 1986972, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Frogner House Apartments - Huitfeldtsgate 19", "stars": 0.0, "cnt_points_in_1km": 61, "cnt_restaurants_in_1km": 66, "deeplink": "http://www.booking.com/hotel/no/frogner-house-apartments-huitfeldtsgate-19.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/800/80046450.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 4}, "review_nr": 120, "review_score": "8.9", "review_score_word": "Fabulous", "price": 354, "price_formatted": "$354", "max_price": 406, "max_price_formatted": "$406", "p_off": 13}, {"hotel_id": 1093856, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Saga Apartments Oslo", "stars": 0.0, "cnt_points_in_1km": 28, "cnt_restaurants_in_1km": 39, "deeplink": "http://www.booking.com/hotel/no/majorstuen-apartments.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/463/46341344.jpg", "room_min_price": {"roomtype_id": 1, "refundable": true, "num_rooms_available_at_this_price": 1}, "review_nr": 457, "review_score": "8.7", "review_score_word": "Fabulous", "price": 168, "price_formatted": "$168", "max_price": 229, "max_price_formatted": "$229", "p_off": 26}, {"hotel_id": 235675, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Saga Hotel Oslo", "stars": 4.0, "cnt_points_in_1km": 35, "cnt_restaurants_in_1km": 47, "deeplink": "http://www.booking.com/hotel/no/saga-oslo-as.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/167/16765034.jpg", "room_min_price": {"roomtype_id": 10, "refundable": false, "num_rooms_available_at_this_price": 2}, "review_nr": 825, "review_score": "8.7", "review_score_word": "Fabulous", "price": 181, "price_formatted": "$181"}, {"hotel_id": 47736, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Frogner House Apartments - Bygd\u00f8y All\u00e9 53", "stars": 0.0, "cnt_points_in_1km": 16, "cnt_restaurants_in_1km": 14, "deeplink": "http://www.booking.com/hotel/no/frogner-house-hotel-norum.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/407/4075639.jpg", "room_min_price": {"roomtype_id": 12, "refundable": false, "num_rooms_available_at_this_price": 9}, "review_nr": 599, "review_score": "8.3", "review_score_word": "Very good", "price": 156, "price_formatted": "$156", "max_price": 197, "max_price_formatted": "$197", "p_off": 21}, {"hotel_id": 280862, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Cochs Pensjonat", "stars": 2.0, "cnt_points_in_1km": 44, "cnt_restaurants_in_1km": 51, "deeplink": "http://www.booking.com/hotel/no/cochs-pensjonat.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/629/62949726.jpg", "room_min_price": {"roomtype_id": 24, "refundable": true, "num_rooms_available_at_this_price": 2}, "review_nr": 2403, "review_score": "8.1", "review_score_word": "Very good", "price": 113, "price_formatted": "$113"}, {"hotel_id": 2776957, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Frogner House Apartments- Helgesens gate 1", "stars": 0.0, "cnt_points_in_1km": null, "cnt_restaurants_in_1km": null, "deeplink": "http://www.booking.com/hotel/no/frogner-house-apartments-helgesens-gate-1.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/116/116707425.jpg", "room_min_price": {"roomtype_id": 1, "refundable": false, "num_rooms_available_at_this_price": 1}, "review_nr": 0, "review_score": null, "review_score_word": null, "price": 187, "price_formatted": "$187"}, {"hotel_id": 23636, "locale": "en-US", "languagecode": "en-us", "currencycode": "USD", "hotel_name": "Thon Hotel Gyldenl\u00f8ve", "stars": 3.0, "cnt_points_in_1km": 26, "cnt_restaurants_in_1km": 33, "deeplink": "http://www.booking.com/hotel/no/thon-gyldenlove.xu.html?room1=A;aid=842825;checkin=2017-12-09;checkout=2017-12-10;lang=en-us;selected_currency=USD", "hotel-photo-medium": "https://aff.bstatic.com/images/hotel/max300/204/20438979.jpg", "room_min_price": {"roomtype_id": 10, "refundable": true, "num_rooms_available_at_this_price": 6}, "review_nr": 781, "review_score": "7.9", "review_score_word": "Good", "price": 168, "price_formatted": "$168"}]`)
KyleAMathews commented 6 years ago

Can one of you create a site which reproduces the problem and push it to GitHub? Or would you like to take a crack at figuring out how the changes in #2969 broke things for you?

ghost commented 6 years ago

I can do the first for sure, just gotta clean up the data for public access. I'll take a crack at the second too if I get some time later.

ghost commented 6 years ago

Here's the site that reproduces the problem: https://github.com/TripElephant/gatsby-issue-3094

kennedyrose commented 6 years ago

I've got the same issue. I don't have any long JSON strings, I just have a lot of small JSON files. I get the error on about 10 of them in dev. Same deal, seems to work fine in 1.9.123.

KyleAMathews commented 6 years ago

Should be fixed in #3218 lemme know if you're still seeing troubles!

gaearon commented 6 years ago

I am still seeing this (or maybe other?) issue when editing a newly added long blog post, even after to upgrading to 1.9.135.

Here is my log after touching a file and immediately editing it back:

info changed file at /Users/dan/p/reactjs.org/content/blog/2017-12-14-improving-the-repository-infrastructure.md
 WAIT  Compiling...                                                                              01:34:15

info changed file at /Users/dan/p/reactjs.org/content/blog/2017-12-14-improving-the-repository-infrastructure.md
 DONE  Compiled successfully in 4336ms                                                           01:34:21

 WAIT  Compiling...                                                                              01:34:21

info changed file at /Users/dan/p/reactjs.org/content/blog/2017-12-14-improving-the-repository-infrastructure.md
 ERROR  Failed to compile with 4 errors                                                          01:34:23

 error  in ./.cache/json/blog-2014-01-06-community-roundup-14-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)

 @ ./.cache/sync-requires.js 28:18085-18179

 error  in ./.cache/json/blog-2014-02-05-community-roundup-15-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)

 @ ./.cache/sync-requires.js 28:18351-18445

 error  in ./.cache/json/blog-2014-02-15-community-roundup-16-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)

 @ ./.cache/sync-requires.js 28:18617-18711

 error  in ./.cache/json/blog-2014-02-16-react-v-0-9-rc-1-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)

 @ ./.cache/sync-requires.js 28:18879-18969

 WAIT  Compiling...                                                                              01:34:23

 DONE  Compiled successfully in 1712ms                                                           01:34:26

 WAIT  Compiling...                                                                              01:34:26

 DONE  Compiled successfully in 3748ms                                                           01:34:31

 WAIT  Compiling...                                                                              01:34:32

 DONE  Compiled successfully in 984ms                                                            01:34:35

 WAIT  Compiling...                                                                              01:34:35

 DONE  Compiled successfully in 447ms                                                            01:34:36

 WAIT  Compiling...                                                                              01:34:36

 DONE  Compiled successfully in 784ms                                                            01:34:37

 WAIT  Compiling...                                                                              01:34:37

 DONE  Compiled successfully in 485ms                                                            01:34:38

 WAIT  Compiling...                                                                              01:34:38

 ERROR  Failed to compile with 1 errors                                                          01:34:40

 error  in ./.cache/json/blog-2017-06-13-react-v-15-6-0-html.json

Syntax Error: Unexpected end of JSON input
    at JSON.parse (<anonymous>)

 @ ./.cache/sync-requires.js 28:40543-40631

 WAIT  Compiling...                                                                              01:34:40

 DONE  Compiled successfully in 1113ms                                                           01:34:42

Note how, in addition to these errors, it also recompiles the same thing five times.

ghost commented 6 years ago

@KyleAMathews Will try it out soon! Thanks so much for looking into this.

ghost commented 6 years ago

@KyleAMathews Just tested #3218 on 660 different sites spanning 56,275 pages. Works like a charm! Thanks again for the fix--we really appreciate it.

KyleAMathews commented 6 years ago

@healingfromjava lol what in the world are you doing? 660 sites? Glad it works though!

attfarhan commented 6 years ago

@gaearon were you able to resolve your issue? I'm seeing a similar error to the one you posted, which results in hot reloading not working for pages generated from markdown files, even after updating to the latest Gatsby version.

gaearon commented 6 years ago

My understanding is it was supposed to be fixed with https://github.com/gatsbyjs/gatsby/pull/3237. I haven't actually checked.

stevensurgnier commented 6 years ago

I'm still seeing this behavior on 1.9.157 with our project. I'll report back with more info.