Infinite warning loop, don't quite know why its happening...?

abooayoob commented 5 years ago

Summary

When i run gatsby develop or gatsby build I get an infinite loop of warnings similar to this:

gatsby_warnings

outputs lines like theese: warn Multiple node fields resolve to the same GraphQL field `SitePage.context.contentList.js.children.section.div.table.tbody.tr.td.p.span._` - [`_`, `$`]. Gatsby will use `_`.

Only way to get out is close vs code but site does not get built.

I am not sure why this happens, any ideas?

Relevant information

If I comment out theese lines, causing the warning, in gatsby, then I don't get the warnings and the site gets built and works fine.

In case you don't need/want to follow the link above right now: gatsby/packages/gatsby/src/schema/infer/add-inferred-fields.js lines 72 - 78:

      report.warn(
        `Multiple node fields resolve to the same GraphQL field \`${prefix}.${
          field.key
        }\` - [${possibleFieldsNames}]. Gatsby will use \`${
          field.unsanitizedKey
        }\`.`
      )

Environment (if relevant)

System: OS: macOS 10.14.2 CPU: (12) x64 Intel(R) Core(TM) i7-8750H CPU @ 2.20GHz Shell: 3.2.57 - /bin/bash Binaries: Node: 10.15.3 - ~/.nvm/versions/node/v10.15.3/bin/node npm: 6.4.1 - ~/.nvm/versions/node/v10.15.3/bin/npm Languages: Python: 2.7.10 - /usr/bin/python Browsers: Chrome: 74.0.3729.169 Safari: 12.0.2 npmPackages: gatsby: ^2.6.0 => 2.6.0 gatsby-image: ^2.1.0 => 2.1.0 gatsby-plugin-jss: ^2.0.9 => 2.0.9 gatsby-plugin-manifest: ^2.1.1 => 2.1.1 gatsby-plugin-offline: ^2.1.1 => 2.1.1 gatsby-plugin-react-helmet: ^3.0.12 => 3.0.12 gatsby-plugin-sharp: ^2.0.37 => 2.0.37 gatsby-source-filesystem: ^2.0.37 => 2.0.37 gatsby-transformer-sharp: ^2.1.19 => 2.1.19

File contents (if changed)

gatsby-config.js:

require("dotenv").config({
  path: `.env.${process.env.NODE_ENV}`,
})

module.exports = {
  siteMetadata: {
    title: `Kontohjelp`,
    description: `Kick off your next, great Gatsby project with this default starter. This barebones starter ships with the main Gatsby configuration files you might need.`,
    author: `@gatsbyjs`,
  },
  plugins: [
    `gatsby-plugin-react-helmet`,
    {
      resolve: `gatsby-source-filesystem`,
      options: {
        name: `images`,
        path: `${__dirname}/src/images`,
      },
    },
    `gatsby-transformer-sharp`,
    `gatsby-plugin-sharp`,
    {
      resolve: `gatsby-plugin-manifest`,
      options: {
        name: `gatsby-starter-default`,
        short_name: `starter`,
        start_url: `/`,
        background_color: `#663399`,
        theme_color: `#663399`,
        display: `minimal-ui`,
        icon: `src/images/gatsby-icon.png`, // This path is relative to the root of the site.
      },
    },
    {
      resolve: `gatsby-source-kontohjelp-category`,
      options: {
        apiUrl: process.env.GATSBY_KONTOHJELP_API_URL,
      },
    },
    // `gatsby-transformer-kontohjelp-hitblock`,
    // `gatsby-transformer-kontohjelp-hit`,
    `gatsby-plugin-jss`,
  ],
}

package.json:

{
  "name": "gatsby-starter-default",
  "private": true,
  "description": "A simple starter to get up and developing quickly with Gatsby",
  "version": "0.1.0",
  "author": "Kyle Mathews <mathews.kyle@gmail.com>",
  "dependencies": {
    "gatsby": "^2.6.0",
    "gatsby-image": "^2.1.0",
    "gatsby-plugin-jss": "^2.0.9",
    "gatsby-plugin-manifest": "^2.1.1",
    "gatsby-plugin-offline": "^2.1.1",
    "gatsby-plugin-react-helmet": "^3.0.12",
    "gatsby-plugin-sharp": "^2.0.37",
    "gatsby-source-filesystem": "^2.0.37",
    "gatsby-transformer-sharp": "^2.1.19",
    "nanoid": "^2.0.3",
    "node-fetch": "^2.6.0",
    "prop-types": "^15.7.2",
    "react": "^16.8.6",
    "react-dom": "^16.8.6",
    "react-helmet": "^5.2.1",
    "react-jss": "^8.6.1",
    "react-redux": "^7.0.3",
    "redux": "^4.0.1",
    "redux-thunk": "^2.3.0",
    "xml2js": "^0.4.19"
  },
  "devDependencies": {
    "esm": "^3.2.25",
    "prettier": "^1.17.1"
  },
  "keywords": [
    "gatsby"
  ],
  "license": "MIT",
  "scripts": {
    "build": "npx --node-arg '-r esm' gatsby build",
    "clean": "npx gatsby clean",
    "develop": "GATSBY_GRAPHQL_IDE=playground npx --node-arg '-r esm' gatsby develop",
    "format": "prettier --write src/**/*.{js,jsx}",
    "start": "npm run develop",
    "dev": "npm run develop",
    "serve": "npx --node-arg '-r esm' gatsby serve",
    "test": "echo \"Write tests! -> https://gatsby.dev/unit-testing\""
  },
  "repository": {
    "type": "git",
    "url": "https://github.com/gatsbyjs/gatsby-starter-default"
  },
  "bugs": {
    "url": "https://github.com/gatsbyjs/gatsby/issues"
  }
}

gatsby-node.js:

import dotenv from "dotenv"
dotenv.config({
  path: `.env.${process.env.NODE_ENV}`,
})

import path from "path"
import fetch from "node-fetch"
import fs from "fs"
import { slugFriendly, xml2js } from "./utils"

export const onPreInit = () => {
  return fetch(`${process.env.GATSBY_KONTOHJELP_API_URL}/GetAll`, {
    method: `POST`,
  })
    .then(response => response.json())
    .then(data => {
      let hitblocks = data.languages[0].categories.reduce(
        (accumulator, current) => {
          const hitblocks = current.hitblocks.map(hitblock => {
            hitblock.category = current.type
            return hitblock
          })
          accumulator = [...accumulator, ...hitblocks]
          return accumulator
        },
        []
      )
      fs.writeFileSync(
        `${__dirname}/initial-items.json`,
        JSON.stringify(hitblocks),
        `utf-8`
      )
    })
}

export const createPages = ({ graphql, actions }) => {
  const { createPage } = actions

  return graphql(`
    {
      allKontohjelpCategory {
        edges {
          node {
            type
            hitblocks {
              title
              hits {
                title
                html
                tags
                lines {
                  name
                  account
                  accounttitle
                  debet_kredit
                  description
                }
              }
            }
          }
        }
      }
    }
  `).then(result => {
    const hits = []
    // return xml2js(hit.html, js => js).then(js => {
    //   if (!js) {
    //     console.log(`xml2js failed for ${slug}`)
    //   } else {
    //     createPage({
    //       path: slug,
    //       component: path.resolve(`./src/pages/index.js`),
    //       context: {
    //         slug,
    //         contentList,
    //         openInitial: hit.title,
    //         js: processXmlJs(js),
    //       },
    //     })
    //   }
    // })
    result.data.allKontohjelpCategory.edges.forEach(({ node }) => {
      node.hitblocks.forEach(hitblock => {
        hitblock.hits.forEach(hit => {
          const slug = `/${slugFriendly(node.type)}/${slugFriendly(
            hitblock.title
          )}/${slugFriendly(hit.title)}`

          const contentList = hitblock.hits.map(hit => ({
            title: hit.title,
            html: hit.html,
            tags: hit.tags,
            lines: hit.lines,
            category: node.type,
          }))

          hits.push({
            slug,
            contentList,
            openInitial: hit.title,
          })
        })
      })
    })

    const badSlugs = new Set()
    const promises = hits.map(hit => {
      const innerPromises = hit.contentList.map(content => {
        return xml2js(content.html, js => js).then(js => {
          if (!js) badSlugs.add(hit.slug)

          const jsProcessed = js ? processXmlJs(js) : null
          if (jsProcessed)
            jsProcessed.children[0].children = [
              content.category === `kontering`
                ? { "#name": `lines`, lines: content.lines }
                : `remove`,
              ...jsProcessed.children[0].children,
              { "#name": `tags`, tags: content.tags },
            ].filter(item => item !== `remove`)

          return {
            title: content.title,
            js: jsProcessed,
          }
        })
      })

      return Promise.all(innerPromises).then(contentList => ({
        slug: hit.slug,
        contentList,
        openInitial: hit.openInitial,
      }))
    })

    return Promise.all(promises).then(hits => {
      hits.forEach(hit => {
        const { slug, contentList, openInitial } = hit

        if (badSlugs.has(slug)) {
          console.log(`xml2js failed for ${slug}`)
        } else {
          createPage({
            path: slug,
            component: path.resolve(`./src/pages/index.js`),
            context: {
              slug,
              contentList,
              openInitial,
            },
          })
        }
      })
    })
  })
}

const processXmlJs = obj => {
  // record index start and index end
  // put them toghether in a collection of similar pairs

  const boxes = []
  const list = obj.txt.children[0].children.slice()
  list.forEach((child, i) => {
    if (child["#name"] === "p" && child._ && child._.includes("[BOX=")) {
      let boxPosition = [i]
      let end = i + 1
      while (end < list.length) {
        if (
          list[end]["#name"] === "p" &&
          list[end]._ &&
          list[end]._.includes("BOX]")
        ) {
          boxPosition.push(end)
          break
        }
        end++
      }
      boxes.push({
        portion: list.slice(boxPosition[0], boxPosition[1] + 1),
        boxPosition,
      })
    }
  })

  // go through collection and modify list accordingly
  // make box nodes of shape:
  // {
  //   "#name": "box",
  //   "children": [portion of list]
  // }
  boxes.forEach(box => {
    list[box.boxPosition[0]] = {
      "#name": "box",
      children: box.portion,
    }
    let i = box.boxPosition[0] + 1
    while (i <= box.boxPosition[1]) {
      list[i] = "remove"
      i++
    }
  })

  //slice is for removing the first item, as that is the heading, and should be seperate.
  obj.txt.children[0].children = list.filter(item => item !== "remove").slice(1)

  return obj.txt
}

gatsby-browser.js:

import "whatwg-fetch"

// Adding "reset type" styles here, will be extracted when building the javascript
// https://www.gatsbyjs.org/docs/creating-global-styles/#add-global-styles-with-css-files-and-no-layout-component
import "./src/style/normalize.css"
import "./src/style/box-sizing.css"
import "./src/style/resets.css"

export { default as wrapRootElement } from "./src/state/redux-wrapper"

gatsby-ssr.js:

export { default as wrapRootElement } from "./src/state/redux-wrapper"

jonniebigodes commented 5 years ago

@abooayoob can you create a reproduction and also is gatsby-source-kontohjelp-category a local plugin you're working on?

abooayoob commented 5 years ago

@jonniebigodes yup, thanks, will try to set up a repro. Yeah, gatsby-source-kontohjelp-category is a local source plugin that calls an internal api where I work. Will cook up something similar.

abooayoob commented 5 years ago

@jonniebigodes Here is the reproduction repo. If you clone, npm install and then try to run with start, you will see similar output, only less.

The problem is that I am creating a page and putting an object in the context field which causes the output. There is no graphql involved actually.

In my original project I am creating many pages, with many of these large and weird objects (created by the xml2js) package.

jonniebigodes commented 5 years ago

@abooayoob i've been looking at the code you supplied and i think i have a solution for your issue. Going to enumerate the steps i took.

Cloned your repo
Installed the dependencies
I issued gatsby develop and i was able to confirm the issue at hand.
I grabbed the code inside data.js which is nothing more than, nothing else than the json you get from your internal api.
I checked it against a tool i use to validate the json and visualize the structure more in a more friendly way and also i had the xml2js page opened and one thing popped up, namely here. And based on that it clicked, it seems that basically gatsby does not like that structure, so based on the documentation for the package i created a new file called data_v1.json renaming alll $ elements into attributes and all _ elements into content.

So what was previously :

exports.data = {
  txt: {
    "#name": "txt",
    children: [
      {
        $: {
          topic_id: "{6B30A50D-87BB-4E5A-8DAE-E24304BB287F}",
          bm: "-670606198",
          source: "_kontohjelp",
        },
        "#name": "div",
        children: [
          {
            $: { id: "-670606198" },
            "#name": "section",
            children: [
              {
                _: "Skattefri dekning av kontingent til serviceorganisasjoner",
                "#name": "h1",
                children: [
                  {
                    "#name": "__text__",
                    _:
                      "Skattefri dekning av kontingent til serviceorganisasjoner",
                  },
                ],
              },

.....

is now:

{
    "txt": {
      "#name": "txt",
      "children": [
        {
          "attributes": {
            "topic_id": "{6B30A50D-87BB-4E5A-8DAE-E24304BB287F}",
            "bm": "-670606198",
            "source": "_kontohjelp"
          },
          "#name": "div",
          "children": [
            {
              "attributes": {
                "id": "-670606198"
              },
              "#name": "section",
              "children": [
                {
                  "content": "Skattefri dekning av kontingent til serviceorganisasjoner",
                  "#name": "h1",
                  "children": [
                    {
                      "#name": "__text__",
                      "content": "Skattefri dekning av kontingent til serviceorganisasjoner"
                    }
                  ]
                },

Modified gatsby-node.js to the following:


exports.createPages = ({ actions }) => {
const { createPage } = actions

const jsonContent= require('./data_v1.json')
createPage({
path: "/bam/",
component: require.resolve('./src/templates/kontohjelp.js'),
context: {
  data:jsonContent
},
})
}

- To keep it separated created a template called `kontohjelp.js` inside `./src/templates/` with the following code:
```javascript
import React from 'react';

const Kontohjelp=({pageContext})=>{
    console.log('====================================');
    console.log(`props:${JSON.stringify(pageContext,null,2)}`);
    console.log('====================================');

    return (
        <div>
            <h3>this is Kontohjelp</h3>
        </div>
    )
}

export default Kontohjelp

Issued gatsby develop and it yelded the following:

kontohj_1

Opening up http:/localhost:8000/bam yelds the following:
Stopped Gatsby and issued gatsby clean and gatsby build && gatsby serve to clean the .cache and public folders and issue a production build and mock it, as i've opened another browser window http:/localhost:9000/bam yelded the same result as above.

Based on the documentation for the package in question and i haven't fully tested it yet, but it could exist a option that takes care of part of the issue at hand, the other part you will probably need to do some extra work on the json itself.

Feel free to provide feedback so that we can close this issue or continue to work on it till we find a solution.

abooayoob commented 5 years ago

@jonniebigodes Thanks a bunch, this fixes the warnings and issue that I was facing!

Although, I don't think it is clear or intuitive why just renaming some fields from $ and _ is the solution to this issue.

I think we can close this issue. But I think there is either a bug or something that needs to be documentet so that others don't encounter the same problem. I can look into it this or next week. Unless someone else want's to have a go at it.

jonniebigodes commented 5 years ago

@abooayoob it could be a dependency of gatsby acting up or something in that nature that generates this sort of issue. If you want feel free to check on it and see if you find something and with that document the issue and the fix for it. 👍

gatsbyjs / gatsby