search

thinhle-agilityio / gatsby-source-apiserver

A gatsby source plugin for pulling data from api server.
63 stars 34 forks source link

[error] Cannot read property 'map' of undefined #40

Open wybitnygracz opened 4 years ago

wybitnygracz commented 4 years ago

I get en error during "gatsby build". It's hard for debugging because i got below error only when i deal with 10000+ records from api (with the same API records limited to 5000 i do not get en error - probalby there is some problem with some record between 5000 and 10000). How can i debug it?

I use lates version of "gatsby-source-apiserver"

ERROR #11321 PLUGIN

"gatsby-source-apiserver" threw an error while running the sourceNodes lifecycle:

Cannot read property 'map' of undefined

TypeError: Cannot read property 'map' of undefined

thinhle-agilityio commented 4 years ago

I think there is one of your records missed some field so it throw error. I suggest to put some default schemaType for the field you think it might be missed

      // Define schemaType to normalize blank values
      // example:
      // const postType = {
      //   id: 1,
      //   name: 'String',
      //   published: true,
      //   object: {a: 1, b: '2', c: false},
      //   array: [{a: 1, b: '2', c: false}]
      // }
      schemaType: postType,
wybitnygracz commented 4 years ago

@thinhle-agilityio i have schema for my entity.

How to find out which record makes the problem? I tried 

try { 
entities = standardizeKeys(entities);
 } catch(e) { 
console.log(schemaType);
console.log(entityType);
console.log(entities); //i get here undefined
throw new Error(e);
}
Kerl1310 commented 4 years ago

I'm getting a similar error. articleType schema is attached at the top of my gatsby-config.js

Schema:

const articleType = {
  type_of: 'String',
  id: 1,
  title: 'String',
  description: 'String',
  cover_image: 'String',
  published: true,
  published_at: 'DateTime',
  slug: 'String',
  path: 'String',
  url: 'String',
  canonical_url: 'String',
  comments_count: 1,
  positive_reactions_count: 1,
  page_views_count: 1,
  published_timestamp: 'String',
  body_markdown: 'String',
  user: {
    name: 'String',
    username: 'String',
    twitter_username: 'String',
    github_username: 'String',
    website_url: 'String',
    profile_image: 'String',
    profile_image_90: 'String',
  },
  tag_list: ['tech', 'software'],
  organization: {
    name: 'String',
    username: 'String',
    slug: 'String',
    profile_image: 'String',
    profile_image_90: 'String',
  },
  flare_tag: {
    name: 'String',
    bg_color_hex: 'String',
    text_color_hex: 'String',
  },
}

Plugin Config:

{
      resolve: 'gatsby-source-apiserver',
      options: {
        // Type prefix of entities from server
        typePrefix: 'internal__',

        // The url, this should be the endpoint you are attempting to pull data from
        url: `https://dev.to/api/articles/me/published`,

        method: 'get',

        headers: {
          'Content-Type': 'application/json',
          'api-key': config.devApiKey,
        },

        // Request body
        data: {},

        // Name of the data to be downloaded.  Will show in graphQL or be saved to a file
        // using this name. i.e. posts.json
        name: `articles`,

        // Nested level of entities in response object, example: `data.posts`
        //entityLevel: ``,

        // Define schemaType to normalize blank values
        // example:
        schemaType: articleType,

        // Request parameters
        // Only available from version 2.1.0
        params: {
          per_page: 10,
          page: 1,
        },

        // Optional payload key name if your api returns your payload in a different key
        // Default will use the full response from the http request of the url
        payloadKey: `body`,

        // Optionally save the JSON data to a file locally
        // Default is false
        localSave: true,

        //  Required folder path where the data should be saved if using localSave option
        //  This folder must already exist
        path: `${__dirname}/src/data/auth/`,

        // Optionally include some output when building
        // Default is false
        verboseOutput: true, // For debugging purposes

        // Optionally skip creating nodes in graphQL.  Use this if you only want
        // The data to be saved locally
        // Default is false
        skipCreateNode: false, // skip import to graphQL, only use if localSave is all you want

        // Optionally re-source data when it changes and
        // `gatsby develop` is running.
        // Requires `ENABLE_GATSBY_REFRESH_ENDPOINT=true`.
        // See https://www.gatsbyjs.org/docs/environment-variables/#reserved-environment-variables
        // Default is false
        enableDevRefresh: true,

        // Optionally override key used to re-source data
        // when `gatsby develop` is running.
        // Requires `enableDevRefresh: true`.
        // See setting directly above this one.
        // See also https://github.com/gatsbyjs/gatsby/issues/14653
        // Default is `id`
        refreshId: `id`,

        // Pass an array containing any number of the entity configuration properties (except verbose, auth0Config),
        // any not specified are defaulted to the general properties that are specified
        // Only available from version 2.1.0
        entitiesArray: [
          {
            typePrefix: 'internal__',
            url: `https://dev.to/api/articles/me/published`,
            method: 'get',
            headers: {
              'Content-Type': 'application/json',
              'api-key': config.devApiKey,
            },
            name: `articles`,
            schemaType: articleType,
          },
        ],
      }

JSON gets saved to a local file fine, just fails to create the graphql nodes

jacobcarrington commented 3 years ago

I get this issue when my API returns no results, which is possible and acceptable in my case. The fix is pretty easy and will put up a PR.

normalize.js

// Create nodes from entities
exports.createNodesFromEntities = ({entities, entityType, schemaType, devRefresh, enableRefreshEndpoint, refreshId, createNode, createNodeId, reporter}) => {

to 

// Create nodes from entities
exports.createNodesFromEntities = ({entities = [], entityType, schemaType, devRefresh, enableRefreshEndpoint, refreshId, createNode, createNodeId, reporter}) => {

Since entities is undefined, but the function is expected it to be an array, this is an easy way to insure this still runs with no results.