Timeout Error when the return limit is increased to grab entire graph

[jrogerthat]

Working on loading the nodes and edges for a graph. I am able to increase the limit to 197, but the actual node/edge total of 198 returns a timeout error.

If I remove the limit parameter, the node limit returned defaults to 10.

Code for the graphQl query:

   const response = await api().post('multinet/graphql', {query: `query {
        graphs (workspace: "${this.workspace}", name: "${this.graph}") {
          nodeTypes {
            name
          }
          edgeTypes {
            name
          }
          nodes {
            total
            data (offset: ${this.offset} limit: ${this.limit}) {
              key
            }
          }
          edges {
            total
            data (offset: ${this.offset} limit: ${this.limit}) {
              key
              source {
                key
              }
              target{
                key
              }
            }
          }
        }
      }`});

Girder log with error:

Executing GraphQL Request
request: query {
        graphs (workspace: "doesItWork", name: "test") {
          nodeTypes {
            name
          }
          edgeTypes {
            name
          }
          nodes {
            total
            data (offset: 0 limit: 198) {
              key
            }
          }
          edges {
            total
            data (offset: 0 limit: 198) {
              key
              source {
                key
              }
              target{
                key
              }
            }
          }
        }
      }
An error occurred while resolving field Node.key
Traceback (most recent call last):
  File "/Users/jen/.local/share/virtualenvs/multinet-girder-NQjY1GFC/lib/python3.7/site-packages/graphql/execution/executor.py", line 450, in resolve_or_error
    return executor.execute(resolve_fn, source, info, **args)
  File "/Users/jen/.local/share/virtualenvs/multinet-girder-NQjY1GFC/lib/python3.7/site-packages/graphql/execution/executors/sync.py", line 16, in execute
    return fn(*args, **kwargs)
  File "/Users/jen/Documents/Github/multinet-girder/multinet/multinet/resolvers/entity.py", line 38, in <lambda>
    fields['key'].resolver = lambda node, *_: node.data['_id']
TypeError: 'NoneType' object is not subscriptable
Errors in request: 1
'NoneType' object is not subscriptable
127.0.0.1 - - [20/Jun/2019:10:48:26] "POST /api/v1/multinet/graphql HTTP/1.1" 200 68 "http://localhost:8080/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_14_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.90 Safari/537.36"

[waxlamp]

How long does the request for 197 take?

Part of the reason the offset and limit parameters are there is to enable splitting up a request for a lot of data into several requests for less data. The general way to solve this issue is to convert the "big" request into a "small" request that runs in a loop, aggregating the results as they come in. Let me know if that will be possible for your case.

[jrogerthat]

How long does the request for 197 take?

Part of the reason the offset and limit parameters are there is to enable splitting up a request for a lot of data into several requests for less data. The general way to solve this issue is to convert the "big" request into a "small" request that runs in a loop, aggregating the results as they come in. Let me know if that will be possible for your case.

Hey Roni! So the request for 197 does not take that long- Maybe a couple of seconds. I am suspecting there is something else going on here - as it runs fine as long as I request n-1 nodes. If I request any number less than 198 it runs fine, but with 198 it times out.

[waxlamp]

After some investigation, it turns out the problem here was an edge referring to a non-existent node. When that node is "retrieved" from the database, the data on it is null, which leads to the problem.

Since graph validation would have exposed the problem, I'm closing this issue in favor of #71.