Why materialize?

[hgwood]

Paths that either were not matched or were matched but did not produce any value are "materialized" into {$type: "atom"} in the response JSON Graph. This is on purpose (see https://github.com/Netflix/falcor-router/issues/40) but it makes the JSON Graph bigger and I'm not sure what the goal is. Can't the client deal with undefined value? The JSON Graph documentation on get does not specify that non existing path must return an atom. On the contrary, it shows an example where the response is undefined so it looks like it is allowed.

See also: http://stackoverflow.com/questions/39355282/falcor-returning-un-wanted-atoms

[limscoder]

Hey, I'm the one who asked the stack overflow question.

Reading the documentation on the atom type it is not obvious to me why that type would represent undefined.

It's also confusing that atom gets placed at the leaf nodes in the jsonGraph, instead of at the first branch that's undefined. For example item.51.Name is an atom in the rendered jsonGraph, even though item.51 is undefined in the virtual jsonGraph. Parsing the rendered jsonGraph makes it seem like the path item.51 is present, but missing the Name property, however the item.51 path is not there at all and has no properties.

I'd like to have a explanation about why it works this way to make sense of it.

[trxcllnt]

@hgwood @limscoder presently routes which produce no data are materialized to empty atoms so the client knows that those values truly are undefined (as opposed just not cached yet).

The JSONGraph merge needs to handle the case that the response may be only a subset of the paths that were requested. If you request for three paths, but the JSONGraph response only has data for two of the paths, the JSONGraph merge knows that there's still one left, and retries the request (but only for the missing path). This process is repeated until we've hit our retry limit, or we've received data for all the paths requested.

So materializing is our way of letting the cache know that it shouldn't build missing paths and re-request for this data.

I believe we have a plan to move away from materializing soon, and only communicate the paths for which no data was returned.

[limscoder]

@trxcllnt thanks for the explanation.

One more question.

If you request for three paths, but the JSONGraph response only has data for two of the paths, the JSONGraph merge knows that there's still one left, and retries the request (but only for the missing path)

What situation causes a "missing path" in the response?

[trxcllnt]

@limscoder if you request for foo['a', 'b', 'c'], and the response is { jsonGraph: { foo: a: '1', b: '2' } }, then we're like, "hey what about foo.c? Let's ask for it again just to be safe."

If foo.c really is undefined (or there isn't a route configured to serve foo.c in the router), we materialize a value in place for it, so the client will just merge atom(undefined) into the cache instead of asking for it again. We use an atom because undefined alone doesn't survive JSON serialization, and null is a valid value that can be returned from calling get.

[limscoder]

@trxcllnt

What I still don't understand is under what conditions the server would return { jsonGraph: { foo: a: '1', b: '2' } }? It appears that if a requested path is not populated, then falcor-router always adds empty atoms to the response like this: { jsonGraph: { foo: a: '1', b: '2' , c: {$type: 'atom'} }. What conditions would cause c to be completely absent from the response?

[trxcllnt]

@limscoder if the router didn't materialize an atom(undefined) for foo.c, the client would think it was missing and ask for it again. This is necessary for a future in which Falcor will support streaming data sources, and each individual response is a subset of the total request.

[limscoder]

if the router didn't materialize an atom(undefined) for foo.c, the client would think it was missing and ask for it again

What I don't understand is how the value could be missing, as it seems like it is always materialized no matter what you do on the server-side.

[trxcllnt]

What I don't understand is how the value could be missing, as it seems like it is always materialized no matter what you do on the server-side.

It would be missing if the router didn't materialize. The OP was, "why does the router materialize?" and the answer is that it's a signal to the client that the data isn't there. I don't know another way to phrase this.

[limscoder]

@trxcllnt

Ok, so the router always materializes, no matter what?

I was confused because it sounds like the client has different behavior for "real undefined" vs materialized "atom undefined", so I thought it was important for the server to express both forms. Presumably, there's some reason why the client would want to retry "real undefined" values, but the server is circumventing this feature by always materializing, and it appears that the client's retry logic will never be executed when using falcor-router as the backend.

Which leaves me with the question: why not just disable retrying on the client instead of materializing undefined values?

Anyway, I have a much better understanding of this now, thank you. Feel free to close the issue.

[hgwood]

@limscoder

why not just disable retrying on the client instead of materializing undefined values?

From https://github.com/Netflix/falcor-router/issues/189#issuecomment-248181616

This is necessary for a future in which Falcor will support streaming data sources, and each individual response is a subset of the total request.

From https://github.com/Netflix/falcor-router/issues/189#issuecomment-247722403

I believe we have a plan to move away from materializing soon, and only communicate the paths for which no data was returned.

Sounds the client is ready to support streaming but the router is not.

Thanks to both of you.