File watching architecture in arg type parsing

[tmeasday]

Let's consider the architecture of file watching when parsing a component for (arg) types, using react-docgen and similar in the metadata server.

Consider the fairly typically use case that looks something like (=> is a dependency):

Button.stories.ts => Button.ts => types.ts

In order to understand the arg types of the Button stories, we need to understand the prop types of the Button component, which requires understanding types.ts. Similarly, if types.ts changes, we need to invalidate/recalculate the types for the Button stories.

Currently we parse the arg types for a set of stories via webpack loaders that run as part of the main webpack process. Webpack tracks the dependencies. When the relevant files change, both the component is HMR-ed and the types are regenerated via the loader and the SB store is updated via an HMR mechanism also.

---

When we migrate to generating the types in a seperate process to webpack, we will be faced with the problem of both webpack and the metadata server needing to watch essentially the same set of files for any given component.

There are two problems with this:

a) the task of figuring out which files are required to be watched is non-trival, especially when you consider all the different file types that could be involved across React, Angular, Vue and all other frameworks we support. The question of which of a files imports are relevant to it's types is not a question we can answer in a general way.

b) there is redundancy in both the builder (i.e. webpack) and the metadata server watching the same files.

We've discussed a couple of ways this could be addressed:

1. Don't watch any files in the metadata server, update from browser.

When the builder (i.e. webpack) sees a change, it will HMR. Detect this in the browser, and trigger an invalidation of the metadata, causing another request to the metadata server, which will then recalculate the types.

The challenge here is:

a) are we sure the builder is watching all the right files? (it may ignore some TS files, if they aren't necessary for the JS build) b) can we actually detect the HMR in the browser if something like fast refresh is involved?

2. Don't watch any files in the metadata server, update from builder.

We could make it part of the specification of a "storybook builder" is to emit a node-side event when a component or any of its dependencies change. We can that use that event to tell the metadata server to invalidate+recalculate metadata for the component.

I'm not sure how easy or difficult it is to do that. I suspect a reasonably simple webpack plugin could tell you when any of the dependencies of a CSF file have changed.

3. Watch files in the metadata server.

Part of the specification of a "type generation" addon (we will end up needing ones for React, Vue, Angular etc, although prior art exists) would be in telling the metadata server a list of files that any given component file depends on. We then watch those files, even though this is duplicative with what webpack (or other builder) is doing. Perhaps there is some way to share a pool of watchers as an optimization.

What do you think about the three approaches @bebraw?

[bebraw]

When it comes to this particular problem, I wonder what kind of information we can get from TypeScript exactly. For this specific use case, maybe that's the solution space to explore. If TS can tell us metadata has changed, it would solve the problem for all without webpack. The bonus here is that we could end up with a solution that works with any bundler or future solution since we would be isolated to our own server.

The issue with 1. is that all we know is that change happened but we would still have to know if it's affecting a type somewhere so we would either have to reprocess all types (feels wasteful) or somehow maintain a graph of type references (this is what TS is doing as far as I understand). In latter case we're looking at TypeScript and its capabilities.

If we can push the problem of watching and dealing with types to TypeScript, then it could combine with the metadata server nicely as it could listen to the TS bit and then relay messages to the frontend that will then pick up the changes. It feels like a good direction to explore and it's maybe worth it to see how IDEs like VS Code deal with the problem as they have a similar challenge there.

[tmeasday]

So you mean to run TypeScript in watch mode and piggyback off its file watcher? I think this could be an interested direction to investigate. @shilman do you have thoughts?

[bebraw]

So you mean to run TypeScript in watch mode and piggyback off its file watcher? I think this could be an interested direction to investigate.

Yeah, most likely there's a programmatic API for that.