Proof of concept brain dumps

[itsdouges]

Hey folks! I thought it would be great to align on some initial architecture before diving into code in this repo, and an easy way to get started could be to talk over the spikes we've done and what's worked/what hasn't worked, with some thoughts for completeness.

I'll make a reply with mine 😄. It's getting late so might not reply until the morning.

cc @nksaraf

[DefinitelyMaybe]

OMGosh yeah! I've been thinking about remaking the three.js editor too!

I hadn't even got to the point of thinking, oh how about I just write values back to source, neat!

Umm not sure if this video helps or not but it's from one of the devs working on Flux talking about issues they ran into jumping into 3D web dev

[nksaraf]

Yeah this video was incredibly insightful and we should incorporate learnings back

[DennisSmolek]

One thing is there’s been some previous work here: https://github.com/AndrewPrifer/react-three-editable

That’s become the r3f extension of: https://www.theatrejs.com/

[Creative-Ataraxia]

this is a game changer; would love to contribute to this project

[itsdouges]

OK so let me dig into the proof of concept I posted on Twitter. You can see the source in a gist here.

It uses a few tools:

• ts-morph for AST manipulation and extracting type information
• Deno for running a web server

And implements a few features to test some things out:

• Scene-to-source transparent wiring up
• Get scene objects via api (for a jsx file, return all components in it)
• Get scene object props & types via api
• Update scene object (persist to in-memory ts-morph source file)
• Save (persist ts-morph source file to file system)

Scene-to-source transparent wiring up

It transforms the code using ts-morph and then outputs it into a temp file for the editor to import. Each jsx element is wrapped in a group if it's considered to be a scene object, currently the logic is very simple.

Once the editor has the transformed code it's a matter of traversing up the threejs scene tree on click, finding a parent that has the __r3fEditor data. We make an educated guess as to what the scene object we should use for the transform.

I find scoping it to a single file at any point in time works well to understand what the boundaries are in the scene (file). If everything in the (threejs) scene is available to be edited it would be very difficult to understand what you're modifying. I find component boundaries to be simple & sensible. When wanting to edit a child component it's a matter of focusing on its file. Keen to align on this point because we're sitting at opposite ends atm.

🔒 Uses data on the frontend initial render to allows a scene object to be transformed (or not) ✅ Both custom and native components can be wired up 🤔 The wiring up logic might not be perfect and there are some heuristics used to guess what the right scene element is 📐 Simple method to enable transforms ✋ Keeps logical component boundaries for scenes 😬 The temp file logic is kind of whack and would be better suited to a bundler

Get scene objects via api

The api returns a list of components in the file, returns names and their position in the source code. Might need to pivot from the position in source as I found bugs where you couldn't modify multiple objects at once sometimes. Need to dig into it.

You would then be able to use the response to populate UI when appropriate.

👀 Components don't need to be rendered into the React tree to be visible in the editor ⌛ Loading scene objects for editor UI is outside of the critical render pass of the scene

Get scene object props & types via api

The api returns props and types on a jsx element at a position location. Found this very useful as you can load this data only when needed, which currently would be when focusing on the object. Types from ts-morph elevate the editor ceiling to the very useful area imo as you can know what props are available even if they aren't defined yet.

You can then populate the UI with this data when appropriate.

📖 Has knowledge of all available props, not just ones currently defined ⌛ Loading an objects data for editor UI is outside of the critical render pass of the scene 🤔 Extracting types is a bit fiddly (but it can be done! And there are contributors who'd want to help like Travis Arnold)

Update scene object/save api

The api would take params and update the ts-morph in memory source file. You could update it multiple times until you're happy. It wouldn't actually mutate the fs source file until saving.

The save api is here, it isn't anything special just saves via its own api. One down side of using ts-morph mutations in memory is that if you change the source file on disk you'd need to refresh the in memory source using the refreshFromFileSystemSync function. Code for that can be found here.

🤝 Can batch up mutations without us needing to implement it 💾 Can save at some point in the future 🤔 In memory source file is a bit fiddly when modifying the fs source file

---

Based on my spike I'm:

• pretty keen on using ts-morph for mutations and type extraction
• pretty keen on keeping to a single file for the scene editor unless there is a compelling reason not to... I can't currently see how it could be done in a way that isn't confusing and accidentally let consumers modify something they wouldn't want to but happy to be convinced
• pretty keen on considering any file with a component (that returns threejs elements) to be a "scene" that can be edited
• flexible sticking to a bundler plugin for mvp, but I think we should evaluate what the editor should be in the long term (standalone app, bundler plugin, similar to storybook, etc) soon. For effective adoption I don't think we'll reach far enough if we scope it to bundler plugins (but, again happy to be convinced).
• strongly say we should bundle the large majority of dependencies (esp. UI!) regardless of the direction we take so we don't pollute consumer dependencies with ours.

As for MVP scope I think focusing on the scene-to-source wiring up and the editor transformations on scene objects (translate, scale, rotate) would be great, ignoring all UI for now until that part is in a usable state. But depending how we divvy things up to work on it might make sense to do some UI stuff (I can go either way).

[itsdouges]

I've dug into your spike @nksaraf - can you go into a bit of the architecture, pros/cons/likes/dislikes? Does it only work for base primitives (e.g. group/mesh/etc)?

I find the babel transform injecting the UI into each module a bit heavy vs. exposing the data instead via APIs and loading it outside of the critical render path/scene tree. Was the editable component inspired from TheatreJS?

Keen to get aligned before jumping in, I think it's worthwhile talking about the general architecture before going much further. Let me know if that works for you 🙏.

[nksaraf]

Heyy thanks for your notes! Have been sick for the last two days so not too responsive.

But my thinking is the backend infra (scene to source and loading component graph) is not the first priority. I think the editor itself is the main problem to solve in a way that's minimal and intuitive. I would still say we should be handle whole scenes.. I was playing with a demo wheee it would be useless to do just one file.. but whole scene or single file is a very tiny difference and just user preference I believe .. we can support both easily. My main concern is the editor runtime, extensibility and UI.. the backend is very swappable based on if we get constrained because of any choice.. we know clearly what we need from our backend: annotate the scene jsx and write props back.. whether that's Babel or ts morph.. kinda irrelevant and super low implementation detail ..

It also helps with the momentum of the project to focus on the frontend so we can start putting stuff in front of people and get feedback about Ui UX concerns .. deciding on an architecture before we know the extent of the needs could be a mistake

[nksaraf]

Also my PoC doesn't just work with promitives .. works with all components.. right now we only mark those as editable that have transform related props. The inspiration for the name was yheatrejs but for us it's working invisibly. User doesn't need to do anything about it

I think the biggest pro for me has been iteration speed on features which is incredibly valuable. I use recast for writing props so we don't change the structure of the users code.. like all the Jsx elements remain on the same position as before the edit.. that way we can know without HMR that all the positions we have are still correct.. don't neeed to keep refetchint. Also uses a vite based websocket RPc to talk to the backend so that we might be able to do super real time live editing.. like the user drags a slider and the text in the code is changing live.. not sure if this is good yet but definitely a cool demo

[itsdouges]

Hope you're feeling better 😄

But my thinking is the backend infra (scene to source and loading component graph) is not the first priority. I think the editor itself is the main problem to solve in a way that's minimal and intuitive.

🤔 if I had to order my preference of features IMO they are:

1. Wiring up scene-to-souce (for a given scene object, be able to tie it back to a React component)
2. Scene object transformations (translate/scale/rotate)
3. Persisting scene updates to code
4. Improvements to object transformations (snap to grid, disallow intersections, etc)
5. Ability to transparently show scene helpers (like cameras, lines, hidden objects, etc)
6. Polish polish polish

None of the above needs any UI except for (5) which would just be a checkbox... I'd then say:

7. Initial UI MVP (tree of elements, read only data)
8. Edit UI MVP
9. ... more here ...

For context I order them this way because I'm interested in making it easier for the game I'm making when it comes to creating scenes (levels). It's all custom components in a scene!

I see an editor similar to scenes in Unity being very useful for me, rather than a 3D editor such as Spline/Blender. For me there two primary use cases for it:

1. Editing a scene (level) that has a load of custom components in it, maybe some primtive elements, but I'm only interested in macro changes, moving around scene objects that are something! Such as a tree, or a player, or a NPC, item. Each could be made up of individual primitives, but it's irrelevant in this mode.
2. Focusing on a custom component (maybe I'm making a shader, or texturing it, or anything, and I want to focus on a single component, think storybook for r3f basically).

Being able to switch between both "modes" instantly would be very desirable. But combining both? I'm not so sure...

Maybe we're philosophically at two ends of the spectrum when it comes to what an editor should be? Which is why your spike is "bottom up" and transforms every jsx element to editable components + baking the UI into them, and mine is "top down" only transforming what is considered to be the open scene + the editor owning the rendering. 🤔

[nksaraf]

I agree with the priorities there but I don't think they need a lot of work beyond what either of us have working. In some ways its quite a simple mechanism. the UX, example, scene transformation, and snap to grid and stuff.. this is kind of stuff im talking about when I mean UI. And ofcourse the tree and panels.

Also unity is also my inspiration. If you see there you will see your top level entities there but also all the children are behind the toggle so you can expose them and edit them individually.. also double click for focus. Never liked storybook lol.

But I think combining or in a way seamless navigating between editing the app vs focused on one element is very natural in these kinds of apps. Coz the editing experience is pretty similar.

I have wondered whether the editor should be responsible for rendering. I was thinking about exporting a Canvas component to swap instead of the default one which handles play/pause mode.

[nksaraf]

One of the reasons I don't want to limit to single files is that it puts in too much opinion on how you structure things. i would rather the tool not dictate how I organize things. Imagine you are using a router for different scenes. You would anyway be showing only a single scene at a time. And then the editor only show what the current rendered scene has. Be it one file or multiple.

[itsdouges]

Why do you think it would impose a structure/organisation on consumers?

[nksaraf]

because if you dont write your scene a certain way those components/entities don't show up in the editor and you have to move between files that represent a common scene to change things that belong in the same context

[nksaraf]

For a world of r3f apps where everybody writes their scenes as single files of imported components, this works.. but as trying to address the variety of types of scens that are authored, from just landing pages, to games. I think there will be enough abstractions here that making a single file version would be trivial. Actually it already is, I can make a version in an hour or so

[mattblackdev]

How about we setup the repo and aim for these 3 initial points:

• Scene-to-souce (for a given scene object, be able to tie it back to a React component)
• Camera fly controls & object picking
• Object transformations (move/scale/rotate) with gizmos

I am also building a game and would really appreciate a tool like this.

[itsdouges]

No worries @nksaraf looks like you're set for the direction you want to take so I'll leave you to it.

[nksaraf]

Trying to take things forward, but I do have the single file scene use case in my head and an idea of how to implement it!

[itsdouges]

No worries, I'm continuing what I've described in this issue either way.

https://twitter.com/itsdouges/status/1601391250724163591

[itsdouges]

All the best folks