open-source-ideas / ideas

💡 Looking for inspiration for your next open source project? Or perhaps you've got a brilliant idea you can't wait to share with others? Open Source Ideas is a community built specifically for this! 👋
6.52k stars 222 forks source link

WebGL Interactive 3D/2D Builder w/ 2way Python Bindings 🎨🖌️🖥️ #309

Open fire17 opened 2 years ago

fire17 commented 2 years ago

Project description

The goal is to create an online interactive visualizer. Ideally, one could create a python object that is rendered in 3D scene in the browser. The client-server object is bounded, any changes in the vis will update the server, changes in the server should be rendered in the browser in realtime.

This way we can have simple, dynamic visualizations, for data or content.

Anybody knows 3Blue1Brown and heard of manim (math animations) the tool he is using to create his videos, think that, but actually interactive. Instead of a video we can make and load interactive models , have the client change parameters and see their effects in realtime.

If 2D can be overlayed you could essentially draw up any ui and make wix-style website/app builder and make literally anything.

The fundamentals to make these work are:

  1. proper realtime 2way bindings between server and client
  2. proper interactive visual scene with raycasting, mouse hover object detection, "free transform" object mouse picking, etc.

Relevant Technology

Python - server should be a simple flask server Sockets - keep a client-server session to pass data between them, socket.io is classic JS visual 3D frameworks such as 3js or babylon.js , JS 2D Vue or Svelt Frontend hosting - hopefully the site could be psuso-static page and passible as a github-page. If not then Heroku or Vercel or Hostinger/self-hosted Backend - probably ngnix that leads to flask-socket-io

Later on I'll post comments with relevant projects that have partly similar technologies . Seen some cool interactive web 3d visualizations that feature mouse interactions and 2d overlays, and seen some dynamicly binded variables using sockets, but not both.

This could be really great because after building the infrastructure, it should be really easy to spawn UI or 3D objects from python code, and be very easy to visually create objects and have them be saved to the server. The "skin" or UI template could be changed anytime , but the structure of the data will remain.

Complexity and required time

Complexity

Required time (ETA)

Categories

fire17 commented 2 years ago

FreeCAD does have similarities interms of the interactivity, and pretty sure that it has python bindings (if not blender has and is also an alternative), but having it run on the browser is pretty fundimental, cause then it would be seamless to save and share interactive scenes with someone without any special software installed, neither for the creator nor the viewer/user

Bringing dynamic visualizations to the web could spawn many useful foss applications , such as a modern free website/app builder, online code interpreter and ide, interactive presentations , realtime data visualization dashboard , virtual no-pc operating system and so on...

As stated before, there are plenty of interactive visualisations, but not so many web interactive ones , and have not found an example for one that has 2way realtime bindings to a server. If you can find a working foss example of that, in any stack (like node perhaps), please reference it as it might be possible to create python bindings for it

fire17 commented 2 years ago

There's this video: Build the 3D World in React with ThreeJS s and React Three Fiber https://youtu.be/9ZEjSxDRIik

Similar to what he is doing , can you make it so that clicking the box sends an update to the server, and having the server being able at any time to simulate calling the function as if the user clicked it

Maybe the best technology that is similar is some sort of multiplayer .io game. That is focused on 3D/2D creation tools like FreeCAD.

fire17 commented 2 years ago

These seem super useful: React Three Fiber Fundamentals https://youtu.be/VNd8n566R48

Dashing Off A Dashboard with Python and React https://youtu.be/wirjC_Zp2ZY

fire17 commented 2 years ago

@KOLANICH cool to know But I need a python3d-browser solution If anyone has a reference to a working template of a 3d js env with working with websockets , please share it here with us🙏