search

Yellow-Dog-Man / Resonite-Issues

Issue repository for Resonite.
https://resonite.com
141 stars 2 forks source link

Create blender import/export plugin for resonite packages #2335

Open gentlecolts opened 5 months ago

gentlecolts commented 5 months ago

Describe the Underlying Issue

It would be extremely convenient if we had a means for seamlessly moving content to and from blender.

To Reproduce

No response

Expected Behavior

There are obvious cases where blender scenes and Resonite cannot be 1:1, however models, rigs, and lighting (minus area lights) could be, and materials could still make some best effort (at least for PBR materials), all in all producing an experience that'd be better than attempting to do the same with intermediary format

Screenshots / Video

No response

Additional Context

No response

Reporters

zangooseoo

Frooxius commented 5 months ago

We're very unlikely to implement something specifically like this. Resonite Packages are designed for storing already existing content in Resonite, but it's not designed as an interchange format or format to be generated outside of Resonite.

Implementing that would be a significant engineering task - we'd need to implement a large chunk of Resonite's codebase for Blender to support this.

What would be more viable though is something that'd be more similar to the Unity SDK - https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/6 - a Blender SDK of sorts.

It's still a big chunk of work, but on technological level, this would be a lot more feasible approach. However given the size of this task, we'd need to see significant interest in this before we'd commit to implementing it. We're more likely to prioritize Unity SDK first too.

However if we take approach with the on-network "DOM" style interoperability for the Unity SDK, this would make a Blender SDK simpler to implement, as it would be built on the same mechanism - and even if we don't implement it ourselves, it'd be much easier for the community to do so.

gentlecolts commented 5 months ago

Oh yeah, I for sure expect this to be a big ask, and definitely something for the long-term.

we'd need to implement a large chunk of Resonite's codebase

If it's possible to split out the relevant portions, Blender plugins are python, and python can interop with c#, and some quick googling brings up the project pythonnet, which is specifically built for this purpose without having to involve ctypes. If I'm not mistaken, and forgive my ignorance on the technical details, but the upcoming performance changes are going to lead to some amount of restructuring the project anyways right? Might be worth keeping on back of mind, especially if there's plans for other forms of file format interop.

it'd be much easier for the community to do so.

This was honestly something I kind of expected the community to be working on the moment I heard resonite package was a thing, I already know there's existing efforts to try and get some unitypackage interop, and at some point I had floated the idea with friends of having a plugin that syncs a blender scene with a live reso session. Anything that further enables community solutions for this seems like a great step forward.

FlameSoulis commented 5 months ago

So.... if memory is serving me well, doesn't the Resonite Package also offer a non-varient version on export? If so, then that would be a version of a package that is just the base files and whatever component data is included.

I think what is mostly being asked is "how can one best talk in a way that makes Resonite not hate you" when it comes to making an export option for another system. Overte already does this with their Blender plug-in.

TisFoolish commented 5 months ago

Having a plugin that does the bare minimum for items and avatars I think would be enough.

EG for an avatar it lets you pick what type of material meshes use, assign the relevant colors and textures, place where the hands, grabbers, tool tip slot, tool shelf, and camera would be, and then on import into res the import process automatically sets up everything. The camera object could behave like a Blender camera in that you can view from its position/orientation.

I realize that the resonitepackage format is not set up for something like this as it's more of a 1:1 container and likely something else would arise instead, but I think this could be a way to minimize the amount of effort needed to make a plugin for exporting from Blender to res while not having to do the usual intermediary steps.

gentlecolts commented 5 months ago

items and avatars

Definitely an important use case, but If I'll be completely honest the use case I had in mind was making it easier to keep vrc and resonite worlds in sync

bdunderscore commented 5 months ago

For keeping VRC and resonite worlds in sync specifically, exporting from Unity is still probably better. This is something I hope to experiment with soon, as part of my Modular Avatar project (the core of the exporter should be extensible to worlds as well, if someone wants to put in the legwork to polish that part up...)

Geenz commented 5 months ago

I feel like for that specific use case, something like #6 would be far more effective here. As at that point, users can simply use our SDK and develop tooling to synchronize between Resonite and other platforms that are Unity based. The reason I say this is because there's a lot more to scenes than just geometry, just materials, just textures, etc. There's a very big functional aspect here as well.

With the methods and mechanisms we'd implement in #6 as a direct consequence of it, we could look into a "blender SDK" at some point similar to, say, how the Sony Santa Monica team has a Maya plugin that just connects to their engine running on a PS5. But that would need to come after #6 as it lays a lot of the ground work for that.

If you just want to export scenes that are more 1:1 with what is represented in blender, then you're asking more for improvements to the import aspect of the pipeline. If you're wanting things to be easier to "round trip" then today you're asking for:

This requires some work to our import and export tooling for assets more broadly so that we can "break up" our pipeline across multiple plugins better than what we're presently able to.

gentlecolts commented 5 months ago

I'm away from my pc at the moment so apologies for the more brief response, but just to get my two cents out. The main reason I'd be in favor of focusing on Blender over Unity is that Unity is a less reliable third party, being proprietary and especially given how they've acted as of late, while blender is free open source software. If I'm not mistaken, the team is also further trying to move away from Unity anyways right? Interop with Blender would give paths towards working with asset pipelines involving other engines, such as Godot or Unreal, and I imagine work done on a blender plugin would more directly translate to plugins for other similar and useful software like Maya. If interop is in fact a goal, it's seems like it'd be harder to make "seamless" connections to other engines, plural, than to focus on making an import/export framework that the community could potentially build on, unless we plan to remain anchored to unity

Frooxius commented 5 months ago

@FlameSoulis The variants don't really matter. The base assets themselves are also heavily Resonite specific types - this includes mesh and data model assets. Those are the problematic bits here.

@TisFoolish There's unfortunately no "bare minimum" like that. The biggest part of this would be implementing support for generating and serializing data model, which is a necessary component. Once that is done, then supporting all components is the easy part. But there's no getting around of having to implement the bigger/harder bits even to support just setting up avatars (not to mention that avatars are fairly complex)

@gentlecolts While I agree that Unity is becoming more unreliable and I like/trust Blender way more over it, the main practical purposes of Unity SDK would be that: 1) Majority of social VR creators are used to its workflow. Having them learn Blender workflow could be an up-hill battle 2) A lot of users want to port their existing content in Unity to Resonite. Blender SDK wouldn't help with that unfortunately.

I think it's also important to mention that Unity SDK doesn't actually anchor us to Unity. It lets people to author/convert content from it to Resonite, but that doesn't require we keep using Unity.

Like I mentioned in my initial post, best approach for something like this would be analogous to Unity SDK in #6, since that will likely be implemented by introducing some sort of "DOM" that lets manipulate the data model externally.

We'd likely implement the Unity SDK with that first (given the interest in that feature and the benefits). We could look into adding Blender SDK as well, unless community takes that on instead in the meanwhile.

TisFoolish commented 5 months ago

The biggest part of this would be implementing support for generating and serializing data model, which is a necessary component. Once that is done, then supporting all components is the easy part.

I don't see why you'd need to do that in blender at all for my suggestion. Everything is just metadata that the engine uses to set up necessary components, slots, and references on import.

Thinking on it what I'm suggesting isn't that far of from the VRM format, just with a bunch of extra features for res.

Frooxius commented 5 months ago

The biggest part of this would be implementing support for generating and serializing data model, which is a necessary component. Once that is done, then supporting all components is the easy part.

I don't see why you'd need to do that in blender at all for my suggestion. Everything is just metadata that the engine uses to set up necessary components, slots, and references on import.

Thinking on it what I'm suggesting isn't that far of from the VRM format, just with a bunch of extra features for res.

I've already answered that in the same post. Resonite Package contains Resonite specific assets - specifically data model. To generate that, you need to implement something that's able to manage and serialize it.

Your assumption that everything is metadata is incorrect - you need to have actual data there too to do something - e.g. to describe the structure of the avatar. Metadata is something that just describes data - if you don't have any data, you can't have metadata.

If you're suggesting VRM + extra, there's already topic for that. But if you need extra data beyond that, you're asking us to introduce yet another custom package format, specifically for Blender, which adds even more complexity to the whole thing and it's not worth doing over what I suggested initially.

aaronfranke commented 3 months ago

Frooxius noted above that Resonite Packages are not intended to be an interchange format and are not intended to be generated outside of Resonite. Another option to solve this problem would be for Resonite to implement importing a format that is designed for interchange. Meaning that a Blender addon could export advanced game-ready objects in this engine-agnostic interchange format, without having any Resonite-specific code, and then Resonite can import this.

OMI has been developing glTF extensions for this purpose. We have extensions for things like physics shapes (boxes, spheres, etc), physics bodies (dynamic, kinematic, mass, velocity, etc), audio emitters (playing MP3 or WAV files globally or at a specific point in the world), seats, spawn points, and more. You can find the repo with OMI's extensions here: https://github.com/omigroup/gltf-extensions/tree/main/extensions/2.0

The glTF extensions are built upon the glTF format as a base, which can be thought of as an open standard replacement for FBX files, while glTF with extensions may one day replace Unity packages, or at least reduce the amount of engine-specific setup needed after importing a 3D model file.

Khronos, the creators of glTF, are also working on extending glTF. The lights extension was mentioned above, allowing for lights to be exported from Blender. Some other interesting extensions worth mentioning are KHR_animation_pointer, allowing for any property to be animated (light color, camera FOV, and more), and KHR_interactivity, allowing for engine-agnostic scripting using behavior graphs (like visual scripting).

Frooxius commented 3 months ago

We could look into adding some of these at some point. Generally open exchange formats are better for this kind of thing.

My only question is what is the adoption and support of these extensions?

aaronfranke commented 3 months ago

@Frooxius Most of these extensions have been implemented in Godot Engine, including in the editor and in platforms based on Godot such as The Mirror and V-Sekai. There is also ongoing work implementing export support in a Blender add-on, but nothing ready for production yet.

I have spoken with one person each from Unreal and Blender, and both of their answers were the same, they are only interested in official built-in support for extensions if they are finalized ratified Khronos extensions. However, this does not stop third-parties from implementing support via plugins.

In general the status quo is everybody making their own format. It is hard to get into contact with platform owners and convince them to adopt an engine-agnostic format, since dominant platforms such as VRChat would prefer lock-in. Let alone convince them to build a format with us in the open. Ultimately I believe the effort will be better spent if we have a pipeline of Resonite, Blender, Godot, etc being able to share content via glTF + OMI/etc extensions, instead of separately implementing pipelines for Blender -> Resonite, Godot -> Resonite, Unity -> Resonite, VRChat -> Resonite, etc (and potentially, the reverse of all of those, and all other possible combinations...).

Frooxius commented 3 months ago

@aaronfranke Ah I see! The Godot Engine part does make it interesting. Is that implemented officially? Or is it a plugin/add-on?

I think we can consider these, especially if it's relatively simple to add. I'd have to poke around it some. One thing we'd also like to do is open up the system for importers/exporters, which would allow for community help with this too.

aaronfranke commented 3 months ago

Officially in Godot Engine:

Godot Plugins:

This is an ongoing effort, the glTF format itself is very new (the 2.0 version that everyone uses is only about 7 years old), OMI is only 3 years old, and several of these extensions are not yet finalized.

Also, if there is anything important missing that would be helpful for a Blender/etc -> Resonite pipeline, feel free to propose new extensions. Some extensions that I have in my head that haven't been developed yet are extensions for vehicles and an extension for skybox/environment.

Frooxius commented 3 months ago

Ah sweet, thank you very much for the info!

Would you actually mind creating a separate issue for these extensions? I feel this is sufficiently separate from just Blender specifically.