New Page: Display content of Remix 3D

[JimSEOW]

Microsoft 3D Story Remix is an excellent way to drive imagination and creativity right at the early age in education system, together with classroom Windows 10S, Office 365, oneNote for classroom, Microsoft Team for education.

I believe many developers here would like to contribute by BRINGING Remix3D content to UWP through Windows Template Studio.

=> We need a new page wizard that use e.g. SharpDX (SwapchainPanel+ XAML Interop) to import e.g. Obj or Remix3D file into UWP App through Window Template Studio.

=> With this page wizard, we can also add pseudo AR, by superimpose Remix 3D content with Camera capture background live video.

ALL for the purpose to MOTIVATE and INSPIRE the next generation at EARLY AGE - consistent with Microsoft latest effort to use AI to improve classroom reading skills.

[kburgoyne]

@JimSEOW -- I want to provide you with a little feedback, but I suspect how to go about using Remix3D content in combination with SharpDX is outside the scope of this issues area for reasons I hope my comments will explain. Thus I don't really want to prolong this thread here.

First, the Remix 3D thing seems like a good idea. I haven't looked into the details, but it seems extremely likely that supporting that environment is a layer above where SharpDX operates. Such support (if it doesn't already exist) would be some library or combination of libraries that used SharpDX.

SharpDX already does work with SwapChainPanel as part of a XAML page. That's how I use it for my project under Win10 and Win10M UWP. It was also how I used SharpDX in Win8.1 Universal projects.

My project includes having regular XAML controls (grids, canvases, buttons, etc) overlaying the SwapChainPanel. I use SwapChainPanel to display my 3D virtual world context, and the XAML controls over the top of it to create a "heads up display" (HUD) over the top of the 3D SwapChainPanel viewport. So far it has worked great for both x86 and ARM using both the Debug and Release (Native) compiler builds.

The important thing to remember is SharpDX is really only an interface layer that allows C# to call the C++ DirectX APIs. Thus SharkDX does not provide any functionality beyond what a C++ app would have if it only tried to use the DirectX APIs and nothing else. If the C++ DirectX APIs don't provide some type of functionality, then SharpDX isn't going to provide it either. SharpDX doesn't actually "do" anything itself beyond handling the transporting of data back and forth between C# and the C++ DirectX APIs.

I haven't looked into the developer details of using stuff from the Remix3D environment. I would think the main thing you'd want to research is the 3D file format being used for the models. "Something" is going to have to read those files and translate the data into vertex buffers and other DirectX primitive data structures. Obviously Microsoft is doing that with their code, but you'll want to figure out what tools, libraries, or whatever Microsoft (or somebody else) is providing (or not) for other developers to use.

[JimSEOW]

@kburgoyne SharpDX has been around for 7 years. Now is the chance to apply it where it would matter most and have the highest purpose of inspiring the children at early age.

3D Remix supports the following file formats: FBX, STL [Assimp.NET], and 3MF [ThreeMf for 3D printing format]

3D Remix is the stepping stone for SharpDX to democratize (non-Unity3D) 3D content to .NET UWP - making SharpDX more accessible than before to UWP for Mixed Reality for the vast .NET community.

What need to be done? [1] We now have a working UWP XAML SwanchainPanel interop sample in the public github for DirectX11. We need also DirectX12.

I have read with respect that you have done a lot in checking the compatibility issue of the SharpDX 4.01 with respect to the different insider builds (e.g. 10.0.15063 and higher, for both W10 and W10M (ARM).

=> This discussion is open to anyone who believe EDUCATION is key to killing Fake News and Truth matters and wants to do something that will make a great impact and target education at early age. Furthermore, I believe IT WOULD BE SO COOL that eventually Microsoft is convinced to have something similar to ARkit [World Tracking, Scene Understanding, light contextual adjustment rendering] - available to WinOnARM tablets Q42017 or surface phone with e.g. dual camera 2018?
=> It would inspire .NET 3D developers to imagine how much JOY this would bring to Children, after they have create Story Remix as Video in the classroom, they could take it out running in the garden under sun, fresh air, and nature to bring the Story Remix creation to AR LIVE (with world tracking). With a tablet or phone, they can share with a group of children squeezing behind. Imagine what this will do to their parents standing nearby.

=> THIS IS THE FUTURE, we shall strive for. THE FUTURE where MICROSOFT Mixed REALITY, comes with both hardware (headset e.g. HOLOLENS, Glass (2019)) and software (ARKit like). empowering Children EVEYWHERE, first or third world, squeezing their heads very tight together behind one tiny screen, share their creation in the most affordable way.- outdoor and at the same time make new friends.

=> IF anyone reading this and believe in this VISION. Just DO IT!

[mrlacey]

I think some more information is needed here so we can be clear about what the new page should do and include by default.

• You say it should show content from the remix3d website. But should it be restricted to only content from that website?
• How should it connect and interact with that site? If at all?
• How would it work if offline?
• Is this just about having a page that can display 3D content? If so, what formats?
• Should it be only limited to displaying content? If not, then what else should it include?
• Should a file (or files) be included with the code to display it?
• And what about handling working with files? Does this new page have to know how to save and open files? That's generic functionality which we don't yet handle.

How WTS will work with data from (remote) sources is still up for discussion. Please feel free to contribute to that conversation at: #396

Also, this:

ALL for the purpose to MOTIVATE and INSPIRE the next generation at EARLY AGE - consistent with Microsoft latest effort to use AI to improve classroom reading skills.

That's not the goal of this project though. How does motivating and inspiring the next generation relate to using AI to teach reading? And how does any of that relate to the goals of this project?

[mrlacey]

=> IF anyone reading this and believe in this VISION. Just DO IT!

We take contributions. They're so much more effective for advancing open source software than vague demands for adding functionality and opinions about how you'd like things to be.

[crutkas]

Is the goal to just display a remix open? That is all the page does? How do you think someone could extend / build on it?

I wouldn't go too deep into the details here as well for implementation. Let's nail scenarios

[kburgoyne]

I view this as a very admirable idea, but it seems like an completely separate project to create a library that uses SharpDX below it. Due to time limitations I haven't had the chance to explore Remix3D to the depth I'd love to. (Or Azure to the depth I'd love to, or XYZ to the depth I'd love to...)

I'm going to assume the Remix3D support is not part of DirectX. I certainly wouldn't have advocated complicating DirectX with all that extra logic unless there was an extremely good reason. (Feel free to supply a reference to educate me otherwise.) Rather any existing Remix3D support is something which relies upon DirectX below it.

SharpDX should be viewed as only the "C# DirectX" portion, and thus any Remix3D support would be a separate DLL(s) project that leveraged SharpDX below it.

Can anybody provide a link to a nice concise web page that does a good job of explaining Remix3D from a developer's (as opposed to user's) viewpoint.

Also... are we outside the scope of SharpDX's issues forum with this discussion? I don't want to impede the discussion, but I do want to be respectful of whatever purpose this forum is intended.

[JimSEOW]

We focus FIRST on the JOY, healthy social interaction running in nature {in contrast to many hours non-moving indoor playing computer game} that it would bring to Children - after they have created Remix 3D and Story Remix indoor inside the classroom. Now they get to go out to the garden, flash air nature, to showcase what they have created indoor BUT through gamification to see who are the most creative in finding the right outdoor scene to superimpose what they have created indoor, and share the video or snapshots through social media {to grandma grandpa etc}.

In terms of increasing difficulty (a) Simple AR. Whatever they have created, Remix 3D or a sequence of Remix 3D animation superimpose on Green SCREEN, the children can now go outdoor, play the animation of download Remix 3D in the middle of any live video capture from camera. In other words, the children can bring any downloaded or created Remix 3D content to any live scene and have fun sharing.

[1] For this, A new Page that allow the insertion of Remix 3D content (e.g. FBX format using Assimp.NET), overlap on top of the video stream coming from camera. With button to capture the video, image, for social media sharing.

[2] Slightly more difficult, the content from Story 3D Remix. ( a series of keyframe animated Remix 3D) in front of Green Screen. The NEW PAGE provide the template to import Story 3D remix (not sure what is the format), and allow the Creative edited keyframe series of Story Remix 3D to be overlaid LIVE with video stream from Camera. The template provide button events to capture the video and image snapshort.

I think if we can start with the FIRST ONE, it is already a Good Start.

[JimSEOW]

In term of Non-Unity3D AND XAML-standard, Xamarin Forms, .NET standard 2.0 and .NET Core 2.0 friendly, I could think of two key 3D development environment (that are HoloLens Compatible). Xamarin's UrhoSharp and SharpDX. However, only SharpDX RIGHT NOW will be WinOnARM (native) and W10M (native) ready. UrhoSharp is not available for WinOnARM (native) and W10M (native) - unless Microsoft get those guys to make ARM Urho3D.dll available for UrhoSharp binding to work.

[mrlacey]

This seems all over the place and very off topic.

If you're interested in helping people get started with using 3D models creatively then check out Story Remix.

If you want to include 3D models in your UWP app then take a look at this sample. (Originally for Win8 but the same principle applies for UWP.)

There seems to be a whole shopping list of ideas and requests here much of which is way beyond the scope of this project and some which doesn't even exist yet. I don't see any appropriate, relevant, actionable requests here that match what we're trying to do and the maturity of the project. It seems more like an idea for a single app.

[crutkas]

@JimSEOW and @kburgoyne, you guys are diving into implementation. Which is awesome but we need to nail what the goal of the page is. Implementation means nothing until we figure out the goal of the page and how someone would take it as a base page to make it their own. Please lets stop on implementation until we nail the heart of what is needed.

From what Jim stated, this went from what i saw as

1. Display a 3D object.
2. Do camera based AR
3. Take content from remix Studio. (don't know if this is even possible)

So if we pause and take a step back, these are very different targets and have drastically different outcomes and how the page could / should be done.

I think 1 and 2 are interesting but different pages. For 1, i'm honestly not sure how what good us doing a template for that page is. What is the next step for the broad audience. If it is to make a game, I feel they should just use Unity or Unreal or a different game engine, not WTS.

For 2, i think that is actually an interesting thing but we would have to take a step back and figure out how to implement it. That would be a separate issue from what is listed here.

[JimSEOW]

@mrlacey

This seems all over the place and very off topic.

In opposite, it works well with your proposal for new page to take photo or even video

It seems more like an idea for a single app.

On contrary, it is an extension to your proposal :-) In this case, adding 3D content to XAML (Grid layout with background layer is either photo or live stream video AND foreground layer is SwanchainPanel for XAML interop (either SharpDX or Microsoft Angel)

If you want to include 3D models in your UWP app then take a look at this sample. (Originally for Win8 but the same principle applies for UWP.)

@jimseow We now have a working UWP XAML SwanchainPanel interop sample in the public github for DirectX11 AND Microsoft Angel code contributed by Xamarin with C# template

In summary: All the technologies are available. We just need a NEW PAGE to empower UWP .NET developers to bridge Remix3D indoor use case to outdoor Mixed Reality.

@crutkas Thanks for your support.

=> Let us focus on one use case as you have suggested Do camera based AR

Please give me a few days to sort out the design as you have requested - so we can empower more UWP developers different ways to bring JOY to children from indoor classroom to outdoor.

[crutkas]

I'm closing this issue out as it sounds like you want an camera based AR template. When you're ready to share the what it will do, how it is a solid starting point, please create a new issue.

Key things to stress here is not implementation details right now. Lets nail the scenario.