search

ThomasGorisse / realitycore

Render 3D content in your Augmented Reality Android apps
Apache License 2.0
25 stars 4 forks source link

wow, is this Repo the one you mentioned before? #1

Open issacclee opened 3 years ago

issacclee commented 3 years ago

Hi Thomas! Is this repo the realitykit inspired AR framework with filament as its rendering core which you mentioned before? Excited!

ThomasGorisse commented 3 years ago

Maybe ;-)

Mostly everything is ready but I will try to commit only verified and completely documented files during coming days instead of pushing all and making corrections after.

issacclee commented 3 years ago

Very cool! Am I right to assume the repo is ready to be pulled and inspect upon? Have you committed most of the code that you intended to commit yet?

ThomasGorisse commented 3 years ago

Hi @issacclee,

Actually there is only about 10% of the already done work pushed to the repo. I tried to follow the IO and participate as much as I can on the AMA. Unfortunately the only answer to more than 50% of the questions : "Where is Sceneform ?" was "You don't need Sceneform, just do the same with OpenGL". Not even a single world about the work done by the community to maintain it or about the possible Filament usage with ARCore.

Anyway, even if we are only 2 of us using this new SDK, I'll push the rest of the code in coming days because I know all the capabilities an ARCore/Filament framework can bring.

Thanks for your support.

issacclee commented 3 years ago

Hi @ThomasGorisse , As always, I greatly appreciate your work and I'm sure many developers around the world who benefited from your work on keeping up the Sceneform fork feel so as well.

Recently I've being communicate with a lot of enterprise customers from Asia(China mostly). They all agree in principle that they would like to integrate basic 3d/ar interaction in their product(e-commerce app etc), but they are reluctant to package something like Unity(UAAL) into their existing app code base. While on the apple side they are happy to use realitykit to make such experience happen, however android devices makes up about 80% of market share. So there's definitely a need for project like this to exist, even though it's relatively niche at the moment.

issacclee commented 3 years ago

BTW, I'll spread the word around Chinese developer community and maybe do some document translation when you've finish the initial commits of the code that you intended to.

ThomasGorisse commented 3 years ago

I hope this repo will be useful.

I have discovered so much enjoying things I wouldn't expect doing it that it would be a shame not to share it. Like the idea of considering the real world returned by the Depth API as a simple Filament 3D shape with UVs in the Scene. I went from "That's a crazy concept, it will never be reliable but let's try it" to "OMG, that's actually working well, I can now place objects behind my sofa!!!" Or great discussions with the Filament team about using the returned Environment Lighting as a Skybox (you can see what it will look like here : https://github.com/google/filament/discussions/3607

Unity is great and can do a lot of things. But, as you mentioned, a lot of clients just want to only integrate AR in their actual App. Unity is not lite to integrate and cannot really interact with the rest of the app and the Model viewer Intent is not enough. In my point of view, Unity is just like Flash at its beginning. Great and beautiful but when comes the time to know "What can be the concrete usage?", I think, just like for Flash, that the answer will be "Mostly Games" since it's not completely integrated in the Android app ecosystem.

All that to say thanks a lot for bringing me back the desire to finish pushing after the not encouraging ARCore IO AMA discussions.

bounty-torn commented 3 years ago

"Where is Sceneform ?" was "You don't need Sceneform, just do the same with OpenGL".

I was happy they addressed my question during the I/O AMA but I was also disappointed with the response. I'm so glad that you are working on this implementation. I am a noob developer, trying my hardest to keep up, but I am happy to help in any way I can. I will surely have questions as this project matures. Thanks for your work on this, @ThomasGorisse!

grassydragon commented 3 years ago

Hi, @ThomasGorisse! Do you still need help with the materials?

ThomasGorisse commented 3 years ago

Hi @grassydragon,

Yep, more than ever. Actually, I don't completely know if we should do the things in Sceneform or RealityCore. I spend a lot of time making corrections and evolutions on Sceneform and I let RealityCore a little bit down this days.

I really think this repo could be awesome and a great move for AR on Android but there is so much need of users on Sceneform that I don't know where to go.

I was actually at the point of moving Sceneform material classes to Kotlin.

What do you guys think?

@RGregat would you also like to participate here?

grassydragon commented 3 years ago

I was actually at the point of moving Sceneform material classes to Kotlin.

Do you mean in Sceneform or moving them here?

ThomasGorisse commented 3 years ago

In Sceneform. Cause there are actually 4 classes with an enormous count of boilerplate/unused code wich could be cleaned in 50 lines of code thanks to Kotlin extensions.

RGregat commented 3 years ago

@ThomasGorisse Count me in, but except for a few Udemy courses I never programmed in Kotlin. My product codebase is completly written in java and as a solo developer I have currently no time to migrate. On the other hand, is this repo a good opportunity to get used to Kotlin :) And I like the ECS idea. I used something ECSish for a frontend map view and I was very pleased with the result.

ThomasGorisse commented 3 years ago

@RGregat

Count me in, but except for a few Udemy courses I never programmed in Kotlin.

Thanks!!!! You already do a very good job on Sceneform, it will be cool to have you in here.

My product codebase is completly written in java and as a solo developer I have currently no time to migrate.

In a way, I envy you for not knowing the wonderful world of Kotlin yet. You will see how much it can change your developer life and maybe will be sick when you have to go back to Java for some projects.

On the other hand, is this repo a good opportunity to get used to Kotlin :)

I also think so and I will guide you within the PR for making things more clean and Kotlin certified for you.

And I like the ECS idea. I used something ECSish for a frontend map view and I was very pleased with the result.

I was first a little bit doubtful when I saw the ECS. Like "OK, here comes some new conceptual think made for people who wants to spend hours making code worst at the end". But it is completely made for case like in AR where many calculations are made at a same time by different actors.

Quick example:

val modelEntity = loadGlb("model.gmb")) //You have access to the model even if the gltf is not completely loaded yet
modelEntity.material["wheels"] = loadMaterial("material.filamat") //Behind it only the entity id is passed to Filament so it can handle the fact that the model isn't completely loaded yet and wait for only the loaded parts of the model it needs to apply material
model.material["wheels"].externalTexture = loadTexture("texture.png") //Same thing, even if the material is not completely loaded we can tell Filament to apply the texture to the material entity id when things are ready for it.
scene.addEntity(model) //The entity is added and will be rendered progressively
model.position = Position(0.0,1.0,0)

The only things that must be done or not (that's a subject I'm thinking about this times) in a worker thread (not in the UI one) is the file InputStream to Buffer.

Comparing to Sceneform case, if any component (Renderable, Material, Texture, Anchoring) takes a long time to load there will have no impact on over calls and we can use not already loaded things since there are just passed to Filament as an Entity Id (= an int)

RGregat commented 3 years ago

That sounds indeed really good. Of course I have a few things I would like to see in realitycore:

:)

grassydragon commented 3 years ago

I implemented point cloud rendering in my project using spheres with a transparent unlit material. I add and remove the Filament entities from the scene dynamically and reuse the previously created entities.

ThomasGorisse commented 3 years ago

Wow. So much infos at the same time !!!

My brain is going to explode with all this things to think about at the same time.

So...

  1. @RGregat Thanks for the list

    It's good to have a result scope in mind when implementing what they are made of. Mostly everything seems to be realistic.

    For the high level usage part (StrokeRenderable, predefined materials,...), @SimonMarquis is doing an awesome work here : AR-Toolbox, which made me definitely think that we can do mostly everything that Unity can from an ARCore-Android/Filament SDK when I started maintaining Sceneform. And which could be included or linked to this repo for simplifying/demonstrating capabilities.

  2. BTW @SimonMarquis, if you see this comment, why not moving your dependency to Sceneform SDK for Android - Maintained?

    If you need help for migrating, we are here.

  3. @grassydragon You mean without using Sceneform?

  4. @ThomasGorisse ;-) Your way to implement the Filament extensions in a Coroutine context looks good

    Material.kt

    
suspend fun AssetManager.loadMaterial(
fileName: String,
apply: Material.Builder.() -> Unit = {}
): Material = loadAsset(fileName, create = { buffer ->
Material.Builder().payload(buffer, buffer.remaining())
    .apply(apply)
    .build(filamentEngine)
}, destroy = { it?.destroy() })

fun Material.destroy() { filamentEngine.destroyMaterial(this) }


   *Io.kt*

suspend fun AssetManager.loadAsset(filename: String, create: suspend (ByteBuffer) -> R, destroy: ((R?) -> Unit)?): R = withContext(Dispatchers.IO) { val asset: R? =null try { open(filename).use { inputStream -> val bytes = ByteArray(inputStream.available()) inputStream.read(bytes) val byteBuffer = ByteBuffer.wrap(bytes) withContext(Dispatchers.Main) { create(byteBuffer) } } } finally { destroy?.invoke(asset) } }

   and linking all the Filament loaders to the view lifecycle scope like this is a good idea since it will remove the ARFragment needing

   *ARView.kt*

val lifecycleScope: Flow = callbackFlow { val listener = object : OnAttachStateChangeListener { override fun onViewAttachedToWindow(v: View?) { removeOnAttachStateChangeListener(this) offer(findViewTreeLifecycleOwner()!!.lifecycleScope) }

    override fun onViewDetachedFromWindow(v: View?) {
    }
}
addOnAttachStateChangeListener(listener)
awaitClose { removeOnAttachStateChangeListener(listener) }

}

init { if (!context.isOpenGlVersionSupported(Filament.MIN_OPENGL_VERSION)) { throw OpenGLVersionNotSupported } MainScope().launch { lifecycleScope.collect { lifecycleScope -> lifecycleScope.launchWhenResumed { try { session.resume() renderer.start()

                awaitCancellation()
            } finally {
                renderer.destroy()
                session.pause()
            }
        }
    }
}

}


   but what do you guys think about it because it's a great idea to push it soon ?
ThomasGorisse commented 3 years ago

Note : fun View.findViewTreeLifecycleOwner() from androidx.lifecycle is a great add for managing all from the ARView instead of an ARFragment and as an SDK user that can be really great to just have to add the ARView to the layout Throwing exception if OpenGL version is too low since it's a manifest requirement but not for the ARCore compatibility = Simple 3D view if no ARCore.

How to handle the same usage without hit test or augmented image or augmented face (AugmentedFaceNode PR will be merged soon to Sceneform) is another question that should, maybe, be handled by the user.

grassydragon commented 3 years ago

@grassydragon You mean without using Sceneform?

Yes, the code only depends on ARCore and Filament but still Sceneform is used in the project for the AR initialization and ViewRenderables.

ThomasGorisse commented 3 years ago

@RGregat and @grassydragon I added you to the sponsored usernames. If you enable sponsoring on your account it will be linked to this repo and the Sceneform one.

ThomasGorisse commented 3 years ago

In fact, you need to create your sponsor account (If you want so) before I can add you