Implementation Change with new Unity Inference System

[Aoost]

Hi Kyle,

I'm wanting to implement some ML in a UAHirise project I'm working on such that I achieve dynamic shifting between one feature and the next on a given plane. I'm wondering if your project still works or requires some new structure given that Unity has released the Unity Inference System to replace TensorFlow Graphs?

Thank you for all your hard work.

Adam

[Aoost]

Hi Kyle,

I'm wanting to implement some ML in a UAHirise project I'm working on such that I achieve dynamic shifting between one feature and the next on a given plane. I'm wondering if your project still works or requires some new structure given that Unity has released the Unity Inference System to replace TensorFlow Graphs?

Thank you for all your hard work.

Adam

Seems like Unity InferenceEngine will take a trained .nn so should I just produce them as described in TensorFlow and use Unity Inference Engine instead of the Unity/TensorFlow bridge repositories?

(sorry if these are stupid questions, very new to TensorFlow)

[pearsonkyle]

Hey,

I think it's really cool you want to do a ML project with Hirise data, had I enough time I would've liked to do the same. Let me know if I can help out on the project at all.

I'm not up to date on the current developments of either platform and things could have changed since I first made this project. But from what I remember the Unity inference engine is a really nice tool and it lets you train your neural networks inside of Unity. This has its pros and cons, the biggest con is that you would have to import all of your HiRise data into a Unity scene and train the algorithm there. Training inside of Unity is more for things like reinforcement learning, in my opinion.

I used TensorFlow Graphs because it allowed me to train a model outside of Unity and then I was able to import that model for later use. I remember in the past it was a pain getting a TensorFlow Graph exported to Unity, so I made/found a function to do that. To be honest, I'm not familiar with .nn files, I would usually use .h5 files. Unity is also very particular with its TensorFlow version. I had to use TensorFlow v1.4 when I exported my models to a graph.

Hope this helps, let me know if you have more questions.

Cheers, Kyle

[Aoost]

Hi Kyle,

I don't know if you've heard of Refik Anadol https://www.instagram.com/p/BxMLkAIgMVU/?utm_source=ig_web_copy_link? He's been a huge inspiration for the project.

What we'd like to achieve is a landscape that morphs dynamically according to musical input, but for a dome projection use case, not a 2D projection/screen. Thinking that ML might be the smoothest way to do this on a generative landscape hence, me checking out your repo.

What kind of handles can you attach to the height generation in latent space once its all set up in unity? Hoping to have time later this evening to build your repo into a project and see for myself, but would be helpful to know from the outset if you think I'm looking at the wrong tool. Not sure how this affects using TensorFlow vs the Inference System.

My experience is with VR production, not really ML so I'm really diving in the deep end here. Appreciate you being so willing to give advice

:)

Best,

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Wed, Jul 3, 2019 at 5:31 PM Kyle A. Pearson notifications@github.com wrote:

Hey,

I think it's really cool you want to do a ML project with Hirise data, had I enough time I would've liked to do the same. Let me know if I can help out on the project at all.

I'm not up to date on the current developments of either platform and things could have changed since I first made this project. But from what I remember the Unity inference engine is a really nice tool and it lets you train your neural networks inside of Unity. This has its pros and cons, the biggest con is that you would have to import all of your HiRise data into a Unity scene and train the algorithm there. Training inside of Unity is more for things like reinforcement learning, in my opinion.

I used TensorFlow Graphs because it allowed me to train a model outside of Unity and then I was able to import that model for later use. I remember in the past it was a pain getting a TensorFlow Graph exported to Unity, so I made/found a function https://github.com/pearsonkyle/Mars-Machina/blob/master/Python/autoencoder.py#L104to do that. To be honest, I'm not familiar with .nn files, I would usually use .h5 files. Unity is also very particular with its TensorFlow version. I had to use TensorFlow v1.4 when I exported my models to a graph.

Hope this helps, let me know if you have more questions.

Cheers, Kyle

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pearsonkyle/Mars-Machina/issues/1?email_source=notifications&email_token=AHDMNHTW3RMLWX36K23LI5TP5TA5FA5CNFSM4H5GBXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZE2PJQ#issuecomment-508143526, or mute the thread https://github.com/notifications/unsubscribe-auth/AHDMNHVBD2WN6IUHXDKFGKTP5TA5FANCNFSM4H5GBXAA .

[Aoost]

http://refikanadol.com/

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Thu, Jul 4, 2019 at 2:03 PM Adam Oosthuizen adam.oosthuizen@gmail.com wrote:

Hi Kyle,

I don't know if you've heard of Refik Anadol https://www.instagram.com/p/BxMLkAIgMVU/?utm_source=ig_web_copy_link? He's been a huge inspiration for the project.

What we'd like to achieve is a landscape that morphs dynamically according to musical input, but for a dome projection use case, not a 2D projection/screen. Thinking that ML might be the smoothest way to do this on a generative landscape hence, me checking out your repo.

What kind of handles can you attach to the height generation in latent space once its all set up in unity? Hoping to have time later this evening to build your repo into a project and see for myself, but would be helpful to know from the outset if you think I'm looking at the wrong tool. Not sure how this affects using TensorFlow vs the Inference System.

My experience is with VR production, not really ML so I'm really diving in the deep end here. Appreciate you being so willing to give advice

:)

Best,

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Wed, Jul 3, 2019 at 5:31 PM Kyle A. Pearson notifications@github.com wrote:

Hey,

I think it's really cool you want to do a ML project with Hirise data, had I enough time I would've liked to do the same. Let me know if I can help out on the project at all.

I'm not up to date on the current developments of either platform and things could have changed since I first made this project. But from what I remember the Unity inference engine is a really nice tool and it lets you train your neural networks inside of Unity. This has its pros and cons, the biggest con is that you would have to import all of your HiRise data into a Unity scene and train the algorithm there. Training inside of Unity is more for things like reinforcement learning, in my opinion.

I used TensorFlow Graphs because it allowed me to train a model outside of Unity and then I was able to import that model for later use. I remember in the past it was a pain getting a TensorFlow Graph exported to Unity, so I made/found a function https://github.com/pearsonkyle/Mars-Machina/blob/master/Python/autoencoder.py#L104to do that. To be honest, I'm not familiar with .nn files, I would usually use .h5 files. Unity is also very particular with its TensorFlow version. I had to use TensorFlow v1.4 when I exported my models to a graph.

Hope this helps, let me know if you have more questions.

Cheers, Kyle

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pearsonkyle/Mars-Machina/issues/1?email_source=notifications&email_token=AHDMNHTW3RMLWX36K23LI5TP5TA5FA5CNFSM4H5GBXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZE2PJQ#issuecomment-508143526, or mute the thread https://github.com/notifications/unsubscribe-auth/AHDMNHVBD2WN6IUHXDKFGKTP5TA5FANCNFSM4H5GBXAA .

[pearsonkyle]

I am familiar with Refik Anadol, I follow him on instagram. I was actually inspired by some of his art pieces which is why I created this HiRise one and a few others. Although I later found out I couldn't produce the same quality of images with just one GPU. That guy and his studio has some serious computing power.

I love the idea you're going for, I've been wanting to do something similar but using a fourier transform of an audio wave as the latent space input. Machine learning is certainly a way you could morph surfaces together although not the only way. Currently, you need quite a bit of computing power to morph surfaces in real-time particularly at a higher resolution. In my own research, I've only been able to achieve a resolution of ~128x128 with my current resources (just one GPU). I feel like this may limit your options for displaying things.

Check out this post I made: https://www.instagram.com/p/BzHmDp-l4Wj/?utm_source=ig_web_copy_link

You could try something like that, where you process audio data in real time and then use it to create a surface. The surface in that post is essentially a spectrogram. The surface doesn't necessarily have to be rectangular either

I'm not sure what you mean by "what kind of handles can you attach to the height generation in latent space?"

[Aoost]

Hi Kyle,

I think for this project ML is not really necessary and as you say, will likely take more compute power than I have access to. I had thought of doing the project in a similar way to what you've described. Is the video you sent generating new meshes as the audio plays or are you just manipulating an existing mesh?

I've been able to produce promising results pushing height around using the new Unity PBR shadergraph but I've had trouble applying them to Terrains in Unity, which don't seem to be optimised for manipulation at runtime, even though it would probably be better to paint a landscape with the hiRise Data and manipulate it than it would be to use different meshes in a tapestry of sorts.

Ideally what I'd like to see is an interpolation between heightmap data that is driven through audio, but again, I'm unsure of how to apply a heightshader to a terrain at this point.

Let me know what you think,

Thanks again for your help.

Best,

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Mon, Jul 8, 2019 at 6:29 PM Kyle A. Pearson notifications@github.com wrote:

I am familiar with Refik Anadol, I follow him on instagram. I was actually inspired by some of his art pieces which is why I created this HiRise one and a few others. Although I later found out I couldn't produce the same quality of images with just one GPU. That guy and his studio has some serious computing power.

I love the idea you're going for, I've been wanting to do something similar but using a fourier transform of an audio wave as the latent space input. Machine learning is certainly a way you could morph surfaces together although not the only way. Currently, you need quite a bit of computing power to morph surfaces in real-time particularly at a higher resolution. In my own research, I've only been able to achieve a resolution of ~128x128 with my current resources (just one GPU). I feel like this may limit your options for displaying things.

Check out this post I made: https://www.instagram.com/p/BzHmDp-l4Wj/?utm_source=ig_web_copy_link

You could try something like that, where you process audio data in real time and then use it to create a surface. The surface in that post is essentially a spectrogram. The surface doesn't necessarily have to be rectangular either

I'm not sure what you mean by "what kind of handles can you attach to the height generation in latent space?"

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pearsonkyle/Mars-Machina/issues/1?email_source=notifications&email_token=AHDMNHSXPTTTQC2U2P2IWA3P6NTPHA5CNFSM4H5GBXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNUFVQ#issuecomment-509297366, or mute the thread https://github.com/notifications/unsubscribe-auth/AHDMNHUGYDGRSM3DF5NICOLP6NTPHANCNFSM4H5GBXAA .

[pearsonkyle]

The video I sent I am manipulating an existing mesh by computing new vertices each time step. The script meshGenerator.cs in this repo can help you generate a mesh the way I did in the video.

[Aoost]

which repo?

On Tue, 16 Jul 2019, 05:55 Kyle A. Pearson, notifications@github.com wrote:

The video I sent I am manipulating an existing mesh by computing new vertices each time step. The script meshGenerator.cs in this repo can help you generate a mesh the way I did in the video.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pearsonkyle/Mars-Machina/issues/1?email_source=notifications&email_token=AHDMNHXJDCXMDWBWZMQ5RITP7VBDFA5CNFSM4H5GBXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ7TWHY#issuecomment-511654687, or mute the thread https://github.com/notifications/unsubscribe-auth/AHDMNHUDWRUIG65YD6MXYR3P7VBDFANCNFSM4H5GBXAA .

[Aoost]

Do you by any chance know of a repo or method to conform meshes to one another?

I want to create a conformation of DTM outputs from Blender, because simply painting a terrain is not editable at runtime.

Best and Thanks,

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Tue, Jul 16, 2019 at 10:59 AM Adam Oosthuizen adam.oosthuizen@gmail.com wrote:

which repo?

On Tue, 16 Jul 2019, 05:55 Kyle A. Pearson, notifications@github.com wrote:

The video I sent I am manipulating an existing mesh by computing new vertices each time step. The script meshGenerator.cs in this repo can help you generate a mesh the way I did in the video.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pearsonkyle/Mars-Machina/issues/1?email_source=notifications&email_token=AHDMNHXJDCXMDWBWZMQ5RITP7VBDFA5CNFSM4H5GBXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ7TWHY#issuecomment-511654687, or mute the thread https://github.com/notifications/unsubscribe-auth/AHDMNHUDWRUIG65YD6MXYR3P7VBDFANCNFSM4H5GBXAA .

[Aoost]

I saw that you were referring to the mesh generator in your Mars-Machina repo :)

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Tue, Jul 16, 2019 at 6:20 PM Adam Oosthuizen adam.oosthuizen@gmail.com wrote:

Do you by any chance know of a repo or method to conform meshes to one another?

I want to create a conformation of DTM outputs from Blender, because simply painting a terrain is not editable at runtime.

Best and Thanks,

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Tue, Jul 16, 2019 at 10:59 AM Adam Oosthuizen < adam.oosthuizen@gmail.com> wrote:

which repo?

On Tue, 16 Jul 2019, 05:55 Kyle A. Pearson, notifications@github.com wrote:

The video I sent I am manipulating an existing mesh by computing new vertices each time step. The script meshGenerator.cs in this repo can help you generate a mesh the way I did in the video.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pearsonkyle/Mars-Machina/issues/1?email_source=notifications&email_token=AHDMNHXJDCXMDWBWZMQ5RITP7VBDFA5CNFSM4H5GBXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZ7TWHY#issuecomment-511654687, or mute the thread https://github.com/notifications/unsubscribe-auth/AHDMNHUDWRUIG65YD6MXYR3P7VBDFANCNFSM4H5GBXAA .

[pearsonkyle]

I'm not sure I know of another repo to interpolate between meshes. That's something you'll probably have to create. There is another option in Unity called BlendShapes or something along those lines that is used to deform meshes for animations. However, that requires each mesh to have the same number of vertices

I'd also recommend this youtube video: https://www.youtube.com/watch?v=eJEpeUH1EMg&t=421s It's where I started before making that mesh script

[Aoost]

I guess what I'd ideally have is a way to conform meshes to one another when they get close enough. I found something a while ago, but never saved the repo.

I have managed to interpolate between heightmaps on a mesh through a shader.

I am aware of the blendshapes method, but I'd need my meshes to conform to one another so as to have a consistent level to polace a 3360 camera on.

I have been through that series of mesh generation by Brackeys, he's great :)

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Tue, Jul 16, 2019 at 10:13 PM Kyle A. Pearson notifications@github.com wrote:

I'm not sure I know of another repo to interpolate between meshes. That's something you'll probably have to create. There is another option in Unity called BlendShapes or something along those lines that is used to deform meshes for animations. However, that requires each mesh to have the same number of vertices

I'd also recommend this youtube video: https://www.youtube.com/watch?v=eJEpeUH1EMg&t=421s It's where I started before making that mesh script

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pearsonkyle/Mars-Machina/issues/1?email_source=notifications&email_token=AHDMNHQPWT47PMFTT5M7FPLP7YTX3A5CNFSM4H5GBXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2CAFAY#issuecomment-511967875, or mute the thread https://github.com/notifications/unsubscribe-auth/AHDMNHUYNS4NWG6PAQK2UCTP7YTX3ANCNFSM4H5GBXAA .

[Aoost]

Did you use a separate mesh generator for the sphere you used in the instagram post or did you read the vertex data in?

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Wed, Jul 17, 2019 at 1:09 PM Adam Oosthuizen adam.oosthuizen@gmail.com wrote:

I guess what I'd ideally have is a way to conform meshes to one another when they get close enough. I found something a while ago, but never saved the repo.

I have managed to interpolate between heightmaps on a mesh through a shader.

I am aware of the blendshapes method, but I'd need my meshes to conform to one another so as to have a consistent level to polace a 3360 camera on.

I have been through that series of mesh generation by Brackeys, he's great :)

Adam Oosthuizen

Virtual Reality and 360° Content Producer

(+27) 079 502 0847 <+27%2079%20502%200847>

On Tue, Jul 16, 2019 at 10:13 PM Kyle A. Pearson notifications@github.com wrote:

I'm not sure I know of another repo to interpolate between meshes. That's something you'll probably have to create. There is another option in Unity called BlendShapes or something along those lines that is used to deform meshes for animations. However, that requires each mesh to have the same number of vertices

I'd also recommend this youtube video: https://www.youtube.com/watch?v=eJEpeUH1EMg&t=421s It's where I started before making that mesh script

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/pearsonkyle/Mars-Machina/issues/1?email_source=notifications&email_token=AHDMNHQPWT47PMFTT5M7FPLP7YTX3A5CNFSM4H5GBXAKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD2CAFAY#issuecomment-511967875, or mute the thread https://github.com/notifications/unsubscribe-auth/AHDMNHUYNS4NWG6PAQK2UCTP7YTX3ANCNFSM4H5GBXAA .