Explore Street View depth maps

vict0rsch commented 5 years ago

formulate needs if we get in touch with Google

vict0rsch commented 5 years ago

The Elevation API is not yet open to the public/research. May happen in a few months. Can't really count on it.

vict0rsch commented 5 years ago

In the mean time, this is still open: https://maps.google.com/cbk?output=xml&ll=45.508457,-73.532738&dm=1

And we can generate such an image using https://github.com/cc-ai/floods-gans/blob/master/various_scripts/depth_map.py

Figure_1

Now open questions:

can we map an exact StreetView position to such a query?
- can we get the original image from which this is computed?
- can we specify pitch, camera view, angle and so on
is the code fast enough + scalable?
how to compute a water plane from this data?
how to embed said plane into original image?

GitHub
cc-ai/floods-gans
Using GANs to simulate how a house would look like, had it suffered from a flood - cc-ai/floods-gans

gcosne commented 5 years ago

Before going into matching depth-map and street view images automatically, I would like to know if you found GPS coordinates where you managed to match it qualitatively playing on streetview? If so could you please communicate them? I can't find a single place where I can match it visually.

vict0rsch commented 5 years ago

I agree I have not and it is an open question since the API is not documented (or is it?)

Vahe987 commented 5 years ago

@DatCorno on the backend we are using the same API or it's different?

gcosne commented 5 years ago

Reconstructing the spherical img There are some steps to automatically reconstruct the spherical image relative to a pano_id that should be developped but in the meantime this App do the work for us.

Reconstructing 3D point cloud Here is a notebook colab to explore depth-map where I tested if we can superpose depth map to the panoramic img without success and where I translated some medium advices to generate the 3D point cloud (there is work to do before having a height in meter)

vict0rsch commented 5 years ago

Ok so this is not good enough for us to work with right? This API is still open but basically it's not as informative as we expected?

gcosne commented 5 years ago

I would not say that, I believe we should write our own code following the steps in this paper explaining how to reconstruct the panorama ourself from the tiles and see if it match the depth map: There is a lot of potential in this approach.

Vahe987 commented 5 years ago

@gcosne how long it will take in your opinion, if we write our own code?

gcosne commented 5 years ago

Somehow I find a panorama that almost match the depth map :

match

Remark : I discover one thing: Height of the Lidar is about 2.5 meters. Using the formula Normalized/minimum(height) * Lidar-Height allow to translate the normalized coordinates in meters.

Then I tried to" flood "the image until a certain height but I think the lidar doesn't have a lot of points below 1.5m. Hence the segmentation map generated is good when height >2.m but get sparse when we go below 1.7m. Flood up to 1.7m flood_17

Flood up to 1.3m flood_13

vict0rsch commented 5 years ago

Pas mal du tout. La question qui nous reste sur les bras c'est: est-ce qu'on peut travailler avec ces panoramiques? i.e. est-ce que c'est suffisamment "relatable" pour en faire qqch et est-ce qu'on peut l'avoir pour n'importe quel endroit / si non lesquels?

Il faut en parler à Alice et mettre au point un premier groupe de testing utilisateur pour se faire un avis, imho

vict0rsch commented 5 years ago

oups French, I meant: this is not bad at all! we ought to research that with users to understand if that's usable / relatable for our purpose + figure out if there are restrictions on localizations for which we can get this depth map + original panorama

sashavor commented 5 years ago

That's amazing progress!

We can even just start with flooding the ground/road, and see if we can raise it little by little

On Thu, May 30, 2019 at 12:32 PM Gautier Cosne notifications@github.com wrote:

Somehow I find a panorama that almost match the depth map :

[image: match] https://user-images.githubusercontent.com/25148268/58647292-36c47380-82d5-11e9-8075-15e111323e36.png

Remark : I discover one thing: Height of the Lidar is about 2.5 meters. Using the formula Normalized/minimum(height) * Lidar-Height allow to translate the normalized coordinates in meters.

Then I tried to" flood "the image until a certain height but I think the lidar doesn't have a lot of points below 1.5m. Hence the segmentation map generated is good when height >2.m but get sparse when we go below 1.7m. Flood up to 1.7m [image: flood_17] https://user-images.githubusercontent.com/25148268/58648040-cae30a80-82d6-11e9-9374-578f7579ab2a.png

Flood up to 1.3m [image: flood_13] https://user-images.githubusercontent.com/25148268/58648079-e4845200-82d6-11e9-9272-1dc3c52ac8dd.png m

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cc-ai/kdb/issues/23?email_source=notifications&email_token=ADMMIIRQVDS6K5QQOJUX6RTPX76QBA5CNFSM4HHZBXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWS2BTY#issuecomment-497393871, or mute the thread https://github.com/notifications/unsubscribe-auth/ADMMIIQHUONW244GIYBAVELPX76QBANCNFSM4HHZBXOQ .

-- Sasha Luccioni Postdoctoral Researcher Climate Change AI https://mila.quebec/en/person/sasha-luccioni/

Vahe987 commented 5 years ago

very interesting, good job, so you found the magic number I was asking :) for the flooding part maybe we can find another technick, like reconstructing the scene in 3D and flood afterwards ... but not sure for now, maybe too heavy, we can think more about it

gcosne commented 5 years ago

Testing with depth estimation : Using very light single image depth estimator rather than 'google api', we almost have usable masks.

To Be Done:

Identify if we have region where google api works for us.
Calibrating depth estimator with google api.

Qualitative Results flooding up to 1.5m :

Vahe987 commented 5 years ago

well, the flood part doesn't work ... btw, what about people on street view images? should we "drown" them?

sashavor commented 5 years ago

Lol I don't think so, just ignore them

On Mon, Jun 3, 2019 at 12:39 PM Vahe notifications@github.com wrote:

well, the flood part doesn't work ... btw, what about people on street view images? should we "drown" them?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cc-ai/kdb/issues/23?email_source=notifications&email_token=ADMMIIVJAFCSB32YTTSJYBTPYVCNVA5CNFSM4HHZBXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWZ7PNA#issuecomment-498333620, or mute the thread https://github.com/notifications/unsubscribe-auth/ADMMIIVHCHBKN52QJFY3XOLPYVCNVANCNFSM4HHZBXOQ .

-- Sasha Luccioni Postdoctoral Researcher Climate Change AI https://mila.quebec/en/person/sasha-luccioni/

gcosne commented 5 years ago

Update : I translated DownloadPanorama C++ function from this incredible project in python.

We can now automatically extract xml information and generate both depthmap and panoramic images within a script. I think I will explore more of PaulWagener project, I sent him an email.

Cropping the image so they are the same size rather than resizing them actually work better. Not perfect yet but I will fix that soon.

pano_img depth1

sashavor commented 5 years ago

So it's working now?! It even matches up images and depth maps??

On Mon, Jun 3, 2019 at 2:43 PM Gautier Cosne notifications@github.com wrote:

Update : I translated DownloadPanorama https://github.com/PaulWagener/Streetview-Explorer/blob/master/src/Panorama.cpp C++ function from this incredible project in python.

We can now automatically extract xml information and generate both depthmap and panoramic images within a script. I think I will explore more of PaulWagener project, I sent him an email.

Cropping the image so they are the same size rather than resizing them actually work better. Not perfect yet but I will fix that soon.

[image: pano_img] https://user-images.githubusercontent.com/25148268/58836405-c1370b00-8626-11e9-9f19-62166eef6df0.png [image: depth1] https://user-images.githubusercontent.com/25148268/58836437-d6ac3500-8626-11e9-9b73-efdf7b55096f.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cc-ai/kdb/issues/23?email_source=notifications&email_token=ADMMIIXSBTAILQ5QSNLCNNDPYWGAXA5CNFSM4HHZBXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW2Y5AA#issuecomment-498437760, or mute the thread https://github.com/notifications/unsubscribe-auth/ADMMIIXEKQKU5DBRUN7PTX3PYWGAXANCNFSM4HHZBXOQ .

-- Sasha Luccioni Postdoctoral Researcher Climate Change AI https://mila.quebec/en/person/sasha-luccioni/

gcosne commented 5 years ago

Not yet but I do trust it will, I have to play with yaw heading.

sashavor commented 5 years ago

Great! good job!!

On Mon, Jun 3, 2019 at 3:06 PM Gautier Cosne notifications@github.com wrote:

Not yet but I do trust it will, I have to play with yaw heading.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cc-ai/kdb/issues/23?email_source=notifications&email_token=ADMMIIVT6BMD4SNEWHCQC5LPYWIWNA5CNFSM4HHZBXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW22PEY#issuecomment-498444179, or mute the thread https://github.com/notifications/unsubscribe-auth/ADMMIIUQE5DILOLDT6L7L4LPYWIWNANCNFSM4HHZBXOQ .

-- Sasha Luccioni Postdoctoral Researcher Climate Change AI https://mila.quebec/en/person/sasha-luccioni/

gcosne commented 5 years ago

Update : We can almost call it a success. seg14 seg11

I said almost because the size of the tiles are not consistent, sometimes the 360° equirectangular image is already built (no overlapping between tiles), sometimes it's not. The good news are that we can identify it by the amount of black padding in the image.

360° panorama without overlapping More than 360° panoramic image amount_of_black

sashavor commented 5 years ago

Good job!

You think the variation is due to a change in their approach somewhere along the way?

On Tue, Jun 4, 2019 at 9:31 AM Gautier Cosne notifications@github.com wrote:

Update : We can almost call it a success. [image: seg14] https://user-images.githubusercontent.com/25148268/58882301-e9b41900-86a9-11e9-83b8-1eb0c256e7c3.png [image: seg11] https://user-images.githubusercontent.com/25148268/58882303-ea4caf80-86a9-11e9-9015-4fe85ac4d8f6.png

I said almost because the size of the tiles are not consistent, sometimes the 360° equirectangular image is already built (no overlapping between tiles), sometimes it's not. The good news are that we can identify it by the amount of black padding in the image.

360° panorama without overlapping More than 360° panoramic image* [image: amount_of_black] https://user-images.githubusercontent.com/25148268/58882718-df464f00-86aa-11e9-98cb-6422f55a9950.png

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cc-ai/kdb/issues/23?email_source=notifications&email_token=ADMMIIQBKCZQYCLYT6NE2TTPYZVDZA5CNFSM4HHZBXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW4SJ7I#issuecomment-498672893, or mute the thread https://github.com/notifications/unsubscribe-auth/ADMMIITBWDTNUS5BMPHWN5TPYZVDZANCNFSM4HHZBXOQ .

-- Sasha Luccioni Postdoctoral Researcher Climate Change AI https://mila.quebec/en/person/sasha-luccioni/

gcosne commented 5 years ago

I assume so, nothing of an issue here but just a thing to be aware.

Next steps :

Finish the work in low resolution.
Create the stitching function for already built panoramic images in higher resolution.

sashavor commented 5 years ago

Sounds good!

On Tue, Jun 4, 2019 at 9:38 AM Gautier Cosne notifications@github.com wrote:

I assume so, nothing of an issue here but just a things to be aware.

Next steps :

Finish the work in low resolution.

Create the stitching function for already built panoramic images in higher resolution.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cc-ai/kdb/issues/23?email_source=notifications&email_token=ADMMIIX4GCXJZW53L6YEL4DPYZV6RA5CNFSM4HHZBXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW4S7CI#issuecomment-498675593, or mute the thread https://github.com/notifications/unsubscribe-auth/ADMMIIUE6PEYHLOKHJN2EE3PYZV6RANCNFSM4HHZBXOQ .

-- Sasha Luccioni Postdoctoral Researcher Climate Change AI https://mila.quebec/en/person/sasha-luccioni/

Vahe987 commented 5 years ago

cool, good job ;)

gcosne commented 5 years ago

Done: Low and High resolution are working. I will clean and commit the code soon.

Next steps : Make it work with the 3D.

Flood the ground.
Flood a certain height.

sashavor commented 5 years ago

Very very nice, Gautier!

On Tue, Jun 4, 2019 at 11:02 AM Gautier Cosne notifications@github.com wrote:

Done: Low and High resolution are working. I will clean and commit the code soon.

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/cc-ai/kdb/issues/23?email_source=notifications&email_token=ADMMIITZPGLLOIQWHZNCWUTPYZ7XVA5CNFSM4HHZBXO2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODW43PFY#issuecomment-498710423, or mute the thread https://github.com/notifications/unsubscribe-auth/ADMMIITY42NP2FUA5322353PYZ7XVANCNFSM4HHZBXOQ .

-- Sasha Luccioni Postdoctoral Researcher Climate Change AI https://mila.quebec/en/person/sasha-luccioni/

gcosne commented 5 years ago

3D reconstruction: There is an echo around the car that makes the ground plane estimation tricky 3D_cloud echo

cc-ai / kdb

Explore Street View depth maps #23