search

Clay-foundation / model

The Clay Foundation Model (in development)
https://clay-foundation.github.io/model/
Apache License 2.0
242 stars 25 forks source link

More detailed image plots #170

Closed brunosan closed 2 months ago

brunosan commented 4 months ago

This changes create a more detailed Image log, with the actual RGB, , and DEM.

It also chooses the most likely item of the first 3 batches to have structure (max stdev), and plots all bands, the predictions and the differences, like this

image

alkalait commented 3 months ago

Hello! Where can I find a how-to on reading these plots? @yellowcap @brunosan

brunosan commented 3 months ago

Hi @alkalait ! I don't actually think this will get merged, but here's a description of what you see.

The code picks the item on the first 3 batches with most stdev as a proxy for features within.

Each Clay has many spectral bands, so the plot tries to see what's being learned by displaying the RGB as a color image, then the average of all red edge, Infrareds, SAR and the elevation model (all this on the top row).

the middle row has the output as the model learns to reconstruct the image. Random at first, and ideally ends up being the same as the input.

The bottom row is just the normalized difference.

In a ideal world, the middle row goes from random to the same as the first row, as the bottom row becomes grey (no difference). Hope it helps.

alkalait commented 3 months ago

That helps, thanks @brunosan. Based on what you said, in this particular instance the model is not doing well at reconstructing the patches. Am I reading it correctly?

brunosan commented 3 months ago

Indeed. I did not leave it long, but yes, the reconstruction is not even close. You basically need to leave it longer on training, but are also tracking a potential issue on #156.

yellowcap commented 2 months ago

We plan to implement RGB plots for the v1 output, we are not planning to update v0 ouptut at this point anymore.