Clay-foundation / model

The Clay Foundation Model (in development)
https://clay-foundation.github.io/model/
Apache License 2.0
262 stars 30 forks source link

Static Documentation for v0.1 #118

Closed brunosan closed 6 months ago

brunosan commented 6 months ago

This PR expands the current documentation based on #64 and tries to clarify structure.

It does not include executable documentation (Experiments, renamed tutorials) since I don't know yet how we can show the outputs of these without adding execution outputs to the repo history, nor having the CI download and execute the code. I'll PR a solution forking from here.

It also bring over the notebook from branch clay-over-ai.

We should have this merged before Wednesday presentation in Davos.

brunosan commented 6 months ago
240105-clay-v0-interpolation.ipynb
240105-clay-v0-reconstruction.ipynb
240105-location-embeddings.ipynb

Currently fail to run. I suspect @srmsoumya made this with a different version of the ClayModule, but I can't track the differences.

Error is:

---------------------------------------------------------------------------
RuntimeError                              Traceback (most recent call last)
...
(
  unmasked_patches,
   unmasked_indices,
   masked_indices,
   masked_matrix,
--->  = model.model.encoder(batch)
...
File [~/code/Clay/model/docs/../src/model_clay.py:304  in Encoder.forward(self, datacube)
 cube, time, latlon = (
 datacube["pixels"],
 datacube["timestep"],
datacube["latlon"],
 )  # [B C H W]
 B, C, H, W = cube.shape
-->  patches = self.to_patch_embed(
       cube
     )  # [B G L D] - patchify & create embeddings per patch
 ...
RuntimeError: Input type (c10::Half) and bias type (float) should be the same
brunosan commented 6 months ago

Yes please make all changes you feel should be done. I can take over later today.