Clay-foundation / model

The Clay Foundation Model (in development)
https://clay-foundation.github.io/model/
Apache License 2.0
299 stars 38 forks source link

Callback function to log Masked Autoencoder reconstructions to WandB #88

Closed weiji14 closed 8 months ago

weiji14 commented 8 months ago

To visually inspect how the Masked Autoencoder is performing over the training run in terms of reconstructing to original image.

Implemented as a Lightning Callback function which runs at the end of a validation loop's mini-batch (on_validation_batch_end). A sample of 6 image pairs (original + reconstructed, so total of 12) are uploaded online.

Example usage with LightningCLI:

Samples of histogram equalized RGB images, and the reconstructed outputs (only random noise as this is early in the training).

image

python trainer.py fit --trainer.max_epochs=2 \
                      --trainer.precision=bf16-mixed \
                      --data.data_path=data/32VLM \
                      --data.num_workers=4 \
                      --trainer.logger=WandbLogger \
                      --trainer.logger.project=clay \
                      --trainer.logger.save_dir=checkpoints \
                      --trainer.callbacks+=LogMAEReconstruction \

TODO:

TODO in the future:

References:

weiji14 commented 8 months ago

Gonna merge this in first and combine with the other wandb callback being developed at #47.