search

nmichlo / disent

🧶 Modular VAE disentanglement framework for python built with PyTorch Lightning ▸ Including metrics and datasets ▸ With strongly supervised, weakly supervised and unsupervised methods ▸ Easily configured and run with Hydra config ▸ Inspired by disentanglement_lib
https://disent.michlo.dev
MIT License
121 stars 18 forks source link

[FEATURE]: Add DMS metric #33

Open JarvisGexiaoXu opened 2 years ago

JarvisGexiaoXu commented 2 years ago

Hi Nathan,

Just curious would it be feasible to add MNNIST to the data options and add DMS (from Beta-VAE paper https://openreview.net/forum?id=Sy2fzU9gl) to metrics. Just started learning PyTorch Lightning, curious about scalability.

Best, JRS

nmichlo commented 2 years ago

Hi there,

MNIST is not specifically a disentanglement dataset. I would be hesitant to add it to disent because of this, also because there are many existing dataset loaders for this dataset.

These is an example that uses MNIST, if you want to look at how to load typical datasets: https://github.com/nmichlo/disent/blob/fe8b66dd0de0f8eefe6bd62c3739405492618517/docs/examples/mnist_example.py

As for DMS, I can open a TODO, however, would you be willing to help contribute this metric?

JarvisGexiaoXu commented 2 years ago

Hi Nathan,

I am a newbie in this field, probably I can start by reading the TF implementation of DMS first, to see if I am capable of this task.

Best, JRS

IceClear commented 1 year ago

@nmichlo Hi, Nathan. Thanks for the awesome work. May I ask how to evaluate the metrics in this repo on MINST? Since there is no ground-truth, it is not possible to set self.__factor_sizes. But I think these metrics do not rely on labels for calculation. Is it possible to calculate these metrics on other datasets? Looking forward to your reply and thanks for your contributions!

nmichlo commented 1 year ago

Hi there @IceClear, thank you for the kind words!

Unfortunately it is not possible to calculate these metrics directly on MNIST, the dataset itself is not intended for disentanglement. Disentanglement datasets require that the datapoints have a one to one correspondance with the set product of the underlying individual ground-truth factors. MNIST unfortunately has more datapoints than there are unique labels.

I guess one way around this could be to treat the MNIST classes as ground-truth factors, but then sample randomly from all possible observations that have that same class as a label. Metrics could still possibly be computed off of such a dataset, but results might not work as expected.

My recommendation would be to stick to the usual datasets for the disentanglement metrics.

IceClear commented 1 year ago

Hi there @IceClear, thank you for the kind words!

Unfortunately it is not possible to calculate these metrics directly on MNIST, the dataset itself is not intended for disentanglement. Disentanglement datasets require that the datapoints have a one to one correspondance with the set product of the underlying individual ground-truth factors. MNIST unfortunately has more datapoints than there are unique labels.

I guess one way around this could be to treat the MNIST classes as ground-truth factors, but then sample randomly from all possible observations that have that same class as a label. Metrics could still possibly be computed off of such a dataset, but results might not work as expected.

My recommendation would be to stick to the usual datasets for the disentanglement metrics.

Noted. Thanks for your reply!