Multi-Modal VAE (THIS IS OLD VERSION)
A very simple experiment with 3D-Face dataset
1) Setup (brief)
-
Two-modal paired image data by:
1) private-1 = elevation (illumination = neutral value fixed)
2) private-2 = illumination (elevation = neutral value fixed)
3) shared = (azimuth, id) -
Let: xI = modality 1, xT = modality 2
2) Models (Competing)
2-a) MuMo-VAE model
- partition of latent variables = (zI, zT, zS)
- dim(zI) = 2, dim(zT) = 2, dim(zS) = 5
- 2 decoders: 1) pI(xI | zI, zS), 2) pT(xT | zT, zS)
- 3 encoders (no parameter sharing): 1) qI(zI, zS | xI), 2) qT(zT, zS | xT), 3) q(zI, zT, zS | xI, xT)
- 3 VAE losses, one for each of {xI}, {xT}, and {(xI,xT)}
2-b) Vanilla VAE regarding (xI,xT) as (concatenated) observation
- no partitioning of latent variables
- there is only one encoder model q(z | xI, xT)
- there is only one decoder model p(xI, xT | z)
- dim(z) = 10
2-c) MMPOE-VAE v1: induce q(zI, zT, zS | xI, xT) from Product-of-Experts
- partition of latent variables = (zI, zT, zS)
- dim(zI) = 2, dim(zT) = 2, dim(zS) = 5
- 2 decoders: 1) pI(xI | zI, zS), 2) pT(xT | zT, zS)
- 2 encoders (no parameter sharing): 1) qI(zI, zS | xI), 2) qT(zT, zS | xT)
- q(zI, zT, zS | xI, xT) = qI(zI|xI) qT(zT|xT) q(zS | xI, xT),
where q(zS | xI, xT) \propto p(zS) qI(zS|xI) qT(zS|xT)
- (why v1?) 1 VAE loss, for {(xI,xT)}
2-d) MMPOE-VAE v2: induce q(zI, zT, zS | xI, xT) from Product-of-Experts
- The same setup as MMPOE-VAE-v1, but ...
- (why v2?) 3 VAE losses, one for each of {xI}, {xT}, and {(xI,xT)}
2-e) WG-VAE v1: no private variables; induce q(z | xI, xT) from Product-of-Experts
- Wu-Goodman's multi-modal VAE
- no partition of latent variables, just shared z
- dim(z) = 10
- 2 decoders: 1) pI(xI | z), 2) pT(xT | z)
- 2 encoders (no parameter sharing): 1) qI(z | xI), 2) qT(z | xT)
- q(z | xI, xT) \propto p(z) qI(z|xI) qT(z|xT)
- (why v1?) 1 VAE loss, for {(xI,xT)}
2-f) WG-VAE v2: no private variables; induce q(z | xI, xT) from Product-of-Experts
- The same setup as WG-VAE-v1, but ...
- (why v2?) 3 VAE losses, one for each of {xI}, {xT}, and {(xI,xT)}
- So, the setup is pretty much the same as the original Wu-Goodman's multi-modal VAE
+ Latent traversal: (xI,xT) -> z or (zI,zS,zT), from which traverse along each axis -> (xI',xT')
(at iter 300K)
Trv-a) MuMo-VAE model
3 instances, each:
True xI | xI w/ zI(1) change | xI w/ zI(2) | xI w/ zS(1) | xI w/ zS(2) | ... | xI w/ zT(1) | xI w/ zT(2)
True xT | xT w/ zI(1) change | xT w/ zI(2) | xT w/ zS(1) | xT w/ zS(2) | ... | xT w/ zT(1) | xT w/ zT(2)
(note: quite accurately identify private and shared factors, but computational issue of having dyadic inf net)
Trv-b) Vanilla VAE regarding (xI,xT) as (concatenated) observation
3 instances, each:
True xI | xI w/ z(1) change | xI w/ z(2) | ... | xI w/ z(10)
True xT | xT w/ z(1) change | xT w/ z(2) | ... | xT w/ z(10)
(note: variation of z(4) or z(7), none of them shared factors, results in changes in both xI and xT)
Trv-c) MMPOE-VAE v1
3 instances, each:
True xI | xI w/ zI(1) change | xI w/ zI(2) | xI w/ zS(1) | xI w/ zS(2) | ... | xI w/ zT(1) | xI w/ zT(2)
True xT | xT w/ zI(1) change | xT w/ zI(2) | xT w/ zS(1) | xT w/ zS(2) | ... | xT w/ zT(1) | xT w/ zT(2)
(note: problematic! eg, zS(1) learns elevation factor, but it should be a private factor in zI)
Trv-d) MMPOE-VAE v2
3 instances, each:
True xI | xI w/ zI(1) change | xI w/ zI(2) | xI w/ zS(1) | xI w/ zS(2) | ... | xI w/ zT(1) | xI w/ zT(2)
True xT | xT w/ zI(1) change | xT w/ zI(2) | xT w/ zS(1) | xT w/ zS(2) | ... | xT w/ zT(1) | xT w/ zT(2)
(note: better identify/discern the private and shared factors, which implies that the loss terms for marginal data, ie, {xI} and {xT}, are necessary?)
Trv-e) WG-VAE v1
3 instances, each:
True xI | xI w/ z(1) change | xI w/ z(2) | ... | xI w/ z(10)
True xT | xT w/ z(1) change | xT w/ z(2) | ... | xT w/ z(10)
Trv-f) WG-VAE v2
3 instances, each:
True xI | xI w/ z(1) change | xI w/ z(2) | ... | xI w/ z(10)
True xT | xT w/ z(1) change | xT w/ z(2) | ... | xT w/ z(10)
+ Pure synthesis: z or (zI,zS,zT) ~ N(0,I) -> (xI,xT)
(at iter 300K)
PureSynth-a) MuMo-VAE model
[xI, xT]
PureSynth-b) Vanilla VAE regarding (xI,xT) as (concatenated) observation
[xI, xT]
PureSynth-c) MMPOE-VAE v1
[xI, xT]
(note: the quality of generated images is not satisfactory.. especially when compared to the v2 model below)
PureSynth-d) MMPOE-VAE v2
[xI, xT]
PureSynth-e) WG-VAE v1
[xI, xT]
PureSynth-f) WG-VAE v2
[xI, xT]
+ Cross-modal synthesis: Given xI, infer zS, zT ~ N(0,I) -> xT (changing the role of I and T)
(at iter 300K)
CMSynth--a) MuMo-VAE model
XI -> XT
[XI | three randomly synthesized XT images]
(note: in the synthesized XT images, illumination (private-T) can vary, but elevation (private-I) should be neutral, and (azimuth, id) should be identical to those of XI)
XT -> XI
[XT | three randomly synthesized XI images]
(note: in the synthesized XI images, elevation (private-I) can vary, but illumination (private-T) should be neutral, and (azimuth, id) should be identical to those of XT)
CMSynth-b) Vanilla VAE regarding (xI,xT) as (concatenated) observation
Of course, N/A
CMSynth-c) MMPOE-VAE v1
XI -> XT
[XI | three randomly synthesized XT images]
XT -> XI
[XT | three randomly synthesized XI images]
(note: again, v1 suffers from poor quality of synthesized images. It seems to be necessary to take into account the marginal data {xI} and {xT} in the training..)
CMSynth-d) MMPOE-VAE v2
XI -> XT
[XI | three randomly synthesized XT images]
XT -> XI
[XT | three randomly synthesized XI images]
CMSynth-e) WG-VAE v1
XI -> XT
[XI | a synthesized XT image]
XT -> XI
[XT | a synthesized XI image]
CMSynth-f) WG-VAE v2
XI -> XT
[XI | a synthesized XT image]
XT -> XI
[XT | a synthesized XI image]
