The problem is way harder in unpaired setting. The model should be regularized to be able to find the correct mapping of modalities. Without regularization, the integration of modalities in the latent is not meaningful (as tested). One way to overcome this issue is to add cycle consistency to the model as follows:
Take x_i (the feature matrix in domain i) and encode it to become z_i.
decode z_i with condition j to become r_ij (the reconstructed x_i through the decoder of modality j). Here we have converted the feature matrix x_i from domain i to domain j.
Encode the r_ij to reach z_ij.
The distance between z_i and z_ij should be minimum.
Tasks:
[x] Add cycle consistency to MultiscAE and MultiscCAE models.
The problem is way harder in unpaired setting. The model should be regularized to be able to find the correct mapping of modalities. Without regularization, the integration of modalities in the latent is not meaningful (as tested). One way to overcome this issue is to add cycle consistency to the model as follows:
x_i
(the feature matrix in domaini
) and encode it to becomez_i
.z_i
with conditionj
to becomer_ij
(the reconstructedx_i
through the decoder of modalityj
). Here we have converted the feature matrixx_i
from domaini
to domainj
.r_ij
to reachz_ij
.z_i
andz_ij
should be minimum.Tasks:
MultiscAE
andMultiscCAE
models.