Best practices for domain adaptation using multiple sources

[omershalev]

I have a single target dataset and multiple source datasets. I wonder how I should apply domain adaptation using ADAPT for that purpose. Which algorithms or methods (feature-based, instance-based, parameter-based) are more suitable for this kind of setup?

[antoinedemathelin]

Hi @omershalev, Thank you for your interest in the Adapt library! For now, little implementations in Adapt are tailored to handle multi-source. The only one is FA (see #86) but, it is very basic.

For the question of which is best between instance, parameter and feature for multisource, I would say that it depends on your problem. I think that people look, in general, for feature-based methods, with the idea to find a common subspace for all sources...

Future planned Adapt developments include multisource methods, however it will take some time... Some existing Adapt implementations could be improved rather quickly for multisource as RegularTransferLR (https://adapt-python.github.io/adapt/generated/adapt.parameter_based.RegularTransferLR.html) but I dont know if it fits your problem.

Best,

[omershalev]

Thank you @antoinedemathelin for the detailed response! My initial impression was that parameter-based methods (e.g., RegularTransferLR) are more suited to the multi-source setup. I would have thought of iteratively re-fitting the model in the following fashion:

src1_model = Ridge()
src1_model.fit(Xs1, ys1)
src2_model = RegularTransferLR(src1_model, lambda_=1.)
src2_model.fit(Xs2, ys2)
tgt_model = RegularTransferLR(src2_model, lambda_=1.)
tgt_model.fit(Xt, yt)

This scheme is relevant to my case as my source datasets are chronologically ordered. Does this make sense?

[antoinedemathelin]

Hi @omershalev, Yes, it seems to be a reasonable approach for multisource. It sounds like a "Bayesian" temporal approach where you update your model with new information ("Xs2, ys2", "Xt, yt") knowing the previous states ("Xs1, ys1", "Xs2, ys2"). Getting improvements with this approach really depends on the problem you have, I guess...

The "multisource" implementation of RegularTransferLR that we have planned is a weighted penalization of the target model based on the differences to the source models:

min_{beta_t} || Xt beta_t - yt || + lambda_1 || beta_t - beta_s1 || + lambda_2 || beta_t - beta_s2 || + ...