search

jolars / slopecd

4 stars 2 forks source link

feat: experiment setup #26

Open jolars opened 2 years ago

jolars commented 2 years ago

This is a meta-issue to discuss and list the simulated and real data setups for the experiments. These just some ideas off the top of my head. Let's discuss it!

Real Data

Gaussian

I suppose we display time to optimality curves for something like $\text{reg} \in \lambda_{\text{max}} \times \{0.1, 0.01, 0.001\}$, a little bit depending on $n$ and $p$ relationship.

@mathurinm, do you want to patch libsvm to support these breheny data sets?

Logistic Regression (if we do it!)

all the usual suspects (rcv1, covtype.binary, news20.binary, gisette etc etc)

Simulated data

Lambda sequence settings

I don't think we need to meddle much with the lambda sequence setup other than to vary the $q$ parameter, maybe something like $q \in \{0.05, 0.1, 0.2\}$.

OSCAR

In principle I don't see any reason to do OSCAR, but maybe it's clever to do so anyway since it might help draw attention to people who are interested in OSCAR. What do you think?

Competitors

Klopfe commented 2 years ago

So what would be your take on the regularization parameters, because if we do $\text{reg} \in \lambda_{\text{max}} \times \{0.1, 0.01, 0.001\}$ and $q \in \{ 0.05, 0.1, 0.2 \}$ this is a lot of different settings for one particular dataset?

jolars commented 2 years ago

So what would be your take on the regularization parameters, because if we do and this is a lot of different settings for one particular dataset?

Yeah, maybe it would be fine just to do it for one or two data sets, just the simulated data and/or move it to the appendix. I don't think it's going to make much of a difference

Klopfe commented 2 years ago

This is a meta-issue to discuss and list the simulated and real data setups for the experiments. These just some ideas off the top of my head. Let's discuss it!

Real Data

Gaussian

I suppose we display time to optimality curves for something like reg∈λmax×{0.1,0.01,0.001}, a little bit depending on n and p relationship.

@mathurinm, do you want to patch libsvm to support these breheny data sets?

Logistic Regression (if we do it!)

all the usual suspects (rcv1, covtype.binary, news20.binary, gisette etc etc)

Simulated data

  • [ ] High-dimensional setup: 200 x 20 000, 20 signals, some type of correlation structure (latent or AR process type)
  • [ ] Low-dimensional setup: 20 000 x 200, 40 signals, vary over some type of correlation structure (latent or AR process type)
  • [ ] High-dimensional sparse setup: 200 x 2 000 000, 20 signals, binary X, sparsity 0.001, some type of correlation structure (AR and/or block)

Lambda sequence settings

I don't think we need to meddle much with the lambda sequence setup other than to vary the q parameter, maybe something like q∈{0.05,0.1,0.2}.

OSCAR

In principle I don't see any reason to do OSCAR, but maybe it's clever to do so anyway since it might help draw attention to people who are interested in OSCAR. What do you think?

Competitors

  • [ ] Proximal gradient descent
  • [ ] Anderson acceleration
  • [ ] Fista acceleration
  • [ ] ADMM
  • [ ] Oracle
  • [ ] Hybrid solver (ours)

I am currently implementing these solvers in the benchmark_slope package for benchopt. I will start benchmarking on small datasets and write config files. On going work