Updates and Questions on Scalar_Delta / Cosine_Similarity_Random_Index

[jscuds]

@msbutler , I thought I'd be able to finish both of these...but I ran into some issues with the latter case.

UPDATES:

_Scalar_DeltaDemo:

• I changed LUNA to include the dictionary grad_func_specs we discussed. I added the comment #JACK to anything I modified for easy identification and/or removal.

• fixed corresponds to a scalar step. I used your LUNADemo parameters, and then added my dictionary. I graphed 3 different cases compared to the LUNADemo case.

_Cosine_Similarity_RandomIndex:

If you have 20 minutes to go over the cos_sim_sq…I don’t think I understand the implementation like I thought I did.

• [x] How many indices are enough? Is the LUNADemo only using 1 output dimension (I know what it says in “luna_architecture” but if cos_sim_sqis based on range(D_out) I feel like it should iterate through 5 aux_functions…but the loop only seems to iterate twice).

• [x] Should the random indices just be choosing 2 random aux functions to compare? Or should there be a user-defined set of the number of pairwise comparisons to make?

General Question Related to Performance Measures:

• [x] Is there a way to get MSE of the final result? I see that it’s used in the iterations, but I was looking for the final MSE so we could compare models. I could generate a list or print statement...but I was wondering if there's an attribute that I'm missing that has it stored after running the .train() method.

@vsavram @mecunha for your awareness.

[msbutler]

@jscuds

1. What do you mean by "how many indices are enough"? Like how many observations should we sample? I would make that a parameter passed into self.grad_func_specs.

We don't need to iterate for each aux_function because ff.forward evaluates every aux_function all at once. The only loop in default_finite_differences iterates through the input dimensionality, but that functionality doesn't currently work.

I think all you need to add to default_finite_diff is something along the lines of:

if 'random' in self.grad_func_specs:
     random_indices = np.random.int(0,x.shape[1], grad_func_specs['random'] ) # where, grad_func_specs['random'] equals the number of indices we want to sample
     x = x[:,random_indices]

2. All the aux_functions should be evaluated, regardless of which observations we sample. You shouldn't change similarity_score

3. We currently don't have that functionality. @mcembalest @mecunha we should add functionality to spit out the MSE for ff, NLM, and LUNA. Ideally in the 24 hours.

[msbutler]

@jscuds awesome work with the finite diff demos. Some feedback:

Scalar Delta Demo

• your code in luna.py looks good
• to prevent errors, use the config and remove all the cells that specify stuff that's already in the config (cell 3)
• at least on a small number of iterations, there seem to be some interesting results!! smaller step size is slower to run, but may yield better uncertainty estimates.

Proposed next steps:

• recompile everything once you're sure that you're training on the info from config (i.e. with max iter = 100)
• compute the log likelihood for each model

• arrange the plots in a multi plot figure. See NLM.demo. A 3 by 1 plot would look good. Some code to accomplish this:

  fig,ax = plt.subplots(1,3 ,figsize=(15,5))
  i = 0
  for model in lunas:
  predictives, predictive_samples = model.predict(x_test)
  log_l = model.get_log_l(x_train,y_train,x_valid,y_valid)
  name = f"Luna: Diff = {diffs[i]}" + f"\n Log Likelihood: {round(log_l,2)}" + f"\n Time: {round(times[i]/(60*60))} Hours"
  ax[i]=bh.viz_pp_samples(x_train, y_train,x_test.flatten(),predictive_samples,name, ax[i])
  
  i +=1
  plt.legend()
  fig.savefig("figs/LUNA_fixed_finite_diff.png")

• remove luna base (as we already have results for this), and run the 3 fixed finite diffs with 3500 iterations, should take 3 to 4 hours. Maybe you ask elaine to run it.

Gonna review the sampling thing now.

[msbutler]

@jscuds

On the sampling analysis: again, this looks great. A few things to note:

• all the feedback above applies
  • in luna.py, when you specify in randint the number of interegers to sample, use math.ceil instead of round, because when grad_func_spec["random"]is 0.1, we want to make sure we get at least one sample.
  • if you haven't already done so, can you test that randint will draw a different sample everytime? just add print(random_indices) below the line to test, and then remove once confirmed
• this log-likelihood vs percentage graph won't make sense until we run it on 3500 iterations
• for this demo and for fixed_finite_diff: make sure you collect all the runtimes in a list!
• running 6 iterations of this on 3500 iterations will take 8 hours, so an overnight run is in order.