Restructure explain() for iterative estimation with convergence detection, verbose arguments ++

[martinju]

Very early draft. Lots of cleanup and moving things around remains, but the general overall structure will probably be close to what we got here.

To be done in this PR (some may be removed here and handled in separate PRs):

• [x] Add iterative kernelshap estimation: Sampling some coalitions at a time and estimating shapley values. Iterate until convergence threshold is met (based on estimated sd).
  • [X] for features
  • [x] for groups
• [x] Add bootstrapping approach estimating for standard deviation
  • [X] for features
  • [x] for groups
• [x] Add paired sampling of coalitions
  • [X] for features
  • [x] for groups
• [X] Add reweighting of sampling shapley kernel weights
• [X] Restructure non-iterative (classical) approach as a special case of the iterative approach for code simplicity
• [x] Rewrite batch_computing to be computed within each iteration
• [x] Remove all traces of iterative arguments put outside of the iter_list
• [x] Consider moving form n_batches to something like max_batch_size + min_n_batches as input argument. The former controls the memory allocation (smaller means less memory consumption), while the latter ensures we get at least some progress updates while doing the updating. An internal function could then set the number of batches at each iteration to min(n_combinations,max(min_n_batches,ceiling(n_combinations/max_batch_size))))
• [x] Remove the stop() calls based on the number of features.
• [ ] Add parallelization of the bootstrapping function
• [x] Check that the new code structure works with parallelization (via future.lapply)
• [x] Add argument verbose with arguments - Add verbose = c("basic","shapley","vS_details"), with "basic" as the default, showing what is currently going on in the function, the filename of the tempfile, and what iteration we are at (+ later estimate of the remaining computationt time) NULL or "" should give no printout at all, "shapley" means printing intermediate shapley estimates, "vs_details" means printing results while estimating the vS_functions (where this is done in more than a single step).
• [x] add save_intermediate = c("to_disk","in_memory") to allow the user so save results to file while doing the estimation. May choose to save everything to disk or just what is needed to continue adding coalitions if we are not yet happy with the results.
• [x] Add tests for the iterative approach
• [x] Set new defaults in explain(): paired_sampling =TRUE, shapley_reweighting = "on_N" ++
• [x] Carefully update all test checking files in at least 3 steps: 1. ensure classical shapley estimates gives the same answer (keeping the same order of setup_approach and shapley_setup), 2. update shapley values with the new ordering, 3. update the internal objects
• [x] Add new argument which is a list of iterative-specific arguments.
• [x] Update documentation
• [x] Update vignette
• [x] Some cleaning (fixing Annabelle's comments ++)
• [x] Add a few tests for verbose
• [ ] Possibly change the locatiosn for some of the internal parameters to make them easier to acces (like the saving path)

Note: All non-exact methods fails now (also the Shapley values estimates) since shapley_setup is now called after setup_approach. All tests for Shapley values pass if these calls are but back to the original order (but we don't want that in the future).

[martinju]

Just some notes for myself on where to catch up after the holiday:

1. Carefully check that the structure and content of the output from explain is as we want it to be at the current stage.
2. Accept all tests with the new structure such that future edits can be checked against the tests
3. Add some basic tests for the new convergence stuff.
4. Consider updating all tests with the new defaults (paired sampling and reweighting)
5. THEN go ahead and add the other features: paralellization of the boostrapping, verbose argument, disk saving, same functionality for groups etc.

Depending on how things go as I get back on this in august, @LHBO might take a look at the code structure some time after point 3 is done.

[martinju]

Slowly reaching a steady state. Apart from the list of undone tasks above, here is a list of components which is currently not in a good state

• n_combinations is currently not obeyed (and should change name)
• In what files new functions are put (this needs some cleanup)
• Documentation
• n_batches is reset within other functions to fix bug, needs to be fixed in an updated create_S_batches up front
• Reduce the number of long running tests, or alternatively, create a small set of simpler tests which are the only one I run when adding new functionality. Currently it takes way too much time running and carefully updating test objects for every update
• hide the regression_surrogate special treatment from the main code in explain()
• Add checks for which approaches that support the adaptive procedure
• MORE TO COME HERE

[martinju]

@LHBO OK, some more work done now. A lot of minor code changes as I have moved from combinations to coalitions everywhere, i.e. changed n_combinations to n_coalitions, id_combinations -> id_coalitions and so on. The key data.table X also got new (general) column names: features -> coalitions, n_featuers -> coalition_size and so on. Note that features is still added as a new column in the end, making it easier to create the binary matrix S for both groups and features. I have also added a few extra helping parameters n_shapley_values is equal to n_features for features, and n_groups for groups. coal_feature_list is the same as the previous group_num, except that it also exists for feature wise explanation.

As a consequence of the name generalizations and helper parameters, I have removed many of the almost-identical functions that are specific for groups (the old feautre_exact/group_exact). These now got the common name exact_coalition_table, just. Similar generalizations are done elsewhere.

Tests are updated after the changes. Something is wrong with groups for forecast, but we'll just ignore that for now.

Feel free to take a quick look at the main components (no need to look at the details at this stage), and let me know if you have comments to the generalizations, name changes etc.

[martinju]

Hi @aredelmeier @LHBO @jonlachmann

This is closing in on a merge. Just a few things misisng now, I think. I hope to be able to merge this some time this weekend.

Here is what remains:

• [ ] Update documentation
• [x] Update vignette
• [x] Some cleaning (fixing Annabelle's comments ++)
• [x] Add a few tests for verbose
• [ ] Possibly change the locatiosn for some of the internal parameters to make them easier to acces (like the saving path)

@jonlachmann You may want to merge this into your forecast fixing branch.

@LHBO If you want to , I actually think you can safely start on the asymmetric stuff from the current stage. What remains will not change much of the code.