Easily compare the effects of cache and permutation with flags

[lukeleeai]

This pull request adds a new feature that allows the user to easily turn on and off the use of cache and permutation. To compare the effects of these options, the user can run scripts/experiments.py with different configurations.

To install the package and run the default experiments in one go, please run the following command.

/BanditPAM/: bash scripts/reproduce_results.sh

If you want to manually experiment with different conditions, please run the following command after installing the requirements and package.

/BanditPAM/: python scripts/experiment.py [options]

If you don't pass any options, the script will run experiments with n_medoids=[5, 10] and n_data = [10000, 30000].

Options

-k, --n_medoids  int/string  default: [5, 10]
-n, --n_data     int/string  default: [10000, 30000]

Example : Run experiments with k=3 and n_data = [1000, 3000]
(Make sure to put a list in double quotes)

$ python scripts/experiment.py -k 3 -n "[1000, 3000]"

Cache (X) Perm (X)            Cache (O) Perm (X)            Cache (O) Perm (O)            

[mnist: 1000 | k: 3]
0.535 (0.041)                 0.149 (0.003)                 0.146 (0.00565)               

[mnist: 3000 | k: 3]
1.71 (0.107)                  0.738 (0.0476)                0.743 (0.0518)     

[height[bot]]

Link Height tasks by mentioning a task ID in the pull request title or commit messages, or description and comments with the keyword link (e.g. "Link T-123").

💡Tip: You can also use "Close T-X" to automatically close a task when the pull request is merged.