search

multimodal-interpretability / FIND

Official implementation of FIND (NeurIPS '23) Function Interpretation Benchmark and Automated Interpretability Agents
https://multimodal-interpretability.csail.mit.edu/FIND-benchmark
46 stars 3 forks source link

Noisy and corrupted functions #1

Open KeremZaman opened 1 year ago

KeremZaman commented 1 year ago

Hi,

Thank you for the great dataset and repo!

I downloaded the dataset, each function has some parameters for corruption and noise in data.json for the numeric subset, but when I look at function_code.py files only some of them impelments noise or corruption. This is expected because the paper says only a small percentage of functions has corruption or noise. Since we have noise/corruption-related parameters for each function in data.json, I can't see which function actually implements them without examining each code file individually.

Is there an easier way to group functions with respect to whether they are corrupted or not / they are noisy or not?

tamarott commented 1 year ago

Hi!

yes, this information appears in numeric/func_name_dict.json. The list state what is the source code for each function, where function_code.py is for a clean function, noised_function_code.py is with additive noise, corrupted_function_code.py is with a domain corruption, and load_mlp is an mlp estimation of the clean function. Eventually, in the dataset itself, each is implemented in a function_code.py file, but you can see what was the source code using this dictionary.