This PR does several major things to the repository:
It removes support for CCS and VINC, and removes the use of contrast pairs
The InferenceServer class is used to do inference, enabling FSDP (--fsdp) inference with much larger models
It also only requires loading the model once, and doesn't load the model if the dataset is cached
Only support LR probes and LM predictions
LM predictions are taken in response to statement + suffix where statement is the statement we're extracting hidden states from and suffix is a piece of text like "\n\nIs the above statement true or false?" that's optionally specified in the yaml template file
logprobs are optionally stored with --save_logprobs
support for non-balanced datasets with --balance False
Added testing for:
The pipeline that turns the raw HF dataset into the input_ids dataset passed to InferenceServer
InferenceServer
default template called "_default" which takes the "statement" column itself
This PR does several major things to the repository:
InferenceServerclass is used to do inference, enabling FSDP (--fsdp) inference with much larger modelsstatement + suffixwherestatementis the statement we're extracting hidden states from andsuffixis a piece of text like "\n\nIs the above statement true or false?" that's optionally specified in the yaml template file--save_logprobs--balance FalseInferenceServerInferenceServer"_default"which takes the"statement"column itself