Implementation of P-pipeline for True-False Dataset (training classifiers)

[fxnnxc]

P-pipeline framework

• see the code labs/p_pipeline/run.py

    1. create raw dataset 
    2. load hiddens 
    3. create probe model
    4. train the model 

The base prober is the mean pooling prober. You can implement any forward logic by inherit MLPProber

class MeanPooledMLPProber(MLPProber):
    def __init__(self, gpt_hidden_size, linear_hidden_size, linear_activation, linear_n_layers, num_outputs, **kwargs):
        super().__init__(gpt_hidden_size, linear_hidden_size, linear_activation, linear_n_layers, num_outputs, **kwargs)

    def forward(self, x):
        x = x.mean(dim=1)
        return self.net(x)

[ojoo-J]

231127 update

It has been updated to the youngju branch.

Arguments

• --using-threshold: Set the final output to 1-dimension and apply sigmoid.
• --threshold: Any prediction above the threshold is considered true.

+) Using the Thresholds-Method requires a way to determine the threshold value.

Probing Models

• MeanPooledMLPProber: Average representation of all tokens
• LastPooledMLPProber: Representation of the last token
• TokenWiseMLPProber: Average of logits over all tokens

+) For debugging, default values have been specified for all arguments.

[fxnnxc]

The first and the last tokens are EOS tokens ([0] and [-1]). Please use [-2] to get the exact last token.

# 8 length tokens
[EOS] [EOS] [EOS] Today weather is good [EOS]
[EOS] [EOS] [EOS] [EOS] Today is good [EOS]