Separate linguistic vs. non-linguistic pushback

[CaptainEmerson]

Per our discussion Monday, Bogdan suggested seeing whether the classifier can distinguish linguistic vs. non-linguistic pushback using linguistic cues. Here are the pushback types:

[CaptainEmerson]

Before I do this, questions for @bvasiles:

• Which of the above types would you consider linguistic?
• Would we exclude non-pushback CLs? Or lump them in with the non-linguistic pushback CLs?

[CaptainEmerson]

For @sophieball:

• Shall I run train_classifier_g?
• Which output files would you need?

[bvasiles]

@CaptainEmerson it's a bit hard to say, but probably linguistic are at least:

• all "harsh communication",
• at least "confrontational comments" and "aggression" under "coercion"
• all "personal attacks"

What do you think?

[sophieball]

Here's the list I used; we can refine it:

reas = ['Curtness',
       'Unnecessary pressure to make changes',
       'Attempts to undermine effort', 'Attempts to undermine work',
       'Aggression', 'Against spirit of open source software',
       'The author would delete their own comments regularly. This caused confusion because different people operated with different information and sometimes there were conflicting facts presented in the comments.',
       'Humiliation',
       'Intimation of negative perf repercussions',
       'Teasing',
       'Belittling', 'Harsh feedback',
       'Confrontational comments',
       'Dismissive attitude towards the need for my change',
       'Lack of willingness to understand the purpose and reason of the change.',
       'Attempts to undermine personal integrity',
       'Requests for change without clear explainations, knowing the PR author is new to the project',
       'Inappropriate jokes', 'Unjustified criticism', 'lack of understanding',
       'Attempts to demoralize',
       'Requesting a change without justification']

[sophieball]

For @sophieball:

• Shall I run train_classifier_g? Yes, the way I run it is just replace the dataframe you feed into train_classifier_g by the one containing only linguistic issues

• Which output files would you need? The same one, train_classifier.log

[sophieball]

Before I do this, questions for @bvasiles:

• Which of the above types would you consider linguistic?
• Would we exclude non-pushback CLs? Or lump them in with the non-linguistic pushback CLs? I used NON-LINGUISTIC pushback CLs vs. linguistic pushback CLs as negative and positive data points. I can also try NON-PUSHBACK CLs vs. linguistic pushback CLs. @bvasiles What do you think?

[bvasiles]

@sophieball I’d probably try non-linguistic vs linguistic first

[sophieball]

ok. that's what I tried so far

[CaptainEmerson]

@sophieball Just to make sure I understand you: you've run linguistic vs non-linguistic for github pull requests, where https://github.com/sophieball/toxicity-detector/issues/104#issuecomment-870165027 refers to your linguistic categories, correct?

[sophieball]

Right.

In our case, each PR may be associated with multiple pushback reasons. I turned pushback reasons into one-hot encoding. Then I sum the linguistic columns commented here. Therefore, if any of the linguistic columns is 1 then I label this PR as linguistic-related pushback, otherwise, non-linguistic pushback.

Suppose the new column, linguistic pushback, is the new label, then a subset of your pushback CLs will be labeled as TRUE; the rest of the pushback CLs + non-pushback CLs will become FALSE. Then you run train_classifier_g. The result will tell us how well is the classifier doing with linguistic pushback.

[CaptainEmerson]

The results are here, shared with @sophieball.

Similar to Sophie's classification, I used the following to signal linguistic pushback:

          attempts_to_undermine_personal_integrity or
          curtness or
          attempts_to_demoralize or
          confrontational_comments or
          harsh_feedback or
          intimation_of_negative_perf_repercussions or
          inappropriate_jokes_about_a_googler_or_googlers or
          harsh_communication or
          requesting_a_change_without_justification or
          unjustified_criticism or
          attempts_to_undermine_work or
          belittling or
          humiliation or
          attempts_to_undermine_effort or
          teasing or
          intimation_of_disciplinary_procedures or
          aggression or
          unnecessary_pressure_to_make_changes

The ones that were "non-linguistic" were any reviews where those were false. Note that we're working with a more limited data set here -- the only reviews for which I have these fields set are for the "volunteered CLs". That means, roughly, that we have roughly 144 reviews in total, of which 91 are linguistic pushback CLs.

[sophieball]

@CaptainEmerson In the results folder, there should be 3 files: pos_label_data_stats.csv, neg_label_data_stats.csv, and data_stats.csv Can you share them as well?

[CaptainEmerson]

Yep, shared with you.