compdemocracy / polis

:milky_way: Open Source AI for large scale open ended feedback
https://pol.is
GNU Affero General Public License v3.0
719 stars 170 forks source link

Unexpected behavior in beeswarm (consensus vs divisive) visualization #1726

Closed Simon-Dirks closed 6 months ago

Simon-Dirks commented 8 months ago

Expected behavior: On the far right side of the visualization ("divisive statements") I'd expect to see voting behavior that differs greatly between groups/participants. E.g., group A votes mostly agree for a statement, group B votes mostly disagree. Or: Overall there's a (more or less) 50/50 split between participants on agree/disagree.

Actual behavior: The most divisive statement according to this visualization shows fairly similar voting behavior across groups (in this case: most people seem to disagree with the statement), as well as overall.

I found statements further to the "consensus" side of the graph that showed (at least to my eye) much more divisive voting behavior (see screenshots below).

In fact, the most divisive statement (statement 1) is even listed in the "Majority" view, showing what most people agreed with.

To Reproduce: We simply ran a Polis conversation, I understand that the exact voting behavior might be hard to reproduce.

Screenshots: image ^ A statement that should be the "most" divisive, but shows fairly similar voting behavior across groups.

image ^ The most divisive statement being listed in the "Majority" view.

image ^ A statement that seems much more divisive to me, but is further to the left on the scale.

Device information: Across all devices/platforms.

Additional context: 193 participants voted 4512 votes were cast 23.38 votes per participant on average 9 commented 66 comments submitted

Note that I removed the statement texts (I'm not sure if I'm allowed to publish them at this point).