bug: Potential loopholes in the formulation of your claims

[Trent-Fellbootman]

Hi, I agree with most of your claims but noticed a few loopholes in how you justify them. Your math work is easily verified but I'm afraid it fails to generalize to the real world.

Specifically, here's the actual pillar your entire paper depends on:

Recall that our idealized world consists of bijective observation functions, which, over discrete random variables, preserve probabilities. So we have:

$$ P_{coor}(x_a, xb) = P{coor}(z_a, z_b) $$

$$ K_{PMI}(x_a, xb) = K{PMI}(z_a, z_b) $$

You never really introduced anything regarding your major claim, i.e., "different modalities converge to equivalent representations" before introducing these equations; these equations are what immediately justify that claim. And if this step is wrong, the logic of your entire paper collapses.

Unfortunately, although this "bijection implies equivalent probability distribution" thing is mathematically correct, it does not generalize.

You're formulating this in the discrete case, where it's true that bijections don't change probability distribution.

However, this is not true in the continuous case. For example, $tan$ is obviously a bijection from $(-\pi/2, \pi/2)$ to $R$ but clearly does not preserve probability density, cuz all sophomores know that you gotta factor in $f'(x)$ when deriving the distribution of $f(x)$ from that of $x$. The probability distributions are equivalent only when that bijection is linear.

In context of your paper, this basic result from probability theory means that when it is hard (or low-frequency) to represent something in one modality but easy in another, the probability distribution diverges. For example, we mention the word "brain" quite frequently in speech, but we don't see pictures of brains as often, cuz it's easy to talk about brain, but not so easy to open someone's skull and take a picture.

Unfortunately, your experiment with colors doesn't really help: since colors are relatively easy to either describe or appear in images, you would obviously expect little to no discrepancy in corresponding distributions on the text and image modals. But if you do another experiment with abstract concepts that can't be pictured at all (like "ideology"), the results will likely be very different.

The fact that information are stored discretely in computers doesn't help either. When you're quantizing something, the probability density gets baked into the tokens, and the bijection assumption no longer holds.

I think this is a serious issue with your paper cuz if what I said above is correct, everything in your paper basically gets invalidated. If even a third year undergrad (me) who was rejected by MIT (twice) and a bunch of other universities can see the problem, I guess a lot of people can as well.

[Trent-Fellbootman]

Btw, consider adding version specifications in your requirements.txt so that your code continues to work in the future [lol]

[phillipi]

You're right that our paper does not prove convergence in the real world. Instead we are just stating a hypothesis.

Section 4, which you refer to, is a highly simplified model which we think might be a starting point for getting theoretical traction. As we mention in the paper, it doesn't hold for continuous variables, and has many more limitations.

The main evidence in support of the hypothesis is not the model in Section 4, it's the experiments and lit review in the rest of the paper. From first principles it's possible to argue that things will converge or that they won't, and the outcome depends on your assumptions. Yet empirically we see evidence of convergence, and that's why we highlight possible explanations that are consistent with convergence.

In any case, lots more to do here before we have a complete theory of the phenomena we are observing.