Lack of normalization after bundle in FHRR

[Zeldax64]

I've noticed that the FHRR class does not normalize after bundling. While some papers [1,2] adopt the approach as is in TorchHD, it is not in accordance with the definition in [3, 4], as there is no norm after superposition. Similarly, the same happens for the HRR class, which could be normalized [2, 4].

I'm aware that the documentation states that the FHRR model is implemented based on "Holographic Reduced Representation: Distributed Representation for Cognitive Structures," but I don't have access to the book to know how FHRR is defined there.

In any case, I believe some improvements to TorchHD would be a nice touch to the library:

1. If no normalization is what was intended by the authors, please state in the documentation that FHRR and HRR do not normalize after superposition.
2. Consider adding a small function to the API to normalize FHRR and HRR. I believe this would be equivalent to what "hard_quantize" is for the MAP class, as it allows the user to easily bipolarize MAP vectors without using Pytorch directly.

[1] Neuromorphic visual scene understanding with resonator networks [2] A Survey on Hyperdimensional Computing aka Vector Symbolic Architectures, Part I: Models and Data Transformations [3] Distributed representations and nested compositional structure [4] Overview of different HD Computing/VSA models

[rgayler]

Just following up on @Zeldax64 's comment, in case it's helpful to the package devs:

The handling of normalisation has been inconsistent(ish) since the earliest days of VSA/HDC. People taking a more abstract, mathematical approach (e.g. Tony Plate) tend to work with un-normalised vectors (because it makes the maths simpler) and occasionally throw in a normalisation when it's needed. People taking a more hardware-ish approach (e.g. Pentti Kanerva) have normalisation at every operator because it is forced on them by the definition of the operators. (To get the effect of un-normalised operators they have to extend the operators from binary to n-ary.)

That need for normalised and un-normalised operators extends through to today. You want un-normalised when you're exploring things because it's tedious to have to account for the non-associativity of operators that's induced by normalisation. On the other hand, if you're thinking about neuromorphic/hardware implementations you need normalisation because physical implementations necessarily have limited ranges of representation.

So, from a package design point of view, I think you need to allow for normalisation to be optional for all operators and available as stand-alone operations.

Also, there are 3 levels of normalisation:

1. un-normalised - no constraints on element values
2. normalised - vector magnitude constrained, but individual element values otherwise unconstrained
3. unitarised - each individual element value constrained to constant magnitude/modulus (this is applicable to complex-valued system, binary is really a special case of this).

There also needs to be some thought put into how to handle normalisation in conjunction with sparsity. Typically sparsity is considered equivalent to having some elements have a zero value. (It might be useful to signal sparsity with NaN values rather than zeroes, but that's a different design discussion.) For normalised vectors, as they become more sparse (more elements set to zero) the magnitude of the nonzero elements must be increased to keep the vector magnitude equal to one. I think this is probably not what people would generally want. Note also that with strictly unitary vectors that allow for sparsity (i.e. each element has magnitude/modulus one or zero if sparsified) the hypervector magnitude necessarily decreases as it becomes more sparse.

This is all just IMO FWIW, so ignore it as appropriate or ping me for a chat if you want clarification/expansion.

[mikeheddes]

Thank you @Zeldax64 for raising this issue. And @rgayler for your insights. The thought behind the current design of Torchhd is to allow for both options, so therefore we do not include the normalization as part of the binding operation. The plan was to provide separate normalization functions (like hard_quantize) but we never got around to implementing them. Now that there is interest from the community for the normalization operations I will try to find some time to implement them. I will also update the documentation to make it clear that the bundling operation does not normalize the resulting hypervector.