Rotation transform binary file incorrectly written?

[tjgalvin]

I am having a little weird behavior that I can't quiet nail down.

I'd like to use the transformation information outputted by pink with the --store-rot-flip option.

As I understand from pink version 1 and some example scripts in this repo, the binary file is written as a struct for each neuron of the form (unit8, float32).

The SOM I have is 45 x 45 x 1 neurons and I have 28220 images. The file size of this binary is 490MB, which implies that the rotation angle is written as a float64.

45*45*(8+64)*28220 / 1024 / 1024 / 8 = 490.49

Reading the data in as a (unit8, float64) leads to very erroneous angles that are certainly not correct (1e306). I would expect that the rotation angles for images to the BMUs should follow a uniform distribution between 0 to 2pi (point sources at the image center can make this a little biased I have filtered most of these out). Here I plot the angles when data is loaded as both float32 and float64

Clearly there is structure when the angles are treated as float32, with four peaks. I'm guessing this is related to the four different ways you can rotate an image just by accessing the memory in different directions.

I think the problem is in the main_generic.h file when writing the transform file. Looking at the order that the images are generated they appear to be ang, ang+90, ang+180, ang+270, but the loop writing out the information treats the angle as a j * angle_step_radians. There might have to be a + results[i] / number_of_rotations * angle_step_radians somewhere in here?

I'm a little unsure about

float angle_step_radians = static_cast<float>(0.5 * M_PI) / input_data.m_number_of_rotations / 4;

I think the the 0.5 * M_PI has already handled the trailing / 4 when calculating the radian rotation increment. I don't see where this factor of 4 has been included in the input_data.m_number_of_rotations attribute - which appears to account for the four easy ways of rotating an image.

Perhaps there is some magic between the two previous points.

I guess my questions are (1) what data type are the rotations actually written out as. The file size implies float64 but the figure and example scripts imply float32 (2) is my analysis about the rotations and the mismatch between the order they are generated and written correct?

[tjgalvin]

BTW - he is a small python module I am playing with to interact with the PINK binary files. Very much a WIP.

https://github.com/tjgalvin/pyink

[BerndDoser]

The data type for the spatial transformation file is (char, float32), see

https://github.com/HITS-AIN/PINK/blob/28a6eb49d13f25124dc02ff80bfb7649001a1486/src/Pink/main_generic.h#L191-L192

I tested a 10x10 SOM with 4000 images. The filesize should be 10*10*4000*(1+4) = 2000000 bytes and I see 2000028 bytes (28 bytes for the header).

Which Pink version are you using?

The angle_step_radians is indeed wrong and must be 2 * pi / number_of_rotations as you expected. I will fix it.

The order of rotation should be correct in my opinion, due to the offsets used in

https://github.com/HITS-AIN/PINK/blob/28a6eb49d13f25124dc02ff80bfb7649001a1486/src/SelfOrganizingMapLib/generate_rotated_images.h#L71

[tjgalvin]

Thanks for the quick response @BerndDoser. Sorry if I got any of the above wrong - it has been a long week.

I am using Version 2.4 Git revision: 6d29352 . My data where two channels. I am wondering after your comment if this is the issue, is the second channel some how coming into this?

Happy I can understand the problem well enough to pick up that little issue.

[tjgalvin]

I have been playing with this over the weekend. I had two issues at play, one with PINK and one with my code.

The angle_step_radians needed to be patched following what we discussed. And you are right, the order is indeed correct because of the offsets you used. I missed that part on the first read through. Sorry about potential confusion!

The second issue was about the file size and data type. Thank you do showing the definition, it convinced me the problem was with me. It took me a little bit to pick up, but my code is memory mapping the PINK binaries to numpy arrays so that I only have to read in what is needed. The default mode of numpy memmap is r+ and it would happily read past the bounds of the real PINK file, appending something as it went. This also had the effect of making the binary file on disk larger. By loading it just once as a float64 I instantly doubled the file on disk. Setting the mode to 'r' threw an out of bounds error and stopped the file growing.