bonsai file format variables, placing

[twobombs]

I have written a conversion script at https://github.com/twobombs/qracknet/blob/master/scripts/makebonsai.sh and I see that there are differences in the notation of the values. Could/can someone point out what the position of the variables are in the converted model3_child_compact bonsai hex file ? This would help great deal in setting the variables in correct order.

i think there is a big endian notation might be a thing going on in the first value, the directional speed for the imported quantum points are all zero at the moment.

^ the converted file from ascii output to binary hex

before the file gets converted to binary, this is what the variables look like in hex-ascii. it has ID, x, y, z, m, zd,yd,zd preceded at the top with the amount of points ( only a small subset of the actual quantum data )

Note: I've opened a separate ticket because the original thread became OT.

[jbedorf]

I'm not quite sure what the question is here. We never work on the binary data of the *.bonsai files directly, that is all handled by the MPI-IO libraries. As such we have no real insight into how the data is ordered or organized as it will be set up and organized for performance. It furthermore depends on the number of processes involved in the writing.

[twobombs]

Reverse engineering the file format showed me initially that it shouldn't be too hard to create a script that sets the data right from a raw ASCII output to bin with values, masses.

Tipsy requires specialized tools and suites that are IMHO both manual and cumbersome for near realitime purpose. I have an output stream of 4 to 20.000 pieces of measured data per second that I would like to have displayed in 3D space.

So there goes my 0.02cts analyzing this data. 🔢

The green outline shows $204E that is the little endian notation of 4E20 that means 20000 in decimal. The model3 file has 22000 points that are displayed, I suspect the bulk of the data is declared there. The other 2000 points at a later stage.

From thereon it goes fairly repetative with numbers, declaring the position and displacement ( which I assume to be 0 in all 3 axis for all points displayed as I see no motion in Bonsai when displaying this data )

So the array would be declared as follows:

00 C2 EB 0B 00 00 00 00 01 C2 EB 0B 00 00 00 00 02 C2 EB 0B 00 00 00 00

The specified format is

ID ( declared in header )

X Y Z M dX dY dZ

I suspect that the dX dY dZ are the 0x00 00 00 00 entry

This leaves X 0x00 Y 0xC2 Z 0xEB

With 0B presumably M. But from what I gathered you're using int[3] values, and this notation is byte sized...

So, as this seems that this is not a road to be travelled, I might start to look for alternatives.

Thank you for the insight from your side. 👍

[jbedorf]

Ah yes, the tipsy file format is straight format. The files with ".bonsai" extension are done using MPI-IO and not straight forward, the file you are looking at seems to be mislabelled, hence my confusion.

The tipsy format is just:

• Header
• Set of darkmatter particles
• Set of star particles.

This is what you see in your file. Tnere are 20.000 darkmatter particles and 2000 star particles. The various structures are defined here: https://github.com/treecode/Bonsai/blob/master/runtime/include/tipsydefs.h We do not use int[3] , but real/float[3]

I'm not 100% sure how your hexeditor looks at the "model3_child_compact.tipsy" file, but this is what mine gives::

hexdump model3_child_compact.tipsy | head

0000000 0000 0000 0000 0000 55f0 0000 0003 0000
0000010 0000 0000 4e20 0000 07d0 0000 7f26 0000
0000020 4201 3b91 03ca bc46 0b96 bd28 3c96 bb8f

So:

struct dumpV2 {
    double time ;  "0000 0000 0000 0000" -> Time is 0.
    int nbodies ;  "55f0 0000" -> 22000
    int ndim ; "0003 0000" -> 3
    int nsph ; "0000 0000" -> 0
    int ndark ; "4e20 0000" -> 20.000
    int nstar ; "07d0 0000" -> 2.000
    int version; "7f26 0000" -> I don't think this was set for this file, we only introduced it later so might be random data.
} ;

[twobombs]

thanks for the reply. The file I'm looking at there is the bonsai version, converted from the tipsy file I'll look some more at this with the tipsy specs in hand to make a choice in this

[twobombs]

Thank you very much for your feedback on the header. I've started on the conversion to tipsy instead of bonsai data format could you state the c-style format of a tipsy data point entry as it seems to differ from the previously mentioned bonsai specs ( I'm greeted with an error when converting to bonsai for sanity check so I'm expecting that the point data isn't up to spec )

I can't seem to get the datapoint spec right when looking into the source. https://github.com/treecode/Bonsai/blob/master/tools/IO/cvt_tipsy2bonsai.cpp

right now I use x,y,z,m,dx,dy,dz in 0x00000000 4-byte hex format

with header it looks like this in ascii ( before translation to hex )

[twobombs]

Hold that thought. My import scripts made the amount of lines/data too high, as it does not clip the data properly so the amount of lines is inconsistent. Will work on the consistency of the tipsy input.

The read route of the cvt_tipsy2bonsai might not like that there comes more data after the amount of data that was announced. ( a tipsy tools request reminded me to check that https://github.com/N-BodyShop/changa/issues/20 )

[jbedorf]

Have a look at this file. It contains the definitions of the Tipsy objects that we expect when reading the files.

right now I use x,y,z,m,dx,dy,dz in 0x00000000 4-byte hex format

For dark matter it should be: m, x,y,z, dx, dy,dz, ID All 4-bytes, except ID which is 8-bytes.

For star particles it is: m, x,y,z, dx, dy,dz, dummy, dummy, ID All 4-bytes, except ID which is 8-bytes.

[twobombs]

Thank you for the specs, saves a lot of trouble searching :) The 8 byte ID is that the type of matter ( 2 for starts etc ) or is it the ID as a number of the object in the amount of objects declated in the header ( 1, 2 ,3, 4,5,6 etc )

[jbedorf]

Both. It is a unique ID per particle that can be traced through out the simulation. Allowing you (for example) to correlate multiple snapshots over time. But we internally also use it to keep track of the type of matter, such that we have that at hand during writing.

See: https://github.com/treecode/Bonsai/blob/master/runtime/include/IDType.h (getID/setID, getType/setType)

[twobombs]

Thank you for the info. The conversion data now aligns and the cvt_conversion does parse.

However the tipsy programs that I use for tests are all crashing with an OOM error, including bonsai. I think I know why. All HEX values are stored reverse on disk. An example of the mode3 file variable specs that you provided shows this:

I've started work on reversing the byte-order of any and all hex that gets pushed into the output file. Tnx again for pointing the way, very happy that this is starting to gain momentum :)

[twobombs]

I've got the file build routines all setup. There is one nagging thing though: I need to convert HEX/DEC INT lists to HEX Float in bash/CLI... do you know a way to convert that in bash/awk ? Its kind of an obscure area :) If I leave it at INT nothing is displayed.

edit: looks like i've found something already. Even though I'm not happy with using a script language interpretor as executor (slowww) this python line should do the trick.

python -c 'import struct; print "%#x" % struct.unpack("I", struct.pack("f", 1234))'

At peak billions of data points will traverse the script, right now at a 'mere' 3.2M datapoints takes 30 seconds on a regular CPU. Maybe awk or bc might be helpful also, but we'll see. The fastest wins.

[jbedorf]

Maybe you can use the printf functionality of bash to achieve the same?

[twobombs]

Maybe you can use the printf functionality of bash to achieve the same?

Tried that, gives an incompatible output.

printf %A 1234
0X1.348P+10

Will look at bulk import for python to recoup some of the losses.

[twobombs]

I'm quite a bit further into the Tipsy file format, when this is all said and done I'll be able to organise an Ask-Me-Anything-on-Tipsy-fileformat meetup.

I've got visuals on nbody ( nvidia SDK demo ) but when I run the/a generated tipsy file on bonsai I get the error below:

Apparently Bonsai does not like negative coordinate values and runs into an OOM which is kinda weird because at ~114k points it should still have a relative small memory footprint. ( target is a NVidia Tesla K80 12GB GPU )

Update: file format needed was big endian, needed to be converted to little endian. The file now loads, but has some weird properties. ( dark matter and/or very small mass ) Will tinker with that. Memory usage was 1300MB for 13k parts, so the OOM was right; 114k points is too much. The K80 has 12GB vRAM per GPU core. Will look into sharing the memory load between the GPUs.

[jbedorf]

It should have no problem with 114k points. Bonsai allocates quite a but of memory at the start for certain buffers, hence you see 1.3GB for 13k particles. On 12GB GPUs I've had it running with over 5million particles without problem. The particular issue in the screenshot above might be that the tree domain got too large because of the endian problem. Try running bonsai with the --log command to get additional output.

[twobombs]

Thank you, indeed, this creates a lot of room for a huge amount of particles. I also found a tweak that separates the rendering from the calculating device, and at the cost of GPU and fps one can win another ~280MB

The log shows that the points are read, but no levels nor leaves are build. That seems logical to me as this is basically a huge amount of random data placed in a cube.

But I don't see anything displayed unfortunately. Will continue work on that.

[jbedorf]

I think the issue here is the huge values of your coordinates. Try reducing your domain size (e.g. make a smaller cube in which you randomly place them). A too large domain means that the tree-build won't work as particles are too far and wide.

[twobombs]

Placed the datapoints 2^16 closer to each other and now groups are forming. Still see no data visually, could be that the mass of the points are too low to make visible or somehow the type is wrong. Thanks for the hint, will keep on looking at this.

[jbedorf]

That looks better yes. Still some large ranges but given that the tree is formed that should help. You can try to use the point visualizer and/or visualize the datastructure. To switch the visualization mode use the 'p' key and for the tree-structure use the 'b' and the +/- keys to increase/decrease the level drawn.

[twobombs]

After a long time I found some time to do again start working on the Bonsai/NBody visualisation. And I feel I'm stuck at a small thing. Maybe you know what's up here. Either the coordinates/ranges of the dataset is so big that it's a black screen, or too small that it's invisible. Either way, trees are build, but compared with the example there are some differences.

Pressing B or any of the other visualisation tools do not seem to work. ( no boxes visible ) So I think that it's either very big field or a very small, dense one. Can you see that in the logs ?

Log of my dataset

Log of Model3 dataset in Bonsai repo after compilation

Corner and size of model3 deviate a lot compared my dataset. As you can see with added gravity the tree does build reasonably well.

What do the variables corner and size mean ? I think that could point the way here.

[jbedorf]

The results look identical, including number of particles are you certain you copied the results from two different runs and not the same screenshot twice?

[twobombs]

Indeed. Now that was dumb :) Here are two new ones

Model3 example

QFT output

The QFT data has star data only, no dark matter.

[jbedorf]

Although the tree is created the box size in the QFT example is huge. I suspect that the visualization is zoomed out so far that you can't see the actual bulk of your particle set somewhere far in the center. What you can try is to change the rendering mode ('P') to cycle through the various visualization methods, maybe something becomes visible that way and then you can zoom in and see if something is visible. Or see if you can remove the outliers (?) in the dataset for initial testing.

The box-size of the example dataset is basically 24x24x24. Yours is: 133 million x 26524 trillion (?) x 4 billion

[twobombs]

Thank you for the analysis. In the calculation of the coordinates I've included the range at which the QFT particle creation occurs. For placement I used the outer ranges of a 32 bit Integer and placed it in a rectangle with the measured values of the 32qbit as mass, the amount of qubits also as an axis. In this way particle placement is very diffuse yet (semi) correct. The dimensions of the 24x24x24 example box is very helpfull. Will look into downsizing the x-y coordinates and tradeoff with precision behind the comma. Thank you, thank you, thank you :)

[twobombs]

After compressing the first slice of Hilbert space is visible. Thank you, thank you, thank you.

[jbedorf]

Great! Glad it is working!

[twobombs]

Closing case.