Initial Particle IR node implementation & tests

[LonelyCat124]

Currently no creation of a Particle IR tree but the node implementations are done so far.

[LonelyCat124]

Fixed that issue.

Now we need to generate the code to handle calling the MPI communication.

Easy thing to do is "call the function". Need to generate the functor (or have it inline to be honest) as well to compute the MPI rank.

Other thing that isn't done yet and needs to be done in some way is to have a conversion from 1D to 3D "rank", e.g. to compute neighbours we need to know the 1D rank of nodes with x+1, y, z or x, y, z+1 etc. if appropriate.

This probably means the boundary object needs additional fields (n_ranks_x, n_ranks_y, n_ranks_z) and these need to be populated somehow (i.e. by whatever is doing the decomposition).

[LonelyCat124]

Ok, maybe it generates valid code, I need to have a last look through the output and then test compiling it once scafellpike is up.

NB Any fixes that were added to the miniapp are not fixed yet.

[LonelyCat124]

Generated code review (precompiler):

1. neighbour_part_rank and neighbour_part_old_position are not placed into the DataTypes object, so will not be accessible. 2.The filenames are being saved as ""file.hdf5"" instead of what they need to be.

Both are fixed now.

[LonelyCat124]

The generated code is now compiling. There are still some issues however in the static coupled system code I need to iron out.

Once that is done I need to check the python script is specifying the correct physics, and once it is then hopefully it runs!

[LonelyCat124]

Everything now compiles, but there is a linking issue with a redefinition of get_box_size and the global variables that I need to work out.

I think these functions should just be put in the main file and not in the header as they are not used outside of the main code.

[LonelyCat124]

Code compiles now with new updates for function declaration, all that I need to do now is fix the python script to generate the appropriate code.

[LonelyCat124]

Code now starts up and reads in the IO stuff, but crashes. I assume there are some errors so I will compare the generated code to the miniapp and go from there.

[LonelyCat124]

The most obvious stuff its missing is that the particle charge, mass, momentum and weight are not read in oops.

Edit: Fixed that, but still failing.

[LonelyCat124]

I think there's also a deadlock or somethign happening in the hdf5 output call I need to investigate to check the input is read correctly.

Edit: No longer appears to happen, bit weird, just gonna blame the HPC machine :)

Generated hdf5_output is missing a linebreak here: int npart = particle_aosoa.size(); MPI_Send(&npart, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);

[LonelyCat124]

Ok, so from looking at the intial output the HDF5 code reading in the data is not correct. It just gets every value as 2.1 which is clearly being read from the grid for some reason, so I'll need to examine this more.

Edit: I wonde rif this relates to the hyperslab selection code i removed.

[LonelyCat124]

There was a bug with box sizes I fixed, but the current state appears to be:

1. Read in sensible particle positions
2. Do very little
3. All particle data is becoming == bz field.
4. Output initial dump
5. Crash It isn't clear when this "everything is now 2.1" is happening so I'm trying to work it out.

[LonelyCat124]

deep copy at end of hdf5 input is wrong oops.

[LonelyCat124]

Ok, the initial output seems to work with that adjusted, just need to fix that in the real code. The code is still broken but thats next ont he list.

[LonelyCat124]

oneD_rank is not doing a return oops.

Also: ==6575==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x7f6a9a9c43a8 at pc 0x0000004d307f bp 0x7f6aa16f0a50 sp 0x7f6aa16f0a48 3 READ of size 8 at 0x7f6a9a9c43a8 thread T15 4 #0 0x4d307e in update_b_field_functor::operator()(int) const /lustre/scafellpike/local/HT04544/sht05/axc67-sht05/generated_test/FDTD_MPI_step.hpp:122

is the output with address sanitizer

[LonelyCat124]

field stuff currently isn't set (as this was previously done in a different function. I will add this in somewhere next, waiting to see if the code otherwise runs or not. -- DONE

neighbour_part_rank should be an int not a double in the particle structure. -- DONE

Error in rank update functor - comparing to x_ranks/y_ranks/z_ranks should be >= -- DONE

Not resetting tag before Irecvs -- DONE

Not resizing non-host particle aosoa after migration so deep copy fails. -- DONE

Not sure that hdf5_output works on 1 rank at the moment. Not recreating the SimdPolicy for after particle migration.

Not changing rank_slice(end) to -1 after moving the data from it. -- DONE

Bug in MPI send for neighbors other than the first. Investigating. First issue is all the .extent(0) should be .extent(1). Other issues investigating still. I think that was only bug in initial MPI call -- DONE i think

Some particles are ending up with position value 0.0 for some reason. These all seem to be at the end of the particle array after communication so need to check this. Bug is in sending/recving multidimensional data. Needs to be extent(1) extend(2/3/4) i etc -- DONE I think

Still having issue when increasing the size of the particle structure, sometimes the last particle doesn't appear to be getting the correct value. I think this is to do with end not pointing to a removed value in that case. I need to check the original code its based on I think here. -- DONE i think

The manually edited code on scafellpike now appears to be working, so I need to update the DSL to generate the correct code next.

[LonelyCat124]

Newly generated code now runs without optimisation, but is still crashes otherwise.

I've also not fixed the issue with single node HDF5 stuff yet.

[LonelyCat124]

Ok the bug now appears to be wanting to send all of the particles - need to fix.

[LonelyCat124]

Fixed the issue, and the missing scatter_view code. Hopefully everything is present now. Checking correctness of the example next.

Remaining known issue: Not sure that hdf5_output works on 1 rank at the moment.

[LonelyCat124]

Momentum is definitely not being conserved so there must be a bug in either the field solver or the particles. I'll go back to the old testcases to check.

[LonelyCat124]

Momentum and Electric field end up wrong. I think there's likely an issue in the interpolation that I need to check then because I don't see anything else, but the standalone repo will have all the changes i've made in commits I hope.

[LonelyCat124]

dt isn't computed correctly is one issue.

Edit: Fixed

[LonelyCat124]

The grid-only test is failing, so there's likely an issue somewhere in the grid solver doh.

[LonelyCat124]

cx1 is wrong (about half what i think it should be).

Edit: Its wrong the first time its called but then correct ??

Edit 2: Code is incorrectly calling the FDTD solver instead of bcs during initialisation

[LonelyCat124]

With the correct initialisation call the grid solver is a LOT closer to crrect. There's still some lingering tiny drift but I think the original code has similar and this is likely caused by rounding or similar (its 1e-7 or less).

[LonelyCat124]

Ok, with that fix the momentum is conserved so its probably the only remaining thing to fix.

[LonelyCat124]

Remaining issues:

1. Program crashes if there are no particles (not sure we care but worth thinking about)
2. Not sure that hdf5_output works on 1 rank at the moment.

[LonelyCat124]

hdf5_output currently working fine on 1 rank in a test.

[LonelyCat124]

One bug in the pir_cabana visitor is it creates a templated functor even if there are no template arguments.

Edit: Fixed.

[LonelyCat124]

Coverage done. Docs next!

[LonelyCat124]

There's some documentation now. One obvious issue in the getting started (for example) is inline code in rst is not done with "`". Instead I need to find out what I should be using (maybe _) and use that instead.

[LonelyCat124]

The code is `` so can do that.

I used sphinx-autodoc to generate the rst files for the python code - what I want to do is divide that into user and dev functionality, so i'll work at that next

[LonelyCat124]

Ok, tests are failing now (need to check why) but the documentation is starting to take shape around the pydoc generation. The basic api pages for user-code are starting to take place, and I will continue with that.