check numerics - KHI growth rates

[PrometheusPi]

As suggested in the tests of #3227, we should test KHI growth rates in PIConGPU again. Currently it looks like the current dev produces slightly too low growth rates. This could be caused by either mismatches in the box sizes and the KHI wave length in the default setup, the use of noisy particle shapes, etc. I will have a look into that issue.

[PrometheusPi]

I just tested the KHI in a large 3D box and got too small growth rates for the eKHI regime.

Dashed lines represent the theoretically predicted growth rate of the magnetic field energy. @ax3l Do you remember when you tested this the last time?

[PrometheusPi]

@psychocoderHPC This test used the TSC particle shape - the next one will use PCS shape.

[PrometheusPi]

@KseniaBastrakova In contrast to your analysis in #3227, I did not get any negative growth rates (so far). How did you compute the growth rate?

[PrometheusPi]

@KseniaBastrakova Sorry my fault, I get the same initial negative growth rates for B_x. But otherwise the growth rate looks similarly bad.

[PrometheusPi]

In contrast to @KseniaBastrakova growth rate data, which at least are closer to theoretical predictions.

[psychocoderHPC]

[deleted my last comment] so ignore it in your mails.

[PrometheusPi]

The 3D data for PCS particles shows a vary similar behavior.

[PrometheusPi]

@psychocoderHPC offline comment might be correct - with 512 cells in propagation directions, we only resolve 1.56 periods of the KHI mode. Thus growth might be highly suppressed. I am (again) surprised how huge these waves are - didn't expect such a small fraction of the wave inside of the simulation box after I tripled the box length - we thus do not resolve a single KHI wave in the default setup.

I will test if having a integer number of waves in the box will lead to the correct growth rate.

[PrometheusPi]

I checked with 3200 cells in x direction - and a cell-size and time-step adjustment so that the 10 waves fit perfectly into the simulation box. The result is not much better:

Growth rate B_z^2:

Growth rate B_x^2:

Either I am doing something really stupid (like a bug in my analysis code) or we have a problem.

[sbastrakov]

Do you think it makes sense to try on the last release version?

[PrometheusPi]

@sbastrakov definitely - I will test older versions as soon as I get some more compute resources. Currently I occupy all k20 resources on hemera myself with a different project. (And the other resources are/should be used by other projects/people.)

[psychocoderHPC]

btw: I am currently validate the AMD results and re-run KHI with positrons/electrons with CUDA. I compared the heating against old documented runs and it looks different. I used this time 4gpus instead of 16 as normal.

new dev branch: 14d31d80e3d3b44293e3af1794fa7cbf88f735e7

old results: https://github.com/ComputationalRadiationPhysics/picongpu/pull/2275#issuecomment-331383352

Note: There was no larger difference over years and now it looks completely different. I will rerun my tests with 16GPUs as soon as we have 16K80 available.

[update] With 16GPUs it is also compleat different compared to my 2017 runs.

• PIConGPU 0.4.3 shows nearly the same plot than the current dev.
• Switching the current dev branch back to the current deposition strategy currentSolver::strategy::StridedCachedSupercells is also not changing the heating much.
• test of PIConGPU 0.3.2 fails: Boost incompatibilities, the environment on HZDR hemera is to new.

[psychocoderHPC]

The heating is nearly equal to 2017 runs. The problem is that the numerical heating plot tool is broken #3324. Heating with corrected tool: 16gpus currentSolver::strategy::StridedCachedSupercells

[PrometheusPi]

@psychocoderHPC Thank you for that input - this points to a real issue.

[sbastrakov]

What is the status of this issue? #3324 was merged, seems it should have fixed it?

[PrometheusPi]

@sbastrakov Sorry - there has been no progress with this issue - it had nothing to do with the heating tool and thus not solved by #3324

[sbastrakov]

Ah, I misunderstood the conversation, sorry.

[sbastrakov]

@PrometheusPi while investigating another issue, we suspect some setups set the initial temperature improperly. Could you check if this is the case for KHI? Maybe that explains the growth rate not matching the theory.

[PrometheusPi]

@sbastrakov Will do.

[psychocoderHPC]

We have now collisions in PIConGPU. We never resolved the debay length in our KHI setup, maybe activating collisions is required. Add collisons.pram to KHI and change the collision pipe to

using CollisionPipeline = bmpl::vector<Collider<
   binary::RelativisticBinaryCollision,
   MakeSeq_t<Pair<PIC_Electrons, PIC_Ions>, Pair<PIC_Ions, PIC_Ions>, Pair<PIC_Electrons, PIC_Electrons>>,
   Params>>;

With collisions enabled the instability is starting later, therefore you should set the simulation iterations to 4000.

[bussmann]

I want to repeat this as it pops up again and again. Adding collisions to correct wrong heating results is not right.

[psychocoderHPC]

I want to repeat this as it pops up again and again. Adding collisions to correct wrong heating results is not right.

What will be the right way?

[bussmann]

First, one has to check if PIC is valid. If si, one has rigorously check what causes the heating. For KHI I would suspect too uniform initialization and not enough particles per cell. The solution of the equation of motion at strong magnetic fields also plays a roll IMHO. So, check initialization, pusher, field solver, sim size

[PrometheusPi]

Due to the downtime of hemera, I set up the KHI test on taurus to investigate the growth rate issue again. The results for the default KHI (esKHI) now perfectly agree with theory

Either I did something stupid during the previous analysis (multiple times - very likely) or we fixed something with 0.6.0 that we were not aware of (in my opinion unlikely).

[PrometheusPi]

If this result stays persistent throughout the upcoming data analysis at other Lorentz factors, we can close this issue.

[PrometheusPi]

Just for those interested, the numerical thermalization is only influencing the growth rate up until t=12 omega_pe. Then the KHI kicks in and since its growth rate is significantly higher (about an order of magnitude) than the growth rate of numerical heating, its influence can be neglected during most of the linear phase of the KHI.

[PrometheusPi]

The effect observed above only originates from a single run. If one runs multiple runs over different hardware and different initialization routines, the growth rate is consistently too low (in the gamma=1.021 esKHI regime).

With MI being MI250x and V100 being V100, with qS being quiet start, rP being random position and rPM being random position and weighting initialization.

The error tubes gibe 2 sigma of the standard deviation of the mean. The dashed gray line shows the theoretical growth rate.

The issue is not coming from a wrong initialization - the mean particle energy per real electron is equivalent to gamma=1.021. The added temperature contributes to around 1e-6 to this value and thus has no influence on the theoretical growth rate.

The discretization should also not cause that much lower growth rate. The following plot shows the theoretical growth rate (in blue) and the possible sampling points in k-Space according to our simulation box size when using 16.cfg and the default setup.

The relative error due to the finite box size is 0.07%. and thus dies not explain the observed difference of around 12% (the growth rate is at 87-88% of the theoretical value.).

[PrometheusPi]

Furthermore, we gain energy during the simulation!!! Theoretically the kinetic energy stored within the particle stream should transfer part of its energy to the fields which than transfer this energy back to the particles thus de-and accelerating mainly electrons. The main reservoir of energy is the ion background (which also looses a bit of energy during the simulation). However if one checks the sum all energy contributions over time one can see that after initially loosing a bit of energy, we slightly increase the total energy around the peak growth rate of the KHI. Be aware that the 2% increase at the end of the simulation is with respect to the initial (all) electron energy not the total energy within the simulation since this just adds a few zeros since the mass of the ions is so much higher that the mass of electrons.