species.param

[cbontoiu]

I obtained conflicting results for two targets of the same initial charge probably because of an accidental difference in the species.param as shown in the image attached.

model tubes: The image on the right is from a project which uses a target made of tubes (50 nm in diameter) distributed in a hollow thick shell (a few microns wide).

model effective density: The image on the left is from a project, where the same hollow thick shell is filled with plasma so there is no granular structure.

The mesh cell was 20-30 nm with 3 ppc. Both models showed exactly the same total charge at t = 0, as they were supposed to do, but while for the effective density model the total charged increased smoothly to a maximum of about 4 times the initial charge, the tubes model showed a jump from the initial charge to 6 times more, as if the whole target was completely ionized between the first and second iterations. The same discrepancy was obtained for the total charge calculated from 'e_all_chargeDensity' and from macroparticles.

Now, is the species.param needed? I see that the LaserWakefield model does not have it, while the atomicPhysics model does and it includes using UsedField2Particle = FieldToParticleInterpolation<UsedParticleShape, AssignedTrilinearInterpolation>;

[cbontoiu]

Even after adding using UsedField2Particle = FieldToParticleInterpolation<UsedParticleShape, AssignedTrilinearInterpolation>; in species.param I get the inconsistency although the initial charge values are equal

Here is how the tubes target looks in the transverse cross-section, but it is a 3D structure extruded along y.

[PrometheusPi]

Hi @cbontoiu, if I understand your first comment correctly, you accidentally added or missed the using UsedField2Particle, right?

If I understand your second comment correctly, you added using UsedField2Particle but the result still differs, right? Could you elaborate what you expected, because this is not clear for me from your plots and description.

[cbontoiu]

@PrometheusPi thank you for your reply

The electron charge density of the two models can be seen in these images at iteration 0, 1 and 160:

I would expect the core of the target in each case to be smoothly ionized as the laser pulse advances through the structure. The total charge of the effective density model and the tubes model should grow smoothly although not along identical graphs. For the tubes model the charge density has nonphysical jump already at iteration 1.

I first though this is due to accidentally missing using UsedField2Particle in the tubes model, but adding it didn't not change the results.

I attach the faulty model. input.zip

All tubes have the same constant base density 1e28 1/m^3.

[PrometheusPi]

Okay - thanks for the clarfification. I still have some questions:

• The left plots show the effective density model the right shows the tube model, right?
• For a single time step, you show 4 plots, 2 for the effective model and 2 for the tube model. What are the differences between this upper and lower plots?
• Why are the (high density) tubes (right side, lower plot) asymmetric with respect to the central axis?
• Where do these small high-density fluctuations already at t=0 come from in the tube model (right side, both upper and lower plot)?
• What do you mean with "tubes model the charge density has nonphysical jump already at iteration 1"? (How do I identify these jumps? - For me 0 iterations and 1 iteration look density-wise identical.)
• If I look at the (black) laser curve (?) at iteration 1, the laser on the (right side) tube model (?) looks around 20%-30% stronger than on the left-hand side (effective density model). Is this intended?

I am a bit confused with the resolution numbers you are stating above in the plots. Assuming a temporal resolution of:

delta_x = 20.348e-9
delta_y = 20.161e-9
delta_z = delta_x

should require a temporal resolution below 3.9e-17 seconds, but the plot states 9 fs - thus I am clearly misinterpreting something here, since this would fail the CFL condition and thus throw a compile-time error. Could you elaborate on how to interpret the resolution values above the plots?

[cbontoiu]

• The geometry is 3D in both models so there are two planes to show yx (top) and yz (bottom);
• Carefully looking at how the tubes are disposed in the third figure of this thread shows that the two axial planes yx and yz cross the tubes differently;
• The tubes are distributed at a constant azimuthal gap around the clock but they don't necessarily have a mirror symmetry, especially for the innermost shell;
• I think the small charge density fluctuations come from the fact that the tubes fall differently into the 3D mesh grid with the tube diameter being 50 nm and the mesh size 20 nm;
• In my previous message the charge density of the tubes model at t = 0 (iteration 0) peaks at about 2.5 C/mm^3and at about 15 C/mm^3 in the next iteration so this is a jump, while the laser, merely entered the left PML region;
• Yes, all density model images are on the left and all tubes model images are on the right;
• The laser pulse is absolutely identical in both models; there cannot be an error here because the workflow is automatic and each image title shows the simulation conditions as saved when the model was launched for simulation;
• There is an explanation for the different amplitude of the laser pulses: the effective model quickly absorbs the pulse energy into ionization, bubble creation and LWFA injection/acceleration while the tubes model, having large vacuum gaps between the tubes depletes the laser pulse at a slower rate;
• The two models match for the injected bunch size and location along y though (I will upload an image).
• The time resolution is automatically calculated using the CFL condition (for the effective model: DELTA_TIME_SI = 5.83e-17 and for the tubes model DELTA_TIME_SI = 3.91e-17);
• Dt in the plot refer to the total pulse length: 9 fs accounts for 9 cycles.

[cbontoiu]

[cbontoiu]

Looking for a solution I tried to increase the particle shape order definition from
- particles::shapes::TSC : 2nd order to - particles::shapes::PCS : 3rd order but I obtained this error:

[ 97%] Building CUDA object CMakeFiles/picongpu.dir/main.cpp.o
ptxas error   : Entry function '_ZN6alpaka6detail9gpuKernelIN5pmacc8lockstep4exec6detail14LockStepKernelIN8picongpu9particles8creation21CreateParticlesKernelENS3_9WorkerCfgILj512EEEEENS_9ApiCudaRtENS_22AccGpuUniformCudaHipRtISE_St17integral_constantImLm3EEjEESH_jJNS8_10ionization8ADK_ImplINSJ_12AlgorithmADKILb0EEENS7_9ParticlesINS2_4meta6StringIJLc101EEEEN5boost3mpl6v_itemINS7_11chargeRatioINS7_20ChargeRatioElectronsENS2_13pmacc_isAliasEEENST_INS7_9massRatioINS7_18MassRatioElectronsESW_EENST_INS7_7currentINS7_13currentSolver9EsirkepovINS8_6shapes3PCSENS12_8strategy16CachedSupercellsELj3EEESW_EENST_INS7_13interpolationINS7_28FieldToParticleInterpolationIS15_NS7_30AssignedTrilinearInterpolationEEESW_EENST_INS7_5shapeIS15_SW_EENST_INS7_14particlePusherINS8_6pusher5BorisESW_EENSS_7vector0IN4mpl_2naEEELi0EEELi0EEELi0EEELi0EEELi0EEELi0EEENST_INS7_10particleIdENST_INS7_9weightingENST_INS7_8momentumENST_INS7_8positionINS7_12position_picESW_EES1O_Li0EEELi0EEELi0EEELi0EEEEENSJ_7current4NoneENSN_INSP_IJLc67EEEENST_INS7_8ionizersINST_INSJ_11ThomasFermiIS25_EENST_INSJ_10ADKCircPolIS25_S27_EENST_INSJ_13BSIEffectiveZIS25_S27_EES1O_Li0EEELi0EEELi0EEESW_EENST_INS7_22effectiveNuclearChargeIKNS2_4math6VectorIfLi6ENS2L_16StandardAccessorENS2L_17StandardNavigatorENS7_10ionization22effectiveNuclearCharge28pmacc_static_const_storage3817ConstArrayStorageIfLi6EEEEESW_EENST_INS7_18ionizationEnergiesIKNS2M_IfLi6ES2N_S2O_NS2P_8energies2AU28pmacc_static_const_storage5817ConstArrayStorageIfLi6EEEEESW_EENST_INS7_13atomicNumbersINS2P_13atomicNumbers8Carbon_tESW_EENST_INS7_12densityRatioINS7_18DensityRatioCarbonESW_EENST_INSU_INS7_17ChargeRatioCarbonESW_EENST_INSY_INS7_15MassRatioCarbonESW_EES1S_Li0EEELi0EEELi0EEELi0EEELi0EEELi0EEELi0EEENST_INS7_14boundElectronsES24_Li0EEEEEEENS2_12ParticlesBoxINS2_5FrameINS2_15ParticlesBufferINS2_19ParticleDescriptionIS28_NS2L_2CT3IntILi8ELi8ELi8EEES3P_S3N_NS2_17HandleGuardRegionINS2_9particles8policies17ExchangeParticlesENS41_9DoNothingEEES1O_S1O_EES3Y_N8mallocMC9AllocatorISI_NS46_16CreationPolicies7ScatterINS7_16DeviceHeapConfigENS48_11ScatterConf27DefaultScatterHashingParamsEEENS46_20DistributionPolicies4NoopENS46_11OOMPolicies10ReturnNullENS46_19ReservePoolPolicies9AlpakaBufISI_EENS46_17AlignmentPolicies6ShrinkINS4L_12ShrinkConfig19DefaultShrinkConfigEEEEELj3EE29OperatorCreatePairStaticArrayILj512EEENS3V_IS28_S3Y_NST_INS2_9multiMaskENST_INS2_12localCellIdxES3P_Li0EEELi0EEES3N_S44_S1O_NST_INS2_12NextFramePtrINS1M_3argILi1EEEEENST_INS2_16PreviousFramePtrIS50_EES1O_Li0EEELi0EEEEEEENS46_19AllocatorHandleImplIS4Q_EELj3EEENS3S_INS3T_INS3U_INS3V_ISQ_S3Y_S24_S1U_S44_S1O_S1O_EES3Y_S4Q_Lj3EE29OperatorCreatePairStaticArrayILj512EEENS3V_ISQ_S3Y_NST_IS4U_NST_IS4V_S24_Li0EEELi0EEES1U_S44_S1O_S55_EEEES59_Lj3EEENS2_11AreaMappingILj3ENS2_18MappingDescriptionILj3ES3Y_EEEEEEEvNS_3VecIT2_T3_EET_DpT4_' uses too much shared data (0xce10 bytes, 0xc000 max)
ptxas error   : Entry function '_ZN6alpaka6detail9gpuKernelIN5pmacc8lockstep4exec6detail14LockStepKernelIN8picongpu26KernelMoveAndMarkParticlesINS2_20SuperCellDescriptionINS2_4math2CT3IntILi8ELi8ELi8EEENSB_6VectorIN4mpl_10integral_cIiLi2EEESH_SH_EENSE_INSG_IiLi3EEESJ_SJ_EEEEEENS3_9WorkerCfgILj512EEEEENS_9ApiCudaRtENS_22AccGpuUniformCudaHipRtISQ_St17integral_constantImLm3EEjEEST_jJNS2_12ParticlesBoxINS2_5FrameINS2_15ParticlesBufferINS2_19ParticleDescriptionINS2_4meta6StringIJLc67EEEESD_N5boost3mpl6v_itemINS7_14boundElectronsENS14_INS7_10particleIdENS14_INS7_9weightingENS14_INS7_8momentumENS14_INS7_8positionINS7_12position_picENS2_13pmacc_isAliasEEENS13_7vector0INSF_2naEEELi0EEELi0EEELi0EEELi0EEELi0EEENS14_INS7_8ionizersINS14_INS7_9particles10ionization11ThomasFermiINS7_9ParticlesINS10_IJLc101EEEENS14_INS7_11chargeRatioINS7_20ChargeRatioElectronsES1B_EENS14_INS7_9massRatioINS7_18MassRatioElectronsES1B_EENS14_INS7_7currentINS7_13currentSolver9EsirkepovINS1M_6shapes3PCSENS1Y_8strategy16CachedSupercellsELj3EEES1B_EENS14_INS7_13interpolationINS7_28FieldToParticleInterpolationIS21_NS7_30AssignedTrilinearInterpolationEEES1B_EENS14_INS7_5shapeIS21_S1B_EENS14_INS7_14particlePusherINS1M_6pusher5BorisES1B_EES1F_Li0EEELi0EEELi0EEELi0EEELi0EEELi0EEES1J_EEEENS14_INS1N_10ADKCircPolIS2N_NS1N_7current4NoneEEENS14_INS1N_13BSIEffectiveZIS2N_S2R_EES1F_Li0EEELi0EEELi0EEES1B_EENS14_INS7_22effectiveNuclearChargeIKNSA_6VectorIfLi6ENSA_16StandardAccessorENSA_17StandardNavigatorENS7_10ionization22effectiveNuclearCharge28pmacc_static_const_storage3817ConstArrayStorageIfLi6EEEEES1B_EENS14_INS7_18ionizationEnergiesIKNS30_IfLi6ES31_S32_NS33_8energies2AU28pmacc_static_const_storage5817ConstArrayStorageIfLi6EEEEES1B_EENS14_INS7_13atomicNumbersINS33_13atomicNumbers8Carbon_tES1B_EENS14_INS7_12densityRatioINS7_18DensityRatioCarbonES1B_EENS14_INS1R_INS7_17ChargeRatioCarbonES1B_EENS14_INS1U_INS7_15MassRatioCarbonES1B_EES2K_Li0EEELi0EEELi0EEELi0EEELi0EEELi0EEELi0EEENS2_17HandleGuardRegionINS2_9particles8policies17ExchangeParticlesENS44_9DoNothingEEES1F_S1F_EESD_N8mallocMC9AllocatorISU_NS49_16CreationPolicies7ScatterINS7_16DeviceHeapConfigENS4B_11ScatterConf27DefaultScatterHashingParamsEEENS49_20DistributionPolicies4NoopENS49_11OOMPolicies10ReturnNullENS49_19ReservePoolPolicies9AlpakaBufISU_EENS49_17AlignmentPolicies6ShrinkINS4O_12ShrinkConfig19DefaultShrinkConfigEEEEELj3EE29OperatorCreatePairStaticArrayILj512EEENSY_IS11_SD_NS14_INS2_9multiMaskENS14_INS2_12localCellIdxES1K_Li0EEELi0EEES41_S47_S1F_NS14_INS2_12NextFramePtrINSF_3argILi1EEEEENS14_INS2_16PreviousFramePtrIS53_EES1F_Li0EEELi0EEEEEEENS49_19AllocatorHandleImplIS4T_EELj3EEENS2_7DataBoxINS2_10PitchedBoxINS30_IfLi3ES31_S32_NSA_6detail17Vector_componentsIfLi3EEEEELj3EEEEES5L_jNS7_20PushParticlePerFrameIS2F_SD_S29_EENS2_11AreaMappingILj3ENS2_18MappingDescriptionILj3ESD_EEEEEEEvNS_3VecIT2_T3_EET_DpT4_' uses too much shared data (0xce00 bytes, 0xc000 max)
ptxas error   : Entry function '_ZN6alpaka6detail9gpuKernelIN5pmacc8lockstep4exec6detail14LockStepKernelIN8picongpu26KernelMoveAndMarkParticlesINS2_20SuperCellDescriptionINS2_4math2CT3IntILi8ELi8ELi8EEENSB_6VectorIN4mpl_10integral_cIiLi2EEESH_SH_EENSE_INSG_IiLi3EEESJ_SJ_EEEEEENS3_9WorkerCfgILj512EEEEENS_9ApiCudaRtENS_22AccGpuUniformCudaHipRtISQ_St17integral_constantImLm3EEjEEST_jJNS2_12ParticlesBoxINS2_5FrameINS2_15ParticlesBufferINS2_19ParticleDescriptionINS2_4meta6StringIJLc101EEEESD_N5boost3mpl6v_itemINS7_10particleIdENS14_INS7_9weightingENS14_INS7_8momentumENS14_INS7_8positionINS7_12position_picENS2_13pmacc_isAliasEEENS13_7vector0INSF_2naEEELi0EEELi0EEELi0EEELi0EEENS14_INS7_11chargeRatioINS7_20ChargeRatioElectronsES1A_EENS14_INS7_9massRatioINS7_18MassRatioElectronsES1A_EENS14_INS7_7currentINS7_13currentSolver9EsirkepovINS7_9particles6shapes3PCSENS1Q_8strategy16CachedSupercellsELj3EEES1A_EENS14_INS7_13interpolationINS7_28FieldToParticleInterpolationIS1U_NS7_30AssignedTrilinearInterpolationEEES1A_EENS14_INS7_5shapeIS1U_S1A_EENS14_INS7_14particlePusherINS1S_6pusher5BorisES1A_EES1E_Li0EEELi0EEELi0EEELi0EEELi0EEELi0EEENS2_17HandleGuardRegionINS2_9particles8policies17ExchangeParticlesENS2I_9DoNothingEEES1E_S1E_EESD_N8mallocMC9AllocatorISU_NS2N_16CreationPolicies7ScatterINS7_16DeviceHeapConfigENS2P_11ScatterConf27DefaultScatterHashingParamsEEENS2N_20DistributionPolicies4NoopENS2N_11OOMPolicies10ReturnNullENS2N_19ReservePoolPolicies9AlpakaBufISU_EENS2N_17AlignmentPolicies6ShrinkINS32_12ShrinkConfig19DefaultShrinkConfigEEEEELj3EE29OperatorCreatePairStaticArrayILj512EEENSY_IS11_SD_NS14_INS2_9multiMaskENS14_INS2_12localCellIdxES1I_Li0EEELi0EEES2F_S2L_S1E_NS14_INS2_12NextFramePtrINSF_3argILi1EEEEENS14_INS2_16PreviousFramePtrIS3H_EES1E_Li0EEELi0EEEEEEENS2N_19AllocatorHandleImplIS37_EELj3EEENS2_7DataBoxINS2_10PitchedBoxINSA_6VectorIfLi3ENSA_16StandardAccessorENSA_17StandardNavigatorENSA_6detail17Vector_componentsIfLi3EEEEELj3EEEEES42_jNS7_20PushParticlePerFrameIS28_SD_S22_EENS2_11AreaMappingILj3ENS2_18MappingDescriptionILj3ESD_EEEEEEEvNS_3VecIT2_T3_EET_DpT4_' uses too much shared data (0xce00 bytes, 0xc000 max)
make[2]: *** [CMakeFiles/picongpu.dir/build.make:76: CMakeFiles/picongpu.dir/main.cpp.o] Error 255
make[1]: *** [CMakeFiles/Makefile2:288: CMakeFiles/picongpu.dir/all] Error 2
make: *** [Makefile:136: all] Error 2
ERROR: Could not successfully run make install in build directory:
       .build

[cbontoiu]

It seems that running with a lower BASE_DENSITY i.e. 1e27 instead of 1e28 as in the model attached, eliminates the jump in charge density. It might be some numerical procession issue.

[PrometheusPi]

@cbontoiu Sorry for my late reply. (Are work trough GitHub messages as a queue.)

Thank you for your clarifications.

I am not sure why, PCS fails. Could you provide details how it fails (out of memory, communication too slow). A higher shape requires more communications and larger boarders.

It is interesting, that with a lower density, the simulation stabilizes. Could it be, that the density of the tubes is so high, that they heat up tremendously with the first interaction with the laser and trigger the cold stream instability causing too much field energy and thus rapid ionization? What is the peak density of the tubes (ions and electrons) at setup time and what temperature are you adding to them?

[psychocoderHPC]

uses too much shared data (0xce00 bytes, 0xc000 max)

@cbontoiu The error message must be scrolled to the right. The compiler can not build because ethe shared memory usage was to high. I assume you compiled with double precision. To workaround this you can change in your setup memory.param the supercell size to 8x4x4 cells using SuperCellSize = typename mCT::shrinkTo<mCT::Int<8, 4, 4>, simDim>::type;

This will allow you to use double precision and the shape PCS.

note: using SuperCellSize = typename mCT::shrinkTo<mCT::Int<4, 8, 4>, simDim>::type; can be used too, based on the setup it could be beneficial to use 8 cells in Y-direction to reduce on device particle memcopies.

[BrianMarre]

@cbontoiu did the suggestions of @psychocoderHPC fix your issue?, if yes please remember to close your issue, thanks!

[steindev]

ping @cbontoiu

[BrianMarre]

@cbontoiu any news?

[cbontoiu]

Ah! How could I have missed your notifications!? I will try this evening and let you know.

[cbontoiu]

Unfortunately, for technical reasons I'm not yet able to test the solution suggested above. My model is attached in the beginning of the thread and I worked with 32bit procession. I'll close this ticket. Thanks for your help.