Closed franzpoeschel closed 3 years ago
ax3l
commented
3 years ago More blocks is a good idea! You can also make the PIConGPU supercell (memory.param) smaller, e.g. 4x4x4, to make even smaller simulations with multiple MPI ranks (just makes the sim a little slower, but we are out for data size here).
Can you please also document the ADIOS2 version that was used in the PR description?
franzpoeschel
commented
3 years ago More blocks is a good idea! You can also make the PIConGPU supercell (
memory.param) smaller, e.g.4x4x4, to make even smaller simulations with multiple MPI ranks (just makes the sim a little slower, but we are out for data size here).
Alright, will try that. (I could theoretically use openpmd-pipe to cut the chunks smaller, but I think it's better to use openPMD 0.12.0, so I'll try running it more in parallel first.)
Can you please also document the ADIOS2 version that was used in the PR description?
Done
franzpoeschel
commented
3 years ago Also, ADIOS2 uses aggregation per node, so every block will be on sourceID == 0. Does that matter for your tests?
ax3l
commented
3 years ago Since we want to use this for read tests, I would say disable node-aggregation so we have more meta-data that we can test
franzpoeschel
commented
3 years ago Done a further run with parameters:
mpirun --oversubscribe -n 16 picongpu -s 550 -d 2 4 2 -g 56 56 28 --openPMD.period 50 --openPMD.file stream --openPMD.infix NULL --openPMD.ext bp --openPMD.json '{ "adios2": { "engine": { "usesteps": true, "type": "bp4", "parameters": { "AggregatorRatio": "1", "InitialBufferSize": "2Gb", "Profile": "On" } }, "dataset": { "operators": [ ] } } } ' --versionOnce --checkpoint.backend openPMD --checkpoint.period 500 --checkpoint.restart.backend openPMD --openPMD.dataPreparationStrategy hdf5 --checkpoint.openPMD.dataPreparationStrategy hdf5 --checkpoint.restarttldr: disable node aggregation, use 16 parallel writers in a 2x4x2 domain decomposition
bpls -D gives:
float /data/550/fields/B/x {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
float /data/550/fields/B/y {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
float /data/550/fields/B/z {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
float /data/550/fields/E/x {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
float /data/550/fields/E/y {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
float /data/550/fields/E/z {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
float /data/550/fields/e_chargeDensity {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
float /data/550/fields/e_energyDensity {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
float /data/550/fields/e_particleMomentumComponent {64, 96, 64}
step 0:
block 0: [ 0:31, 0:31, 0:31]
block 1: [32:63, 0:31, 0:31]
block 2: [ 0:31, 32:47, 0:31]
block 3: [32:63, 32:47, 0:31]
block 4: [ 0:31, 48:63, 0:31]
block 5: [32:63, 48:63, 0:31]
block 6: [ 0:31, 64:95, 0:31]
block 7: [32:63, 64:95, 0:31]
block 8: [ 0:31, 0:31, 32:63]
block 9: [32:63, 0:31, 32:63]
block 10: [ 0:31, 32:47, 32:63]
block 11: [32:63, 32:47, 32:63]
block 12: [ 0:31, 48:63, 32:63]
block 13: [32:63, 48:63, 32:63]
block 14: [ 0:31, 64:95, 32:63]
block 15: [32:63, 64:95, 32:63]
uint64_t /data/550/fields/picongpu_idProvider/nextId {2, 4, 2}
step 0:
block 0: [0:0, 0:0, 0:0]
block 1: [1:1, 0:0, 0:0]
block 2: [0:0, 1:1, 0:0]
block 3: [1:1, 1:1, 0:0]
block 4: [0:0, 2:2, 0:0]
block 5: [1:1, 2:2, 0:0]
block 6: [0:0, 3:3, 0:0]
block 7: [1:1, 3:3, 0:0]
block 8: [0:0, 0:0, 1:1]
block 9: [1:1, 0:0, 1:1]
block 10: [0:0, 1:1, 1:1]
block 11: [1:1, 1:1, 1:1]
block 12: [0:0, 2:2, 1:1]
block 13: [1:1, 2:2, 1:1]
block 14: [0:0, 3:3, 1:1]
block 15: [1:1, 3:3, 1:1]
uint64_t /data/550/fields/picongpu_idProvider/startId {2, 4, 2}
step 0:
block 0: [0:0, 0:0, 0:0]
block 1: [1:1, 0:0, 0:0]
block 2: [0:0, 1:1, 0:0]
block 3: [1:1, 1:1, 0:0]
block 4: [0:0, 2:2, 0:0]
block 5: [1:1, 2:2, 0:0]
block 6: [0:0, 3:3, 0:0]
block 7: [1:1, 3:3, 0:0]
block 8: [0:0, 0:0, 1:1]
block 9: [1:1, 0:0, 1:1]
block 10: [0:0, 1:1, 1:1]
block 11: [1:1, 1:1, 1:1]
block 12: [0:0, 2:2, 1:1]
block 13: [1:1, 2:2, 1:1]
block 14: [0:0, 3:3, 1:1]
block 15: [1:1, 3:3, 1:1]
float /data/550/particles/e/momentum/x {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
float /data/550/particles/e/momentum/y {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
float /data/550/particles/e/momentum/z {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
uint64_t /data/550/particles/e/particlePatches/extent/x {16}
step 0:
block 0: [ 0: 0]
block 1: [ 1: 1]
block 2: [ 2: 2]
block 3: [ 3: 3]
block 4: [ 4: 4]
block 5: [ 5: 5]
block 6: [ 6: 6]
block 7: [ 7: 7]
block 8: [ 8: 8]
block 9: [ 9: 9]
block 10: [10:10]
block 11: [11:11]
block 12: [12:12]
block 13: [13:13]
block 14: [14:14]
block 15: [15:15]
uint64_t /data/550/particles/e/particlePatches/extent/y {16}
step 0:
block 0: [ 0: 0]
block 1: [ 1: 1]
block 2: [ 2: 2]
block 3: [ 3: 3]
block 4: [ 4: 4]
block 5: [ 5: 5]
block 6: [ 6: 6]
block 7: [ 7: 7]
block 8: [ 8: 8]
block 9: [ 9: 9]
block 10: [10:10]
block 11: [11:11]
block 12: [12:12]
block 13: [13:13]
block 14: [14:14]
block 15: [15:15]
uint64_t /data/550/particles/e/particlePatches/extent/z {16}
step 0:
block 0: [ 0: 0]
block 1: [ 1: 1]
block 2: [ 2: 2]
block 3: [ 3: 3]
block 4: [ 4: 4]
block 5: [ 5: 5]
block 6: [ 6: 6]
block 7: [ 7: 7]
block 8: [ 8: 8]
block 9: [ 9: 9]
block 10: [10:10]
block 11: [11:11]
block 12: [12:12]
block 13: [13:13]
block 14: [14:14]
block 15: [15:15]
uint64_t /data/550/particles/e/particlePatches/numParticles {16}
step 0:
block 0: [ 0: 0]
block 1: [ 1: 1]
block 2: [ 2: 2]
block 3: [ 3: 3]
block 4: [ 4: 4]
block 5: [ 5: 5]
block 6: [ 6: 6]
block 7: [ 7: 7]
block 8: [ 8: 8]
block 9: [ 9: 9]
block 10: [10:10]
block 11: [11:11]
block 12: [12:12]
block 13: [13:13]
block 14: [14:14]
block 15: [15:15]
uint64_t /data/550/particles/e/particlePatches/numParticlesOffset {16}
step 0:
block 0: [ 0: 0]
block 1: [ 1: 1]
block 2: [ 2: 2]
block 3: [ 3: 3]
block 4: [ 4: 4]
block 5: [ 5: 5]
block 6: [ 6: 6]
block 7: [ 7: 7]
block 8: [ 8: 8]
block 9: [ 9: 9]
block 10: [10:10]
block 11: [11:11]
block 12: [12:12]
block 13: [13:13]
block 14: [14:14]
block 15: [15:15]
uint64_t /data/550/particles/e/particlePatches/offset/x {16}
step 0:
block 0: [ 0: 0]
block 1: [ 1: 1]
block 2: [ 2: 2]
block 3: [ 3: 3]
block 4: [ 4: 4]
block 5: [ 5: 5]
block 6: [ 6: 6]
block 7: [ 7: 7]
block 8: [ 8: 8]
block 9: [ 9: 9]
block 10: [10:10]
block 11: [11:11]
block 12: [12:12]
block 13: [13:13]
block 14: [14:14]
block 15: [15:15]
uint64_t /data/550/particles/e/particlePatches/offset/y {16}
step 0:
block 0: [ 0: 0]
block 1: [ 1: 1]
block 2: [ 2: 2]
block 3: [ 3: 3]
block 4: [ 4: 4]
block 5: [ 5: 5]
block 6: [ 6: 6]
block 7: [ 7: 7]
block 8: [ 8: 8]
block 9: [ 9: 9]
block 10: [10:10]
block 11: [11:11]
block 12: [12:12]
block 13: [13:13]
block 14: [14:14]
block 15: [15:15]
uint64_t /data/550/particles/e/particlePatches/offset/z {16}
step 0:
block 0: [ 0: 0]
block 1: [ 1: 1]
block 2: [ 2: 2]
block 3: [ 3: 3]
block 4: [ 4: 4]
block 5: [ 5: 5]
block 6: [ 6: 6]
block 7: [ 7: 7]
block 8: [ 8: 8]
block 9: [ 9: 9]
block 10: [10:10]
block 11: [11:11]
block 12: [12:12]
block 13: [13:13]
block 14: [14:14]
block 15: [15:15]
float /data/550/particles/e/position/x {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
float /data/550/particles/e/position/y {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
float /data/550/particles/e/position/z {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
int32_t /data/550/particles/e/positionOffset/x {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
int32_t /data/550/particles/e/positionOffset/y {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
int32_t /data/550/particles/e/positionOffset/z {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]
float /data/550/particles/e/weighting {265489}
step 0:
block 0: [ 0: 3]
block 1: [ 4: 12]
block 2: [ 13: 3288]
block 3: [ 3289: 6170]
block 4: [ 6171: 24251]
block 5: [ 24252: 42329]
block 6: [ 42330: 87523]
block 7: [ 87524:132925]
block 8: [132926:132928]
block 9: [132929:135783]
block 10: [135784:138230]
block 11: [138231:156254]
block 12: [156255:174244]
block 13: [174245:219710]
block 14: [219711:265488]Commit will follow in a minute.
ax3l
commented
3 years ago This is cool, awesome! The simulation box size is a bit narrow and thus the LWFA sim looks a bit adventurous (not really an LWFA anymore), but the data is good enough and super nice in size for testing! (We can just rename it here to avoid confusion to example-3d-bp4.tar.gz)
Optional for merge here, but general comment for PIConGPU:
I think the mass and charge particle records are missing in the output, can you please re-add them in PIConGPU?
https://github.com/openPMD/openPMD-standard/blob/upcoming-2.0.0/EXT_ED-PIC.md
I thought https://github.com/ComputationalRadiationPhysics/picongpu/pull/3389 fixed this, but they seem to be missing here.
ax3l
commented
3 years ago I thought a bit more about a name that tells what's inside (format and dimension wise), e.g. example-3d-bp4.tar.gz :)
ax3l
commented
3 years ago @franzpoeschel I found a bug in this data set that we need to fix and replace. The constant record components have an undefined shape:
$ bpls -al samples/git-sample/3d-bp4/example-3d-bp4.bp/
uint64_t /data/550/particles/e/charge/shape attr = 0
uint64_t /data/550/particles/e/mass/shape attr = 0They should read:
uint64_t /data/550/particles/e/charge/shape attr = 265489
uint64_t /data/550/particles/e/mass/shape attr = 265489just like the non-constant particle record lengths.
ax3l
commented
3 years ago The bug in PIConGPU is here: https://github.com/ComputationalRadiationPhysics/picongpu/pull/3389/files#r663311000
franzpoeschel
commented
3 years ago Good catch! I think we should get back to this once the bug is fixed in PIConGPU?
ax3l
commented
3 years ago Bug fixed in PIConGPU via https://github.com/ComputationalRadiationPhysics/picongpu/pull/3678 :tada:
Yes, ready now to replace :)
I'm currently working on new schemas for ADIOS2, so we should add tests to verify that we keep backwards compatibility.
Created with PIConGPU / LaserWakefield simulation, openPMD API 0.12.0 and ADIOS 2.7.0 with parameters:
This is purposefully executed with two parallel instances in order to create datasets that are composed from multiple blocks. Output of
bpls -D:Todo: