Error message during SC tracking and pytao crashing the python kernel on Segmentation Fault

[ColwynGulliford]

I have an example where I'm tracking a beam handed off from an injector simulation using GPT, and tracked with space charge through a low energy merger and short linac. The linac features RF cavity field maps.

Running the example using vanilla Tao (installed via conda, bmad=20230110, most recent) During the tracking I see that a few particles are lost, and then I get thousands of error messages:

[ERROR | 2023-JAN-20 10:57:38] track1:
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
[ERROR | 2023-JAN-20 10:57:38] track1:
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
[ERROR | 2023-JAN-20 10:57:38] track1:
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
[ERROR | 2023-JAN-20 10:57:38] track1:
[ERROR | 2023-JAN-20 10:57:38] track1:
[ERROR | 2023-JAN-20 10:57:38] track1:
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
[ERROR | 2023-JAN-20 10:57:38] track1:
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
[ERROR | 2023-JAN-20 10:57:38] track1:
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
[ERROR | 2023-JAN-20 10:57:38] track1:
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]
[ERROR | 2023-JAN-20 10:57:38] track1:
    STARTING ORBIT NOT PROPERLY INITIALIZED! [NEEDS AN INIT_COORD CALL?]

If I run the same example, with the same commands, using pytao - it crashes the python kernel. For my workflow, I rely on investigating the tracking dynamics using pytao as I'm patching together a GPT and Tao sim. I use this to make sure things look ok before running long/costly optimizations. And then use it afterwards to check the optimization results are sane.

Below is a MWE for the issues described. Note tao should be run with the included commands.tao file:

! tao -init tao.init -noplot 
set global lattice_calc_on = False
set beam_init position_file = initial_tao_particles.h5
set ele beginning e_tot = 2259164.8483772734
set beam_init n_particle = 1000
set beam_init bunch_charge = 1.9999999999999997e-09
set ele LA.CRMOD.CAV01 voltage = 2977835.689339646
set ele LA.CRMOD.CAV02 voltage = 2977835.689339646
set ele LA.CRMOD.CAV03 voltage = 595567.1378679292
set ele LA.CRMODREV.CAV01 voltage = 595567.1378679292
set ele LA.CRMODREV.CAV02 voltage = 2977835.689339646
set ele LA.CRMODREV.CAV03 voltage = 2977835.689339646
set ele MG.SOLX1 bs_field = 0.050823153906318
set ele MG.SOLX2 bs_field = 0.0539586380119295
set ele MG.SOLX3 bs_field = 0.0725845344351436
set ele mg.spacer1 L = 0.5
set ele mg.spacer2 L = 0.5
set space_charge_com  space_charge_mesh_size(1) = 32
set space_charge_com  space_charge_mesh_size(2) = 32
set space_charge_com  space_charge_mesh_size(3) = 32
set ele * space_charge_method = fft_3d
set bmad_com  csr_and_space_charge_on = True
set space_charge_com  n_bin = 40
set space_charge_com  ds_track_step = 0.002
set ele * csr_method = off
write bmad lat.bmad

set global lattice_calc_on = True
set global track_type =  beam
set global track_type =  single

write beam -at LA.CRMODREV.MAR.END LA.CRMODREV.MAR.END.h5
write beam -at end end.h5
quit

pcla_mwe.zip

[ColwynGulliford]

In the MWE above, there is a Jupyter notebook that can be run to see python kernel crashing. The python code is:

from pytao import Tao

tao = Tao('-init tao.init -noplot ')

with open('commands.tao', 'r') as fid:
    cmds = [cmd.replace('\n', '') for cmd in fid]

for cmd in cmds: 
    print(cmd) 
    tao.cmd(cmd)

[ColwynGulliford]

When running this example in using the python script I get:

(cooler) Colwyns-MBP:pcla_mwe colwyngulliford$ python pytao_example.py
! tao -init tao.init -noplot 
set global lattice_calc_on = False
set beam_init position_file = initial_tao_particles.h5
set ele beginning e_tot = 2259164.8483772734
set beam_init n_particle = 1000
set beam_init bunch_charge = 1.9999999999999997e-09
set ele LA.CRMOD.CAV01 voltage = 2977835.689339646
set ele LA.CRMOD.CAV02 voltage = 2977835.689339646
set ele LA.CRMOD.CAV03 voltage = 595567.1378679292
set ele LA.CRMODREV.CAV01 voltage = 595567.1378679292
set ele LA.CRMODREV.CAV02 voltage = 2977835.689339646
set ele LA.CRMODREV.CAV03 voltage = 2977835.689339646
set ele MG.SOLX1 bs_field = 0.050823153906318
set ele MG.SOLX2 bs_field = 0.0539586380119295
set ele MG.SOLX3 bs_field = 0.0725845344351436
set ele mg.spacer1 L = 0.5
set ele mg.spacer2 L = 0.5
set space_charge_com  space_charge_mesh_size(1) = 32
set space_charge_com  space_charge_mesh_size(2) = 32
set space_charge_com  space_charge_mesh_size(3) = 32
set ele * space_charge_method = fft_3d
set bmad_com  csr_and_space_charge_on = True
set space_charge_com  n_bin = 40
set space_charge_com  ds_track_step = 0.002
set ele * csr_method = off
write bmad lat.bmad

set global lattice_calc_on = True
set global track_type =  beam
Segmentation fault: 11
(cooler) Colwyns-MBP:pcla_mwe colwyngulliford$ 

So it appears tao/pytao crash at or before set global track_type = single

[DavidSagan]

Fixed the NOT PROPERLY INITIALIZED problem.