Update standard plots, add special mode

[Moyoxkit]

Updates to the FLAMINGO configs, also add a special mode to add Eddington bias to all plots that use stellar mass. This should create all the plots needed for Paper1 as well (Some 50kpc ones were missing).

I still need to add the special mode to the config.

[bwvdnbro]

I just tried to run your code using the special mode on the 1 Gpc intermediate-res fiducial model and I ran into some issues:

Traceback (most recent call last):
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/./pipeline_env/bin/swift-pipeline", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline/swift-pipeline", line 316, in <module>
    special_mode.adapt_catalogue(catalogue)
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline/swiftpipeline/config.py", line 102, in adapt_catalogue
    exec(handle.read())
  File "<string>", line 10, in <module>
NameError: name 'np' is not defined

Traceback (most recent call last):
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/./pipeline_env/bin/swift-pipeline", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline/swift-pipeline", line 316, in <module>
    special_mode.adapt_catalogue(catalogue)
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline/swiftpipeline/config.py", line 102, in adapt_catalogue
    exec(handle.read())
  File "<string>", line 15, in <module>
NameError: name 'unyt' is not defined

These can be solved by importing numpy and unyt in apply_eddington_mass_bias.py. This script is called with a somewhat different environment than the registration script, so it does not necessarily have access to all the same modules.

Unable to create plot stellar_mass_star_fe_over_h_lom_50 due to exception: 'DerivedQuantities' object has no attribute 'star_fe_abundance_avglin_50_kpc'.
Unable to create plot stellar_mass_function_extended_50 due to exception: 'DerivedQuantities' object has no attribute 'stellar_mass_eddington_50_kpc'.
Unable to create plot stellar_mass_function_calibration_extended_50 due to exception: 'DerivedQuantities' object has no attribute 'stellar_mass_eddington_50_kpc'.

Some plots still use the derived quantity, so if you want to keep those plots, you should still keep the definition in the registration function.

And then there are a lot of variables that are used in the scripts that seem to be missing from the snapshots altogether:

Traceback (most recent call last):
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/max_SNII_kick_velocities.py", line 86, in <module>
    make_single_image(
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/max_SNII_kick_velocities.py", line 54, in make_single_image
    v_kick_max = get_data(filename)
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/max_SNII_kick_velocities.py", line 23, in get_data
    stars_SNII_v_kick_max = (data.stars.maximal_sniikinetic_feedbackvkick).to("km/s")
AttributeError: 'StarsDataset' object has no attribute 'maximal_sniikinetic_feedbackvkick'

Traceback (most recent call last):
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/birth_density_f_E.py", line 97, in <module>
    H, density_edges, f_E_edges = bin_individual_data(snapshot)
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/birth_density_f_E.py", line 53, in bin_individual_data
    f_E_fractions = data.stars.feedback_energy_fractions.value
AttributeError: 'StarsDataset' object has no attribute 'feedback_energy_fractions'

Traceback (most recent call last):
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/feedback_SNII_events.py", line 86, in <module>
    make_single_image(
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/feedback_SNII_events.py", line 45, in make_single_image
    N_SNII = get_data(filename)
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/feedback_SNII_events.py", line 24, in get_data
    gas_N_SNII = data.gas.heated_by_sniifeedback
AttributeError: 'GasDataset' object has no attribute 'heated_by_sniifeedback'

Traceback (most recent call last):
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/last_SNII_kick_velocity_distribution.py", line 47, in <module>
    stars_SNII_v_kick_last = (snapshot.stars.last_sniikinetic_feedbackvkick).to(
AttributeError: 'StarsDataset' object has no attribute 'last_sniikinetic_feedbackvkick'

Traceback (most recent call last):
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/feedback_AGN_events.py", line 86, in <module>
    make_single_image(
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/feedback_AGN_events.py", line 45, in make_single_image
    N_AGN = get_data(filename)
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/feedback_AGN_events.py", line 24, in get_data
    gas_N_AGN = data.gas.heated_by_agnfeedback
AttributeError: 'GasDataset' object has no attribute 'heated_by_agnfeedback'

[Moyoxkit]

Are the special fields applied before or after the registration? It would be nice to always have some Eddington plots However we don't want to add it twice (first with the special mode, then again in the registration)

Also, about the missing tracers from the snapshot. They are not stored by default, but we might want to look at them every now and again, which is why they are still in the pipeline even-though they lead to errors. I don't know the best way to proceed with this.

[bwvdnbro]

About the missing tracers: can you catch the error and just print some text (like "No tracer data") if the pipeline is run on a simulation that does not have the tracer data? Then the plot will still be made and it will be clear what happened.

The special fields are applied after the catalogue has been processed by velociraptor-python, which means by that time all the fields in the catalogue (including the derived quantities produced by the registration functions) are available.

It won't be easy to disable some of the plots in a special mode, so I think we will just need to keep the Eddington plots as they are, even though they do not make sense if you run in Eddington bias mode.

If the registration function is still run, then you could simply copy the derived field into the original field in the special mode script.

[Moyoxkit]

If I understood you correctly, me re-adding the Eddington bias now will just make it so there are always some Eddinton bias plots, and they will always be correct, it will just be double for the special mode. I've also fixed all the tracers and they no longer lead to errors.

[bwvdnbro]

Yes, that is correct, the old Eddington bias plots will not be affected by the special mode, since they use a different variable. So they will still be correct, but indeed be duplicates if you run in special mode.

[Moyoxkit]

Great! I think this should be ready to merge then, if you have no other comments.

[bwvdnbro]

I am still getting this error, but I guess that is unrelated to this pull request:

Traceback (most recent call last):
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/birth_density_f_E.py", line 97, in <module>
    H, density_edges, f_E_edges = bin_individual_data(snapshot)
  File "/snap8/scratch/dp004/dc-vand2/FLAMINGO/pipeline-configs/flamingo/scripts/birth_density_f_E.py", line 53, in bin_individual_data
    f_E_fractions = data.stars.feedback_energy_fractions.value
AttributeError: 'StarsDataset' object has no attribute 'feedback_energy_fractions'

Still, I will think I will fix this as well, if that is okay with you.

[Moyoxkit]

You probably only get that with the reduced snapshots, as I didn't get that error on my tests. Feel free to fix it though!

[bwvdnbro]

No, this is running on the full L1000N1800 HYDRO_FIDUCIAL z=0 snapshot.

[Moyoxkit]

Sorry, with reduced I'm implying the ones that use select_output, and therefore also don't have the standard --subgrid=FLAMINGO tracers

[bwvdnbro]

I managed to run the full pipeline on the L1000N1800 HYDRO_FIDUCIAL z=0 snapshot in:

• normal mode: https://home.strw.leidenuniv.nl/~vandenbroucke/flamingo_L1000N1800/normal_mode
• special mode: https://home.strw.leidenuniv.nl/~vandenbroucke/flamingo_L1000N1800/special_mode

Not all plots look as expected, but at least all of them work. While flipping between the two versions, I did notice that the "Stellar Mass Function (50 kpc aperture) with Eddington bias with extended ranges" plot looks different between the normal and special mode at the high mass end. At first I thought this might be an issue, but I guess each run of the pipeline uses a different random number seed?

I am happy for this to be merged if you have no further thoughts. I will try to merge the special mode feature in the other repository next week, so that you don't have to run a separate pipeline version.

[Moyoxkit]

Yeah each run uses a different seed, so slight differences are expected, I think this is ready to be merged!