Open JYangQi00 opened 1 year ago
Was this already discussed in Team A, if not can we discsuss this PR?
Was this already discussed in Team A, if not can we discsuss this PR?
This is not discussed. I am still implementing things into it.
I have cleaned the json after running all checking. I will let @JYangQi00 have some time to implement the check of the highest plugin that is promised to be unchanged. I think if any change is found, then there must be some issue and a big error message should be raised.
I have added the functionality to test for the changes to the highest level plugin. "Highest level" in this context actually has a pretty clear definition. Here, the "level" or "layer" that a plugin lies on is the amount of layers which need to be available for the plugin of interest to be made. Here is an example output
Already on 'track_lineage_changes'
Your branch is up to date with 'origin/track_lineage_changes'.
You specified _auto_append_rucio_local=True and you are not on dali compute nodes, so we will add the following rucio local path: /project/lgrandi/rucio/
/home/jyangqi/straxen/straxen/url_config.py:743: UserWarning: From straxen version 2.1.0 onward, URLConfig parameterswill be sorted alphabetically before being passed to the plugins, this will change the lineage hash for non-sorted URLs. To load data processed with non-sorted URLs, you will need to use an older version.
warnings.warn(
You specified _auto_append_rucio_local=True and you are not on dali compute nodes, so we will add the following rucio local path: /project/lgrandi/rucio/
Switched to branch 'test_plugin_changes'
Your branch is up to date with 'origin/test_plugin_changes'.
You specified _auto_append_rucio_local=True and you are not on dali compute nodes, so we will add the following rucio local path: /project/lgrandi/rucio/
/home/jyangqi/straxen/straxen/url_config.py:743: UserWarning: From straxen version 2.1.0 onward, URLConfig parameterswill be sorted alphabetically before being passed to the plugins, this will change the lineage hash for non-sorted URLs. To load data processed with non-sorted URLs, you will need to use an older version.
warnings.warn(
Deleted plugins:
- s2_fwhm
You specified _auto_append_rucio_local=True and you are not on dali compute nodes, so we will add the following rucio local path: /project/lgrandi/rucio/
/home/jyangqi/straxen/straxen/url_config.py:743: UserWarning: From straxen version 2.1.0 onward, URLConfig parameterswill be sorted alphabetically before being passed to the plugins, this will change the lineage hash for non-sorted URLs. To load data processed with non-sorted URLs, you will need to use an older version.
warnings.warn(
Loading s2_fwhm_2: |██████████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.63 s). 0.9 kB/s
New plugin 's2_fwhm_2' has the following bad field fractions:
- s2_fwhm: 0.1
- random_stuff: 1.0
You specified _auto_append_rucio_local=True and you are not on dali compute nodes, so we will add the following rucio local path: /project/lgrandi/rucio/
/home/jyangqi/straxen/straxen/url_config.py:743: UserWarning: From straxen version 2.1.0 onward, URLConfig parameterswill be sorted alphabetically before being passed to the plugins, this will change the lineage hash for non-sorted URLs. To load data processed with non-sorted URLs, you will need to use an older version.
warnings.warn(
event_nans: ['event_top_bottom_params' 'event_info_double' 'event_ms_naive']
Loading event_nans: |█████████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.48 s). 1.3 kB/s
Loading event_basics: |██████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.29 s). 27.2 kB/s
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
Loading event_positions: |███████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.05 s). 59.3 kB/s
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
Loading event_info_double: |█████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.23 s). 54.8 kB/s
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
Loading event_ms_naive: |████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.05 s). 28.0 kB/s
Loading event_top_bottom_params: |███████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.05 s). 82.0 kB/s
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
Switched to branch 'track_lineage_changes'
Your branch is up to date with 'origin/track_lineage_changes'.
You specified _auto_append_rucio_local=True and you are not on dali compute nodes, so we will add the following rucio local path: /project/lgrandi/rucio/
/home/jyangqi/straxen/straxen/url_config.py:743: UserWarning: From straxen version 2.1.0 onward, URLConfig parameterswill be sorted alphabetically before being passed to the plugins, this will change the lineage hash for non-sorted URLs. To load data processed with non-sorted URLs, you will need to use an older version.
warnings.warn(
Loading event_nans: |█████████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.36 s). 1.8 kB/s
Loading event_basics: |█████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.08 s). 101.0 kB/s
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
Loading event_positions: |███████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.05 s). 58.7 kB/s
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
Loading event_info_double: |█████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.31 s). 40.8 kB/s
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
Loading event_ms_naive: |████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.06 s). 26.1 kB/s
Loading event_top_bottom_params: |███████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.05 s). 82.1 kB/s
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/fromnumeric.py:3474: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/cvmfs/xenon.opensciencegrid.org/releases/nT/development/anaconda/envs/XENONnT_development/lib/python3.9/site-packages/numpy/core/_methods.py:189: RuntimeWarning: invalid value encountered in true_divide
ret = ret.dtype.type(ret / rcount)
You specified _auto_append_rucio_local=True and you are not on dali compute nodes, so we will add the following rucio local path: /project/lgrandi/rucio/
Loading event_nans: |████████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.01 s). 84.1 kB/s
Loading event_basics: |███████████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.01 s). 1.1 MB/s
Loading event_positions: |██████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.01 s). 454.3 kB/s
Loading event_info_double: |██████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.01 s). 1.2 MB/s
Loading event_ms_naive: |███████████████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.01 s). 221.3 kB/s
Loading event_top_bottom_params: |██████████████████████████████████████████████████████| 100.00 % [00:00<00:00], #1 (0.01 s). 506.0 kB/s
#################### Lowest Levels Plugins ####################
Change report for 'event_nans':
- event_nans bad fraction changed from: 0.45 -> 0.1
- event_negs mean value changed from: -0.45 -> -0.1
All other columns remained the same
#################### Directly Affected Plugins ####################
Change report for 'event_basics':
- alt_s1_area mean value changed from: nan -> nan
- alt_s1_area_fraction_top mean value changed from: nan -> nan
- alt_s1_interaction_drift_time mean value changed from: nan -> nan
- alt_s1_max_pmt_area mean value changed from: nan -> nan
- alt_s1_range_50p_area mean value changed from: nan -> nan
- alt_s1_range_90p_area mean value changed from: nan -> nan
- alt_s1_rise_time mean value changed from: nan -> nan
- alt_s2_interaction_drift_time mean value changed from: nan -> nan
All other columns remained the same
Change report for 'event_positions':
- alt_s1_r_fdc mean value changed from: nan -> nan
- alt_s1_r_field_distortion_correction mean value changed from: nan -> nan
- alt_s1_x_fdc mean value changed from: nan -> nan
- alt_s1_y_fdc mean value changed from: nan -> nan
- alt_s1_z mean value changed from: nan -> nan
- alt_s1_z_dv_corr mean value changed from: nan -> nan
- alt_s1_z_naive mean value changed from: nan -> nan
- alt_s2_r_fdc mean value changed from: nan -> nan
- alt_s2_r_field_distortion_correction mean value changed from: nan -> nan
- alt_s2_x_fdc mean value changed from: nan -> nan
- alt_s2_y_fdc mean value changed from: nan -> nan
- alt_s2_z mean value changed from: nan -> nan
- alt_s2_z_dv_corr mean value changed from: nan -> nan
- alt_s2_z_naive mean value changed from: nan -> nan
All other columns remained the same
#################### Highest Level Affected Plugins ####################
Change report for 'event_info_double':
- cs1_b mean value changed from: nan -> nan
- cs1_b_wo_timecorr mean value changed from: nan -> nan
- cs2_b mean value changed from: nan -> nan
- cs2_b_wo_timecorr mean value changed from: nan -> nan
- s1_b_area mean value changed from: nan -> nan
- s1_b_area_fraction_top mean value changed from: nan -> nan
- s1_b_interaction_drift_time mean value changed from: nan -> nan
- s1_b_max_pmt_area mean value changed from: nan -> nan
- s1_b_r_fdc mean value changed from: nan -> nan
- s1_b_r_field_distortion_correction mean value changed from: nan -> nan
- s1_b_range_50p_area mean value changed from: nan -> nan
- s1_b_range_90p_area mean value changed from: nan -> nan
- s1_b_rise_time mean value changed from: nan -> nan
- s1_b_x_fdc mean value changed from: nan -> nan
- s1_b_y_fdc mean value changed from: nan -> nan
- s1_b_z mean value changed from: nan -> nan
- s1_b_z_dv_corr mean value changed from: nan -> nan
- s1_b_z_naive mean value changed from: nan -> nan
- s2_b_interaction_drift_time mean value changed from: nan -> nan
- s2_b_r_fdc mean value changed from: nan -> nan
- s2_b_r_field_distortion_correction mean value changed from: nan -> nan
- s2_b_x_fdc mean value changed from: nan -> nan
- s2_b_y_fdc mean value changed from: nan -> nan
- s2_b_z mean value changed from: nan -> nan
- s2_b_z_dv_corr mean value changed from: nan -> nan
- s2_b_z_naive mean value changed from: nan -> nan
All other columns remained the same
Change report for 'event_ms_naive':
All other columns remained the same
Change report for 'event_top_bottom_params':
- alt_s1_range_50p_area_bot mean value changed from: nan -> nan
- alt_s1_range_50p_area_top mean value changed from: nan -> nan
- alt_s1_range_90p_area_bot mean value changed from: nan -> nan
- alt_s1_range_90p_area_top mean value changed from: nan -> nan
- alt_s1_rise_time_bot mean value changed from: nan -> nan
- alt_s1_rise_time_top mean value changed from: nan -> nan
All other columns remained the same
#################### Lowest Levels Plugins ####################
Change report for 'event_nans':
- event_nans bad fraction changed from: 0.45 -> 0.1
- event_negs mean value changed from: -0.45 -> -0.1
All other columns remained the same
#################### Directly Affected Plugins ####################
Change report for 'event_basics':
- alt_s1_area mean value changed from: nan -> nan
- alt_s1_area_fraction_top mean value changed from: nan -> nan
- alt_s1_interaction_drift_time mean value changed from: nan -> nan
- alt_s1_max_pmt_area mean value changed from: nan -> nan
- alt_s1_range_50p_area mean value changed from: nan -> nan
- alt_s1_range_90p_area mean value changed from: nan -> nan
- alt_s1_rise_time mean value changed from: nan -> nan
- alt_s2_interaction_drift_time mean value changed from: nan -> nan
All other columns remained the same
Change report for 'event_positions':
- alt_s1_r_fdc mean value changed from: nan -> nan
- alt_s1_r_field_distortion_correction mean value changed from: nan -> nan
- alt_s1_x_fdc mean value changed from: nan -> nan
- alt_s1_y_fdc mean value changed from: nan -> nan
- alt_s1_z mean value changed from: nan -> nan
- alt_s1_z_dv_corr mean value changed from: nan -> nan
- alt_s1_z_naive mean value changed from: nan -> nan
- alt_s2_r_fdc mean value changed from: nan -> nan
- alt_s2_r_field_distortion_correction mean value changed from: nan -> nan
- alt_s2_x_fdc mean value changed from: nan -> nan
- alt_s2_y_fdc mean value changed from: nan -> nan
- alt_s2_z mean value changed from: nan -> nan
- alt_s2_z_dv_corr mean value changed from: nan -> nan
- alt_s2_z_naive mean value changed from: nan -> nan
All other columns remained the same
#################### Highest Level Affected Plugins ####################
Change report for 'event_info_double':
- cs1_b mean value changed from: nan -> nan
- cs1_b_wo_timecorr mean value changed from: nan -> nan
- cs2_b mean value changed from: nan -> nan
- cs2_b_wo_timecorr mean value changed from: nan -> nan
- s1_b_area mean value changed from: nan -> nan
- s1_b_area_fraction_top mean value changed from: nan -> nan
- s1_b_interaction_drift_time mean value changed from: nan -> nan
- s1_b_max_pmt_area mean value changed from: nan -> nan
- s1_b_r_fdc mean value changed from: nan -> nan
- s1_b_r_field_distortion_correction mean value changed from: nan -> nan
- s1_b_range_50p_area mean value changed from: nan -> nan
- s1_b_range_90p_area mean value changed from: nan -> nan
- s1_b_rise_time mean value changed from: nan -> nan
- s1_b_x_fdc mean value changed from: nan -> nan
- s1_b_y_fdc mean value changed from: nan -> nan
- s1_b_z mean value changed from: nan -> nan
- s1_b_z_dv_corr mean value changed from: nan -> nan
- s1_b_z_naive mean value changed from: nan -> nan
- s2_b_interaction_drift_time mean value changed from: nan -> nan
- s2_b_r_fdc mean value changed from: nan -> nan
- s2_b_r_field_distortion_correction mean value changed from: nan -> nan
- s2_b_x_fdc mean value changed from: nan -> nan
- s2_b_y_fdc mean value changed from: nan -> nan
- s2_b_z mean value changed from: nan -> nan
- s2_b_z_dv_corr mean value changed from: nan -> nan
- s2_b_z_naive mean value changed from: nan -> nan
All other columns remained the same
Change report for 'event_ms_naive':
All other columns remained the same
Change report for 'event_top_bottom_params':
- alt_s1_range_50p_area_bot mean value changed from: nan -> nan
- alt_s1_range_50p_area_top mean value changed from: nan -> nan
- alt_s1_range_90p_area_bot mean value changed from: nan -> nan
- alt_s1_range_90p_area_top mean value changed from: nan -> nan
- alt_s1_rise_time_bot mean value changed from: nan -> nan
- alt_s1_rise_time_top mean value changed from: nan -> nan
All other columns remained the same
Already on 'track_lineage_changes'
Your branch is up to date with 'origin/track_lineage_changes'.
Before you submit this PR: make sure to put all operations-related information in a wiki-note, a PR should be about code and is publicly accessible
What does the code in this PR do / what does it improve?
See this issue for the motivation of this script: #1259
Can you briefly describe how it works?
Every time a PR is made, this script can be ran to find out which new plugins are added, which are deleted, and what the change in the proportion of number of NaNs (0 or -1) for float (integer) fields are. This script also finds which plugins are the lowest-level changed plugins, and checks for changes in the plugins one layer up the dependency tree.
Can you give a minimal working example (or illustrate with a figure)?
There are two branches made for testing:
test_lineage_changes
andtest_plugin_changes
.test_lineage_changes
adds a pluginS2FWHM
(which does what it sounds like) as well as a new data typeevent_nans
provided by theGiveNans
plugin which adds a float field that is half-filled with nans, and an integer field that is half -1's. Thentest_plugin_changes
deletesS2FWHM
and replaces it withS2FWHM2
which puts in a NaN to the first element of the chunk in thes2_fwhm
field. Theevent_nans
intest_plugin_changes
is also changed to only place a nans/-1 in the first element of the chunk. The script is ran like so:./check_lineage_updates.sh <old_branch> <new_branc> <output_folder> <run_id>
And an example can be:
./check_lineage_updates.sh track_lineage_changes test_plugin_changes /scratch/midway2/jyangqi/strax_data 025423
Here's the output of that command:
Please include the following if applicable:
Notes on testing
All italic comments can be removed from this template.