Open maxnoe opened 2 years ago
Hi @maxnoe
before we start using this version I think we should make sure that we have no problems reading those files. While you did the changes in pyeventio, it can still happen that there is a clash between the ctapipe and pyeventio versions and we might end up with needing to make some changes in lstchain to make it compatible with the ctapipe version that is compatible with the updated pyeventio. It might all work out of the box, but well, better to check it.
@jsitarek Since this only added new fields to the SimTelFile
, it is compatible with the current ctapipe versions (otherwise I'd have made it version 2.0, not 1.6.
It's actually the opposite: since the metadata are added by default using the current sim_telarray we already need the latest pyeventio, otherwise we will get the same error that you got when trying to analyze the files.
Hi @maxnoe Let me clarify: I did not mean this change in pyeventio, but if there were some other changes in the meantime that might not be compatible. About the change in sim_telarray, If I understand correctly for the moment is only in the "Testing" version. and I guess we are using one of the releases for the sim_telarray production. Of course I fully agree with you that if the used sim_telarray has this already implemented we need the new pyeventio either way.
Let me clarify: I did not mean this change in pyeventio, but if there were some other changes in the meantime that might not be compatible.
I don't understand what you mean by this. What other changes in the meantime?
I checked in detail. As far as I can see ctapipe needs eventio 1.5.0 - 2.0, lstchain actually requires >=1.5.1, which is basically the same since I do not see the release 1.5.0. After 1.5.1 there was also 1.5.2 and then 1.6.0 with your latest change. My worry was e.g. in 1.5.2. there could have been something added that is not compatible with the current ctapipe or lstchain, and before it was working because it can always downgrade to 1.5.1. This would not be possible if we require 1.6 version. But I see that my lst environment is actually using 1.5.2 version, and there are no more changes in 1.6 for the moment, so indeed it should be all fine, you can disregard my earlier comment.
The current lstchain tests are already running with eventio 1.6. The CI (and anybody setting up an environment) will always get the newest compatible version.
As said, we use (or at least try) semantic versioning, so any breaking change would have resulted in being 2.0, which would then be excluded by the ctapipe requirements. Of course, I could by accident make a breaking change without realizing it..
See also discussion here: https://github.com/cta-observatory/ctapipe/issues/1853
With Prod6, the metadata system will be used and future versions of ctapipe will very likely start relying on some of the metadata information being available. This includes for example using the telescope metaparams OPTICS_CONFIG_NAME
and CAMERA_CONFIG_NAME
to fill the subarray information and also using the global metadata SITE_CONFIG_NAME
ALTITUDE
*LONGITUDE
, *LATITUDE
for array location.
Here is how the LST metadata looks like in a current (2022-06-30) prod6 example run:
In [8]: s.telescope_meta[1]
Out[8]:
{b'NIGHTSKY_BACKGROUND': b'all: 0.244637 ',
b'NSB_AUTOSCALE_AIRMASS': b'0.84 0.29 ',
b'MIRROR_CLASS': b'0 ',
b'FOCUS_OFFSET': b'6.55 0 0 0 ',
b'FOCAL_Length': b'2800 ',
b'EFFECTIVE_FOCAL_Length': b'2923.7 0 0 0 0 ',
b'MIRROR_LIST': b'mirror_CTA-S-LST_v2020-04-07.dat',
b'CAMERA_DEGRADED_EFFICIENCY': b'1 ',
b'CAMERA_DEGRADED_MAP': b'none',
b'MIRROR_DEGRADED_REFLECTION': b'0.8 ',
b'PRIMARY_DEGRADED_MAP': b'none',
b'DEFAULT_TRIGger': b'AnalogSum',
b'CAMERA_CONFIG_FILE': b'camera_CTA-LST-234_analogsum21_v2020-04-14.dat',
b'PARABOLIC_DISH': b'1 ',
b'DISH_SHAPE_Length': b'2800 ',
b'MIRROR_FOCAL_Length': b'0 ',
b'DISCRIMINATOR_PULSE_SHAPE': b'LST_pulse_shape_7dynode_high_intensity_pix1s.dat',
b'ASUM_THRESHold': b'260.7 ',
b'ASUM_CLIPping': b'9999 ',
b'ASUM_SHAPING_FILE': b'none',
b'OPTICS_CONFIG_NAME': b'LST',
b'OPTICS_CONFIG_VARIANT': b'LST Prod-6',
b'OPTICS_CONFIG_VERSION': b'2022-06-15 (Prod-6)',
b'CAMERA_CONFIG_NAME': b'LSTcam',
b'CAMERA_CONFIG_VARIANT': b'LST Prod-6',
b'CAMERA_CONFIG_VERSION': b'2022-06-09 (Prod-6)'}
In the latest experimental release sim_telarray allows adding metadata to the output files.
It allows both exporting the final configuration options per telescope but also setting custom user attributes.
This seems very handy.
E.g. adding this:
to
CTA-PROD5-LaPalma-baseline.cfg
you get this withread_simtel -s -h <file>
(support in pyeventio is on the way):