Update example dataset

[sizun]

The example dataset NectarCAM.Run0890.10events.fits.fz used in the unit tests does not follow the latest data format (it follows the old CDTS data structure) so I suggest replacing it. Any preference regarding the actual events in the file? And how can I upload it to http://cccta-dataserver.in2p3.fr ?

[jlenain]

It would indeed make sense to replace this file with one with the latest data format. No particular preference on the run/event type. I think we should get in touch with Luisa and/or @bregeon for a recipe on how to upload files to the cccta-dataserver.in2p3.fr server.

[sizun]

Replacing the test data file turns out to be more complicated than I thought. The main motivation was for the unit tests to actually test the part of the NectarCAMEventSource class that is relevant for the current data format.

Since only 262 modules are currently cabled in the camera and a file with all 265 modules would be best, we have to rely on an already existing run and we cannot produce a new run of only 10 events with 265 modules.

Existing runs with 265 modules (like muon run 3120 are too large. We can use the tool zfitsrecompressor to extract 10 events but since odd and even numbered events are in different files and zfitsrecompressor takes only one file as input, we would not get consecutive events.

Currently, it seems oe of the unit tests checks that event numbers are consecutive.

I can see 3 possibilities:

• wait until we can cable the 3 missing modules to produce a custom run with just one ZFitsWriter instance so that the first ten events end up in the same file;
• remove the unit test that checks for event numbers to be consecutive;
• produce two 5 five events data files with zfitsrecompressor and modify the tests so that they ensure that the reader can reads events from multiple files in the right order.

[sizun]

@maxnoe I prepared 2 small files to replace the example data used by the ctapipe_io_nectarcam unit tests. I see that you last modified the ctapipe-extra project before it was replaced by cccta-dataserver.in2p3.fr/data/ctapipe-extra Luisa Arrabito told us that she has access to that server but is that how we should proceed? And where should the new files be placed (I guess not in the v0.3.3 subfolder)?

[maxnoe]

@sizun I think you should ask for your own directory on the data server or host it yourself somewhere.

We did this as well for the lst test data, which needs to be password protected.

We are reorganizing the test data and I'd not further add to what is in there.

[sizun]

Is this LST specific directory also on cccta-dataserver.in2p3.fr ? Because I now see in ctapipe_io_lst that you no longer use the method get_dataset_path that calls download_file_cached.

[maxnoe]

No, the lst test data is hosted on a dortmund server

[jlenain]

@sizun acquired a short NectarCAM run, 4717, yesterday, which can be used again in unit tests within ctapipe_io_nectarcam. @kosack uploaded it today on the cccta-dataserver.in2p3.fr data server.

Unit test can then be re-enabled again.

However, see #41 currently failing.

[jlenain]

Closing this issue now, since we have new example datasets available. Thanks @sizun for this run!