search

cta-observatory / dl_data_dumper

Data Format specifications and converters for DL
3 stars 0 forks source link

question about data layout #1

Open kosack opened 6 years ago

kosack commented 6 years ago

Is there a particular reason you went with storing each data array separately, rather than in tables (where each data item is a column?)

If loading all data, it doesn't matter much, but if you want to load a slice (e.g. a subset of events), it makes it more difficult this way.

I only ask, because for the "real" DL1 format, we will likely need it to be tabular, so we can process event-wise. Was there a particular technical reason?

vuillaut commented 6 years ago

Hi Karl.

Storing in tables with events as lines would require a lot of empty cases to keep events aligned, thus making the data format inefficient in terms of volume or am I missing something?

Also if I am right, DL0 data will be organised telescope-wise. Re-organising the data event-wise will be very costly in time. If we can process data per telescope before to deal with stereoscopy (such as Hillas), it would be much more efficient.