search

icecube / skyllh

https://icecube.github.io/skyllh/
GNU General Public License v3.0
11 stars 5 forks source link

pandas dataframes #7

Open chrhck opened 4 years ago

chrhck commented 4 years ago

Dataframes are awesome. Do we want to support them? This would mean either switching over from numpy recarrays completely or adding an abstraction layer that can handle both..

HansN87 commented 4 years ago

Dataframes are awesome! I am in favor of supporting them. Completely switching probably requires some performance comparisons (found this: http://gouthamanbalaraman.com/blog/numpy-vs-pandas-comparison.html).

martwo commented 4 years ago

SkyLLH doesn't use numpy recarray internally. it's all 1d ndarrays. I would be surprised if Dataframes would be faster than that

chrhck commented 4 years ago

From the link @HansN87 posted above it seems Dataframes are always quicker for more than 500k rows, and potentially quicker for more than 50k rows, although having a larger memory footprint. For now I think a good starting point would be to add support for Dataframes in the usercode (e.g. when dealing with analyzing trials).