Open raulcf opened 8 years ago
Some evidence: in data.gov 30/10K files contain ~50% data
Interesting stat.... What does it mean? Amount of data in mbs? Or number of rows?
We could find these stats for BT's warehouses if that helps?
Simon On Wednesday, October 26, 2016, Raul notifications@github.com wrote:
Some evidence: in data.gov 30/10K files contain ~50% data
— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.< https://ci3.googleusercontent.com/proxy/3KYElAXuMB7ZvhVBaiLCSFqX09K_1FicMFlo_nJBIe1Szg41zZIW_wI8zXpX3SYuM-KbQPiV6Qww_B3uNG9I1FVDK852yCmrgoesYeMImohyrG-FyiqFPx9tMoo4xrb71SZcS75wYXjFQ1uQvigwM9kP28rb-Q=s0-d-e1-ft#https://github.com/notifications/beacon/AATC_qy74A2hmIBU77xmvpCGfVs0kY50ks5q30zRgaJpZM4JQwVL.gif>
This is very interesting problem. @raulcf how far are you with this issue? I think I can help here.
Right now is table to avoid redundant reads. By splitting data on memory we could provide more fine-granular parallelism, i.e. per column, while still avoiding redundant reads.