Table freezes with ~5,000 entries

[mitchhankins01]

Hi,

I'm fetching an array consisting of around 5,000 objects. As soon as I drag in a second field the entire table freezes and the page continues to hang indefinitely. Is there any way for me to handle this volume of data in pivottable?

I've tried the following by slicing the array

• 1,000 rows, snappy
• 2,000 rows, slight delay
• 3,000 rows, increased delay
• 4,000 rows, major delay
• 5,000 rows, complete freeze

I've also tried only pulling 3 properties off of each object, reducing the array. This unfortunately had no effect.

[nicolaskruchten]

Generally speaking, this table should handle dataset sizes of up to 100k rows, but there are other bottlenecks one can encounter, such as the number of cells in the resulting table. If you're creating a 5000x5000 table that's 25 million cells and I would expect this to cause problems. How many cells would you expect in the table you're creating? What's the product of the cardinalities of the two attributes?

As an example of correctly handling 100k rows, I can download this file and drag it into the UI at https://react-pivottable.js.org/ and create all sorts of tables. I've got a 2014 MacBook Pro running Yosemite and Chrome, for reference.

[mitchhankins01]

HI @nicolaskruchten,

Thanks for your response, my data consists of the following:

myArray = [
  { name: 'Some Name', value: 'Some Value', value2: 'Some Value2'  }; **Times 5,000**
];

For reference, here is my component code:

class PivotTable extends Component {
  constructor(props) {
    super(props);
    const data = props.data.map(each => {
      return { name: each.name, finData1: each.finData1, finData2: each.finData2 };
    });
    this.state = { data };
  }

  updateState(newState) {
    this.setState(newState);
  }

  render() {
    return (
      <PivotTableUI
        // data={data}
        onChange={s => this.updateState(s)}
        {...this.state}
      />
    );
  }
}

As you can see, I prepare the data by looping over the data provided in props and only return 3 values: a string, a number, and another number.

Considering this small amount of data, I don't understand why the table would freeze. To me, the cardinality seems appropriate.

[nicolaskruchten]

If you can give me a sample of data I can take a look. Just to be clear on the cardinality here though: does your data have a unique value for each key for each row? How often do keys appear in common across rows?

[mitchhankins01]

@nicolaskruchten Unfortunately I'm prohibited from sharing the data due to fed financial laws, but basically the data looks like

myArray = [
  { name: 'Some Name', financial_integer_1: 2847, financial_integer_2: 6324  };
 // ~ 5,000 of theses objects in this array
];

What I am trying to do is to get the 'names' to show up in a column, with the two following columns being sums, counts, or averages of financial_integer_1 and financial_integer_2.

I got it to partially work by using this section:

This works fine and snappy, however, using this method I can only select one attribute at a time, rather than see financial_integer_1 (totalPL in this case) and financial_integer_2 next to each other in columns. When I try to drag either financial values into the table is when it completely freezes; there is no issue when I drag in the 'name' attribute.

Hopefully this is enough information, if not, please let me know and I will do my best to provide it. Thank you for your help.

[nicolaskruchten]

OK so it's a reasonable bet that every value of financial_integer_1 is unique across all the records here right? And same for financial_integer_2? If so, then you're effectively trying to make a 25,000,000-cell table, which is why it's slow :)

[mitchhankins01]

@nicolaskruchten Ah that explains it, I figured since Excel has the same functionality this would be possible in a browser environment as well. I appreciate your explanation though!

[hamdiwanis]

Hello @nicolaskruchten, I'm facing the same issue here and I was wondering if we can work around this by doing pivot table expensive calculations in a web worker. would this work? and any hint's about what parts will benefit the most?...thanks

[nicolaskruchten]

The slow part here is React/the browser... the only way to work with enormous tables would be to virtualize them, but this is out of scope for this library :)

[hamdiwanis]

@nicolaskruchten thanks for your reply, I know this is part of the problem but if it was the only slow part wouldn't only affect the table rendered but not the charts? which not the case, so am I missing something?

[nicolaskruchten]

Charts suffer from the same problem: they're rendered in SVG so the browser can't handle millions of DOM elements there either.

[hamdiwanis]

@nicolaskruchten got it thanks alot