Support array row format

[mike-lischke]

*Is your feature request related to a problem? Please describe.** No.

Describe the solution you'd like Currently a row in the table data has to be an object, with the keys corresponding to column ids. However, in large data sets this leads to a lot of key duplication.

Instead it should be possible to pass in rows as arrays, where each entry belongs to one of the columns based on its index (without id matching, just the order matters). I'm aware that this might conflict with column reordering, so I think when giving data as pure arrays, column reordering can implicitly be disabled.

Describe alternatives you've considered There are no alternatives, except for the current solution.

[olifolkerd]

Hey @mike-lischke

Thanks for the suggestion

What is to stop you casting the the arrays to objects before passing them to Tabulator, using the array index as the column prop name?

Do you have more details about your usage case that would help me put this in context and why things cant be mapped to objects.

This would be quite a big change in places and there are numerous table configurations that this would cause issues for. i would rather not disable functionality as while it may not be needed in your usage case it might be needed in others, so i would prefer to understand the challenge in more detail and see if a more comprehensive solution is possible.

Cheers

Oli :)

[mike-lischke]

A typical use case are results from SQL queries. What you get back are arrays where each entry corresponds to a table row. Column information is not part of that array. Columns have to be determined by other means (e.g. drivers often return field infos in the first response or by other means). Obviously a row which only contains of values is much smaller compared to a row with objects that contain a column id and a value for the cell they address.

What stands out in the object approach is that there is a lot of duplication. Every row contains the column ids, even though all rows have the same layout. The object approach is good for quick display of small data sets, but when it comes to thousands or millions of rows that changes, because an application that gets it's data from a database (which is often case for such large data sets) has to convert all the rows with single values to objects with column id and the value. Quite a waste of CPU cycles.

For your suggestion to cast the array: I don't get the idea, to be honest. Assume you get a DB result like this:

[[1, 2, 3], [4, 5, 6], [7, 8, 9]] 

I cannot pass this directly to tabulator, which expects a format of:

[[{ "col1": 1}, { "col2": 2 }, { "col3": 3 }], [{ "col1": 4}, { "col2": 5 }, { "col3": 6 }], [{ "col1": 7}, { "col2": 8 }, { "col3": 9 }]]

Casting the array with the single values to one that contains objects is plain wrong and would not help.

Note: I don't say the data cannot be converted to the format expected by tabulator (in fact that's what I do currently). What I want is a simpler format of the input for tabulator. Just an array of rows, each with a single value for the corresponding field. The column association is given by the value's array index (which is implicit).

[olifolkerd]

Hey @mike-lischke

Thanks for the details, i understand the the data source issue no problem there.

But i don't understand why mapping the object would be "plain wrong", (though i would say your proposed mapped structure is a bit overly complex) if you are worried about the properties taking up space in the transfer then i totally get that, but once you are client side that then isn't an issue, the props are just pointers at that point and don't cause excessive memory consumption (i've worked with some VERY big tables and not had any issues).

Why not simply map the row array to an object that has the same prop names as the cell indexes:

var data = [[1, 2, 3], [4, 5, 6], [7, 8, 9]];

//data formatted to pass into tabulator
var tableData = data.map((row) => {
    return Object.assign({}, row);
});

this would give you mapped data in this format:

[
    { 0: 1, 1: 2, 2: 3 },
    { 0: 4, 1: 5, 2: 6 },
    { 0: 7, 1: 8, 2: 9 }
]

That would very quickly and easily map the data to a format that Tabulator could interpret. If i was to look at allowing Tabulator to handle row arrays, this is exactly what i would need to do internally anyway as it would break the relationship between data position and column, meaning column moving etc would still work.

It would also have the added benefit that you could do a similar thing with your column header info to generate matching column definitions

Getting data out, if you wanted to send it to a server etc, would then be a trivial matter of mapping the data back from the object back to the array using its field value as its index in the array.

Cheers

Oli :)

[mike-lischke]

You didn't say "mapping the object" but "casting the the arrays to objects before passing them to Tabulator". Casting is not the same as mapping. The latter creates a new structure.

About using indices instead of column ids: that's essentially the same approach, except that with indices you would have to assign your columns those indices instead of speaking IDs. I believe these indices are not only used for assigning data, but also in other places (maybe even in HTML, not sure).

In any case, this approach creates a duplicate of the original data, which is bad. Data might not just be loaded and displayed, but might be kept in a cache for frequent switching between the different data sets. And there can be other scenarios where every byte counts. Duplicating column IDs a million times (for large data sets) seems crazy and is actually not needed.

Anyway, I'm only suggesting a feature which would make professional use of the grid even better.

[olifolkerd]

Hey @mike-lischke

I'm just trying to understand your usage case better so I can deliver features that fit their intended usage case.

And the responses I'm getting are feeling quite defensive, I'm not having a go I'm just trying to better understand what your approach is and rather than constructive feedback this is starting to feel condescending and snarky.

So I'm gonna take a break from this conversation. I think this is a valid idea and a great addition, but unless I can discuss approaches with you constructively I won't be taking it further.

Thanks

Oli :)

[mike-lischke]

Oh, I didn't want to create a negative impression. Actually I'm a bit surprised it came along to you this way. All I said in my previous posting was:

1. The casting will not work.
2. Using indices instead of string IDs does no improve the situation significantly.
3. Key/value entries are not necessary, as you can get columns from the array indices.
4. I do not want talk you into a change, but I'm just suggesting something that would make my (and probably others) usage of Tabulator better.

I cannot recognise any defensive touch here. After all, I have nothing to defence in this thread ;-)

[olifolkerd]

@mike-lischke I would consider language like "plain wrong" to be unhelpful, it doesn't discuss an issue or offer feedback an idea, it is just unnecessarily critical with no justification.

In your response to my initial feedback, the data structure you proposed was not a compatible structure with Tabulator, there were objects for every value not every row, when i point to a solution to that issue with a correct structure, rather than meaning full discussion i get a "lesson" in cast vs map. I humbly apologies for absentmindedly mixing up a term when replying to one of 50 different issues while adding a load of new features to 5.3 in my free time on a Sunday.

I'm not looking to be talked into a change, I think this could be a useful addition, but if I just add a complex feature without understanding the usage case properly then it wont be fit for purpose, so I am trying to understand all of the areas of concern you have around how you will use this functionality so I can make sure that what I build works for everyone.

With all due respect to key/value entries not being necessary, i agree that on the surface that is the case, but there is a lot that happens below the surface on tabulator that could break a simple array, and i am trying to come up with an approach that would work for all usage cases of tabulator, so rather that just being super critical of each point i try to bring up, i would appreciate it if we could try and understand your usage case outside of the Tabulator internals a little more. I get that you want to use an array, but you have already identified this could break some features. i am not prepared to break any features with this new option and will need to come up with an approach that works for all so i would appreciate a bit more of an open discussion.

To do this i am going to need to ask you questions, propose options and gain understanding around your usage case.

Oli

[mike-lischke]

In your response to my initial feedback, the data structure you proposed was not a compatible structure with Tabulator, there were objects for every value not every row,

Yes, you are right. It should not have been array in array. This was the result of thinking about using just two nested arrays as data structure.

I humbly apologies for absentmindedly mixing up a term when replying to one of 50 different issues while adding a load of new features to 5.3 in my free time on a Sunday.

FYI, I'm also an open source developer, writing code in my spare time. So I know exactly what you mean.

I think this could be a useful addition,

Great!

but if I just add a complex feature without understanding the usage case properly then it wont be fit for purpose

Makes a lot of sense.

so I am trying to understand all of the areas of concern you have around how you will use this functionality so I can make sure that what I build works for everyone.

Do you need any more information or certain details from me to explain the idea?

i am trying to come up with an approach that would work for all usage cases of tabulator, so rather that just being super critical of each point i try to bring up

I know, this is the most difficult part. Such a change must not break any existing application/usage of the library, whatsoever.

i would appreciate it if we could try and understand your usage case outside of the Tabulator internals a little more

OK, just let me know what else I could explain.

I get that you want to use an array, but you have already identified this could break some features

I see one potential problem, yes: reordered columns, because the array index in a row would no longer match the actual order of the columns. Not sure if you do that already, but I could imagine tracing any reorder of columns could help here. This can be used to convert the data index to the real column index.

To do this i am going to need to ask you questions, propose options and gain understanding around your usage case.

Yes, please go ahead. I'm ready to answer.

[prakash4mail]

Even i was looking for same kind of data structure to support into tabulator. So i believe i can add some exact data points.

Now coming to the context for this kind of data is, mostly Stock market data which keeps coming in high frequency, so data providers avoid the field names and push it as array instead of objects to save the load. [example Trading view : https://www.tradingview.com/screener/ > look in developer console > Network Tab > Fetch/XHR tab > url: https://scanner.tradingview.com/global/**scan** .]

As for my data:

Example data that i have {"NIFTY 50":[15843.55,-6.65,-0.04,418512354475,32,60,100],"NIFTY BANK":[33387.65,-254.8,-0.76,261726808889,8,32,100]} Or array of array [['nifty',15843.55,-6.65,-0.04,418512354475,32,60,100],['bank',33387.65,-254.8,-0.76,261726808889,8,32,100]]

Now i need to display this data in tabulator: consider the fields as stock,price,change,changePer,etc..._ For both type of structure(provided above), What is the best way to display it and update it when comes from socket connection(fast flow) yet keeping it consistent with Tabulator internal structure for other functionalities to not to fail (also handling the UI rendering smoothly). (A little code snippet would help).

Yes i did read your comment above about converting it to object.assign... just one problem is that it destroys the link to org content src. I ain't an expert but one suggestion is enhancing code that deals with the tabulator's parameter of field.

I have tried this, so is this the best? (too much to write up making code a bit ugly )

new Tabulator('#example-table', {
    data: Object.entries(idc),
    columns: [{title: 'index', field: '0'}, {title: 'price', field: '1.0'},{title: 'change', field: '1.1'},{title: 'change Per', field: '1.2'},
              {title: 'Traded Value', field: '1.3'},{title: 'adv dec ratio', field: '1.4'},{title: 'adv dec vol ratio', field: '1.5'},{title: 'above vwap', field: '1.6'}]
})

Thank you

[olifolkerd]

Hey All.

I have added the new array importer to the 5.3 branch that will do as i initially discussed and take an array of arrays and convert it to objects, which will at least allow the importing of that sort of structured data into the table without mutation.

var arrayData = [
  ["Name", "Age", "Cheese"],
  ["Bob", 23, true],
  ["Jim", 44, false],
]

window.table = new Tabulator("#table", {
    data:arrayData, //load data into table 
    autoColumns:true,
    importFormat:"array",
});

I know this does not work in exactly the way you had desired but as outlined above the proposed enhancement would require a massive rebuild of part of the library to implement holistically. And while this may be something i consider later as i have been planning to add more cell based functionality in version 6.0, this is not something that will be coming in the near or medium term, the version 6.0 release is at least a year way at this point.

In response to @prakash4mail That is quite a specific data structure you have there, and im afraid it simply isn't possible to make tabulator ingest every conceivable array/object structure out of the box. In your case i would recommend that you structure your table using column groups to represent the categories and their same values, if you want them along the top, or if you want it arranged in rows, then change your data to push the object key on to the first element of each array and then ingest it as an array of arrays (as outlined above) with the column headings as the first element in the top level array.

Cheers

Oli :)

[prakash4mail]

Just updating with latest amazement to the awesomeness of your library. You have built your library internal very fundamentally to the point, so the below code worked 👍 without any issues. And i was surprised that this would work @mike-lischke (check if this kind hleps) .

new Tabulator('#example-table', {
    data: Object.entries(idc),
    columns: [{title: 'index', field: '0'}, {title: 'price', field: '1.0'},{title: 'change', field: '1.1'},{title: 'change Per', field: '1.2'},
              {title: 'Traded Value', field: '1.3'},{title: 'adv dec ratio', field: '1.4'},{title: 'adv dec vol ratio', field: '1.5'},{title: 'above vwap', field: '1.6'}]
})

My final question(or help needed to this thing) is:

1. Is there any alternate or best way, instead of "data: Object.entries(idc)" so that directly idc is taken or
2. how to keep passing idc that keeps from websocket continuous stream such that it is very reactive both in UI rendering as well as processing of data by Tabulator.

[olifolkerd]

Given that you are mapping the data, the reactiveData option built in to tabulator won't work.

I would suggest using the replaceData function and calling that whenever you have data to update