Implementing a remote model implementation

[edwardbr]

Hi I love what you have been doing, however I need to have large remote datasets that change the front end real time.

I have used the remote model code https://github.com/6pac/SlickGrid/blob/master/slick.remotemodel.js in the past with great success. Could it be possible to provide a hook to provide one's own custom remote model rather than the DataView provided?

Regards

Edward

[ghiscoding]

You're asking to use something different than DataView, the problem is that all of the lib is based on it. It's not just about loading the data, the biggest reason that I use the DataView is all of what you can do with it easily. Things that I know would break without it, and would require more code to implement, are like Sorting, Filtering, Grouping. So I did use DataView on purpose and that is the only 1 that I want to support. Providing another way to pass the data argument on grid creation is easy, but you will have to implement all functionality yourself, is that really what you want?

However, why don't you use DataView? You can also do live data with it, by using setItems() or with updateItem(). This way, you would still be able to use all features of this lib.

For backend services, I created 2 of them (OData & GraphQL), if that doesn't satisfy you and you wish to use your own implementation, well that is exactly what 1 user did and he provided a demo and here's the code. But even all the backend services uses the DataView, it's easier to support when all use the same thing.

So all that to say, I could provide another way to pass 2nd argument of the grid creation to pass data but I'm not responsible for all the extra work that will be required. I suggest to look into making your own backend service, as shown in previous paragraph

[edwardbr]

Many thanks I already have a service where the metadata and the sorting is done server side, For reasons of performance and quantity of data not all records are uploaded to the frontend only those that are visible. This service also pushes updates to that visible part of the dataset. I will for now continue to use native slickgrid, but I look forward in seeing what you do with this going forward. Regards Edward

[ghiscoding]

Unless you want to help, you will wait forever. I mean, I am happy with what I have working. The backend service handles data well and we also have too much data to show, which is why I added pagination (currently only available with backend service).

For reasons of performance and quantity of data not all records are uploaded to the frontend only those that are visible.

Isn't that pagination? If not, then please explain

[edwardbr]

Maybe I have misinterpreted something. My concerns with paging is if you insert elements inside a page after the page has been created, you have either the problem of getting ever expanding pages, or needing to shuffle all your last elements in all the subsequent pages over by one. It also makes page numbers a bit confusing, as the size of the resultset grows and shrinks. The way I have worked is a sliding result set that includes an offset and count from the resultset managed on the server, and inside that a sub resultset of data of rows that are visible, as the grid gets scrolled the hidden rows are made visible to give a smooth experience, while in the background more data is fetched and records moving outside of the invisible range are deleted. I could not see how to do that easily with the DataView, but perhaps thats my error.

[ghiscoding]

Ah yes I see for the pagination, on our side (and most people in general) we only need the page size from the start. I'm not sure if I handled the pagination re-calculation after CRUD on items (I could double-check that part), but I know it's being recalculated after changing page since it's part of the json result set (totalItemsCount is a property returned by result set from backend).

As for the invisible stuff, I think you're talking about the SlickGrid caching of the rows, which is not in DataView but really in SlickGrid core itself and if I'm not mistaking it's something like 3 or 5 rows on top/bottom that are cached. Since this is a core feature, I'm not sure that there's any changes in performance for using (or not) DataView.

The methods used by DataView for CRUD are the following

this.dataView.insertItem(row, item);
this.dataView.updateItem(item.id, item);
this.dataView.deleteItem(item.id);

// refresh dataview & grid
this.dataView.refresh();

// optionally, you can also invalidate all rows & refresh grid
this.grid.invalidate();
this.grid.render();

So at the end of the day, I've never used the remotemodel.js but I made my own implementation of a Backend Service (for OData & GraphQL) and I think it's working well but again I don't do much data change and/or real time like you seem to do on your side.

[ghiscoding]

I could maybe add a new grid option flag enableDataView that would be set to true by default, and when that is set to false, it would not use any Sort or Filter and so you'll need to implement yourself. However, are you going to need Pagination? Because right now, it only shows up with Backend Service (OData/GraphQL), I didn't do work yet in showing it with regular JSON dataset. For the pagination to work, it would at least require the total item count to work. The 2nd problem would be that none of the Filter would work, because again it's implement with the DataView.

I still think creating your own Backend Service would be the easiest and best thing to do, since it will require you to follow the interface which requires you to implement callback methods like buildQuery, updateFilters, updatePagination (see BackendService interface), ... Going this route, would be great because you can use all the built in stuff like Filters, Sorting, Pagination, Export to CSV and even Grid State. I think what could be great addition in the future is to make the Backend Services totally optional (as separate repos) that you could import if need be, but that's really future future plan (nothing in the short term), and when that is in place then anyone could build their own backend service of their choice (and publish on NPM if they wish to make it available to everyone or just keep locally).

If you still prefer to go back with your first option, to support remotemodel.js, I would be really interested to know if there's any performance gain in going that route.

So as I mentioned in first paragraph, I can add a new flag to disable DataView and pass your own data loader on the grid creation (new Slick.Grid("#myGrid", loader.data, columns, options), that's not really hard (the flag would also be used to remove local sort/filter handling, since there's no DataView available). I'm open on that subject, as long as it doesn't take too much of my time.

[edwardbr]

I have only been working on Angular for the past 5 days so I am trying to run before I can walk.

The backend service that I have implemented takes one restful interface for creating or updating the sorted list on the server. and a second web push interface that delivers updates to the grid generated from that sorted list. As I am somewhat under pressure I will have to hack something in for now, but when I can I will revert to this at a later date. I will be happy to catch up on this at another date.

The backend is in async C++, here's the interface definition for it if you are interested:

response get_sorted_rows(
    const string&                      query,
    //empty on the first call
    const string&                      sorted_list_id,
    std::vector<sort_item*>&    sort_info,
    uint64_t                               first_row,
    uint64_t                               row_count,
    std::list<query_field*>&       requested_fields,
    bool                                     register_for_push,
    sorted_list_response&        resp);

[ghiscoding]

Wow I didn't even know you could run this in C++, but I'll believe you 😮

[edwardbr]

Yes masochism has its benefits!

[ghiscoding]

I just saw that slick.remotemodel.js was made for Octopart, how do you use this then? I don't see a way to pass any URL as a public method (though I didn't look thoroughly). I also find that strange that this plugin is not in the plugin folder. Can you provide a piece of code on how you instantiate it.

[edwardbr]

No I hacked it for my purpose: This now works for me, some prettyification and bug fixing still required, a lot of this is derived from some javascript I wrote almost 10 years ago:

remote-model.service.ts.txt notification.component.ts.txt

[ghiscoding]

So you basically just need my lib to have a way to pass the 2nd argument this.remoteModel.data from this line of your code new Slick.Grid('#grid1', this.remoteModel.data, this.fields, this.gridOptions);

As I said earlier, I can add a flag enableDataView that would be set to true by default (which you would set to false on your side) and when set to false would provide a way to override that 2nd argument. I guess I would also need an @Input() remoteData that you can override in your code with this.remoteModel.data. The flag would also disable built-in onSort and onFilter that requires the DataView, meaning that you would need to implement them yourself.

EDIT For naming, I'm thinking this might be better... enableCustomDataView and @Input() dataView

Thoughts?

[edwardbr]

I am still getting up to speed on the nomenclature of Angular, half my brain is stuck in javascript :-).
One thing that is important is being to relay data that is not rendered in the grid but kept for back references and versioning.
Also there needs to be a way of direct rerendering a grid's particular row of the grid itself once a web push event occurs. Ideally I would like to move most of the code in notification.component into its own controller, rather than have it copied and pasted in all components, as one would have to do at the moment.

[ghiscoding]

I always provide a way to get reference to the SlickGrid Grid & DataView objects (see Wiki, because Angular-Slickgrid doesn't do (and will never do) everything that the core SlickGrid does. So you can always refresh/re-render any cell/row from the grid the old fashion way. I created some methods to re-use that code to make it easier for the user, but technically, it still calls the render, invalidate and all these methods right from the grid object.

I'm also still stuck in the JS world and on top of that I deal with 2 different frameworks (Angular at work, Aurelia on personal side). So I totally know what you mean. Angular-Slickgrid has a lot of JS usage since SlickGrid core is a JS lib, the major thing that I've done is to create a wrapper for SlickGrid to work in Angular and to add TypeScript. So typically, the lib tries to avoid the user typing any of the old code (like new Slick.Grid(...) but still permits it (see first paragraph). I'm still a huge contributor to the SlickGrid core lib (I'm the biggest contributor after 6pac itself), for example I made the Grid Menu plugin in the core and import it in my lib.

Also the reason I was asking about Remove Model as a Plugin, was because I recently made all SlickGrid Controls/Plugins as separate classes in my lib (called extensions in my lib, see extension list). Creating a separate Extension Class for that Plugin would be great, but I wouldn't want to do it if it's strictly made for Octopart only.

[edwardbr]

I strongly agree about not locking yourself in.

Perhaps creating a service passed into the remote model to call whoever's API is the way to do it, The remote model only really needs access to an Observable that does the work of function get_data(offset, count), the response then fiddles with the remote model's data object and then calls:

this.onColumnsLoaded.notify({fields: resp.fields}); and this.onDataLoaded.notify({from: this.from_pos, to: this.to_pos});

onColumnsLoaded is only needed to be called if the schema of the data being returned is different to what was before.

I'm under a bit of time pressure, this is definitely something I'm coming back to!

[ghiscoding]

I made a quick version 1.8.3 with the changes that I wrote earlier

For naming, I'm thinking this might be better... enableCustomDataView and @Input() customDataView

So in theory, but haven't tried yet, you could do something like this

<angular-slickgrid 
     gridId="grid2"
     [columnDefinitions]="columnDefinitions" 
     [gridOptions]="gridOptions" 
     [dataset]="dataset"
     [customDataView]="customDataView"
     (onAngularGridCreated)="angularGridReady($event)">
</angular-slickgrid>

and use the new customDataView with an instance of the RemoteModel, something like this

declare var Slick: any;

export class MyComponent {
  columnDefinitions: Column[];
  gridOptions: GridOption;
  dataset: any[] = [];
  customDataView: any;
  gridObj: any;
  loaderDataView: any;
  loading = false; // spinner when loading data

  ngOnInit() {
    this.loaderDataView = new Slick.Data.RemoteModel();
    this.customDataView = this.loaderDataView;

    this.hookAllGridEvents();
    this.hookAllLoaderEvents();
  }

  angularGridReady(angularGrid: any) {
    // the Angular Grid Instance exposes Slick Grid object
    this.gridObj = angularGrid.slickGrid; // grid object
  }

  hookAllGridEvents() {
    this.gridObj.onViewportChanged.subscribe((e, args) => {
         const vp = this.gridObj.getViewport();
         loader.ensureData(vp.top, vp.bottom);
    });
    this.gridObj.onSort.subscribe((e, args) => {
         loader.setSort(args.sortCol.field, args.sortAsc ? 1 : -1);
         const vp = this.gridObj.getViewport();
         loader.ensureData(vp.top, vp.bottom);
    });
  }

   hookAllLoaderEvents() {
     this.loaderDataView.onDataLoading.subscribe(() => {
       this.loading = true;
       // do something
    });

     this.loaderDataView.onDataLoaded.subscribe(() => {
       this.loading = false;
       // do something
      });
   } 
}

That's about it, the code is based on this remote model example

[ghiscoding]

@edwardbr It's all done and working... woohoo 🙈 🐵 ... well I think it works, I'm now getting 403 because I tried too many times I guess.

I made a new Example 18 demo with the Octopart (basically the same as SlickGrid Octopart Example). and here's the Component file for it, so check it out. There's a little piece of jQuery in the demo, for the text search input, but I'm out of time on this, so I'll leave it as is.

1 thing to note, I ended up removing the flag enableCustomDataView since checking that the @Input() customDataView is more than enough.

Since this is all done, I'll close the ticket

Also please up vote ⭐️ if you haven't already Thanks for feedback

[edwardbr]

That’s really awesome! I will have a go with this. I’ve only just come up for air from a rush for a demo, but if this makes things easier (as I’m sure it will) I’ll be very promptly be swapping mine out.

[ghiscoding]

@edwardbr I did some more refactoring on that demo, and got it all working. I did some changes in Angular-Slickgrid to do more check when we use custom dataview, also I made the header menu sort icon working (it wasn't working anymore). Lastly, Chrome tags this demo as "Not Sure" because it pulls the remotemodel.js file inside the Component and so Chrome tags it as insecure, so if you want to see the demo working, you will get an error and then you need to accept security. Anyhow, here is an animated GIF of this demo (which is the same SlickGrid AJAX demo with Octopart)

If you like, make sure to up vote my lib ⭐️ 😉 Cheers, I'm officially done on that issue :)