search

umccr / data-portal-status-page

0 stars 0 forks source link

Dynamic "Library Run" page population #55

Open reisingerf opened 2 years ago

reisingerf commented 2 years ago

The "Library Run" view does make a series of API calls to gather data before showing any content (I think one call per library). As those calls happen sequentially and take up some time, it looks like the page is stuck.

An improvement is to dynamically show content as data arrives. An additional improvement is to show a sorting order (as there does not seem to be an obvious order now). This could be the Library ID or the date a workflow (with that library) has run?

williamputraintan commented 2 years ago

An improvement is to dynamically show content as data arrives.

So the issue with this, is the grouping of the libraryId with its type. Since the default view is to group based on metadata type, it will need to query the /metadata endpoint for all libraries, grouping it, then we could display these values. I suppose this takes most of the time, as if waiting for workflow status will show a loading icon which will change accordingly when data arrives. I increased the pagination to 300 which takes some time to load this, I could bring this number down to have faster loading.

This could be the Library ID or the date a workflow (with that library) has run?

The current order is the ordered from the api which is the descending of the id (the internal Django Id), I suppose this it will sort based on the last record added to the LibraryRun database. We could set the order of this to the descending of library_id if it makes more sense. I think by date might not be appropriate, because the /libraryrun endpoint do not have date in the response. It is possible but I think it will have too much query.

As a side note on how the libraryRun page get data:

  1. Query /libraryrun endpoint
  2. From the result, query the /metadata endpoint to obtain metadata fields
  3. Group these items by type (from the metadata fields)
  4. Display these values in the tables
  5. When displaying rows, query the associated workflows dynamically (it will show loading icon when waiting for response)
reisingerf commented 2 years ago

Hm... I see. Thanks for the clarification!

In that case, yes dropping the default page size to 100 or even 50 might be worthwhile.

Perhaps we can think about offloading the sorting/filtering/pagination more onto the backend, but that's a separate discussion and probably for a v2 implementation...