Slow search experience

[JuhaS]

• User 1 creates a new entry in a collection
• User 2 only shows that update in her list 4 minutes later
  • To be clarified on if refreshing helps
  • Should we implement light polling

[JuhaS]

@woodpig07 can you maybe give your ideas on what could be good approach.

Few notes:

• We probably need socket.io/polling anyway, I'd prefer websocket for events of all modifications to the repo. Additional benefit is that we could leverage it for notifications like Chopper edited collection entry "Test Entry".
• We probably need some kind of iterative indexing.
  • One option is that we do this on commit/commit basis. This means backend knows it has indexed up to commit-x and date-y. When it gets request to re-index it will fetch newest commit number and check what files have been changed between the two commits. It then (re)-indexes those files.
• We need to update how we trigger indexing.

As a short term solution that targets the usecase where file-changes happen through jekyllpro-cms:

• add websocket on file-changes and re-fetch index on event
• when file is modified through jekyll-cms update the index for that file
• drawback: changes done directly in github still require full hard-refresh of index.

[woodpig07]

I agree on the use of socket.io to notify frontend about the modification to repo, but I believe it would act like a publish/subscribe service for the index document in mongodb, not directly triggered by user commits. Currently we only update the index in a passive way by refreshing the app page. I had built a webhook to update index incrementally based on commit data, I'd like very much to take advantage of that for a real-time (or near real-time) experiences.

Proposed work flow:

1. User 1 browses on cms.jekyllpro.com/repo, the App presents the files extracted from the index saved in database, and websocket client start to listen on change events.
2. Somewhere User 2 make a change to the repo, the webhook catches it and starts to update index quickly as it only has to process the files related to this commit.
3. After it successfully write to database, socket.Io publishes it to client, then frontend requests to load latest index data, all the users on cms.jekyllpro.com/repo will see the updated file list/content.

[JuhaS]

I had built a webhook to update index incrementally based on commit data, I'd like very much to take advantage of that for a real-time (or near real-time) experiences.

Is this already in the repo? Can @haishanh leverage that already?

About using GH webhook I think we should primarily design without requiring this since this will require manual setup from the user side. Instead I would recommend this:

• Modify index-refresh task to be incremental:
  • store commit number of last-refreshed index in db
  • fetch the latest commit nunber from github
  • return immediately if last commit is same as in db
  • use "Compare two commits" endpoint to get list of changed files
  • Index the changed files and write down the new commit-id to database
  • Send websocket event when index updates
• Trigger index-refresh on:
  • Setup polling on frontend every minute to trigger index-refresh
  • When file is being modified through our API trigger index-refresh
• Set up websocket to listen for index-updates for this repo

@woodpig07 do you think this would be ok?

[JuhaS]

Talked with @haishanh

Technical approach (first steps):

• Refactor the collections field to be an object with file path as a key.
• Add the lastIndexedCommit field to the database (per repo+branch)
• Implement logic to trigger re-indexing, fetch the changed files from GitHub API and index the files.
• Trigger re-indexing on any file-change through the API

Next:

• Socket.IO to inform client on index-updates
• Implement polling on frontend

[haishanh]

Sorry for delaying...

This issue is becoming very critical now, as the sbux website repo have over 360 entries. And it's extremely slow at displaying all content after new entry is added.

In this PR https://github.com/Wiredcraft/jekyllpro-cms/pull/61

I've added a new API endpoint /api/repository/updated-collections, which will only return modified/removed entries if there are changes between 2 commits. The frontend will call this API every 8sec.

The Socket.IO feature is not implemented yet.

As this issue is becoming a blocker for the marketing team. I'd like to land the PR ASAP.

@JuhaS @woodpig07 Can you help review the PR?

[woodpig07]

@haishanh please go ahead merge your PR, I left a few suggestions there, hope that's helpful.

[woodpig07]

not relevant anymore