Speed issues with more than 200 records in vuex

[riksnelders]

I'm troubleshooting a bug that causes a dispatch(module/insert | set ) to take over 1 second, repeatedly.

I'm aware of vuex not being able to handle alot of data (especially on mobile devices). And that is probably what's causing the issue. So It may have been a incorrect choice to use this package for this purpose. I'm quiet sure its CPU power related. reproducable when using chrome dev tab and setting cpu throttle to 4x

But this might be a good feature to add to improve speed? see this article

Is there any other way we can quickfix the vuex speed that u can think of right now?

EDIT: Might be a good idea to also mention this in the docs under caveats maybe ?

[mesqueeb]

Thank you for the issue. My library only syncs to firestore every one second, however it should update vuex immediately. You can also disable the one second delay on the sync. I have some documentation about this but I am outside now so I’ll comment again later

_{Sent with GitHawk}

[riksnelders]

@mesqueeb Altho, the dispatch still takes 1 second and sync timer is set to 1ms

[mesqueeb]

Can you share a screen cast ? Or can we do a live vscode session?

_{Sent with GitHawk}

[riksnelders]

Does this help ?

insert record 225069.45000000997 record inserted 225518.4350000054

when running

    console.log('insert record', performance.now());
        this.$store.dispatch('productionRuns/insert', {
          companyId: job.companyId,
          userId: this.getMe.id,
          job: _.omit(job, ['workflowStates', 'status']),
          userName: this.getMe.displayName ? this.getMe.displayName : this.getMe.email,
          workflowId: workflowId,
          uid: job.uid,
          orderNumber: job.orderNumber,
          product: job.product ? job.product : null,
          productionEnd: null,
          productionStart: new Date(),
          jobId: job.id,
          target: job.target ? job.target : 0
        }).then(() => {
          console.log('record inserted', performance.now());
          this.jobStartedToast(job);
        });

module config

const firestore = {
  firestorePath: '/productionRuns',
  firestoreRefType: 'collection',
  moduleName: 'productionRuns',
  statePropName: 'productionRuns',
  namespaced: true,
  serverChange: {
    convertTimestamps: {
      created_at: '%convertTimestamp%',
      updated_at: '%convertTimestamp%',
      productionStart: '%convertTimestamp%',
      productionEnd: '%convertTimestamp%',
    },
  },
  sync: {
    debounceTimerMs: 1,
    guard: ['created_at', 'created_by', 'updated_by']
  }
}
``

[mesqueeb]

Thanks for the code sample!

If you want to know how long it takes until the document is added in vuex you can use this hook: https://mesqueeb.github.io/vuex-easy-firestore/hooks.html#hooks-after-local-changes-before-sync

The timing that the promise of insert is resolved is when the Firebase SDK finishes the actual sync (see src code here), even if you have 1ms sync debounce set.

So basically, if you add a insertHookBeforeSync and you see how long it takes until that is triggered (and thus the doc is added to Vuex), you'll see that it's almost immediately. 😉

So I believe you just misunderstood when the promise of the insert action is resolved.

[riksnelders]

Ok. Thanks that helps. I'll debug a little further whats actually causing this to take so long :)

[riksnelders]

@mesqueeb Luca, u got discord? Still having troubles with this.. And I am not sure whether its Firebase itself, this package or vuex itself. See screenshot, logged performance.now()

update jobs = start of function starting the actualy heavy work cleared is clearing the module done is when openDbChannel Promise resolves amount of records loaded

edit: Question. Are the records fetched from firestore query added in a loop? Seems like it when Im using

serverChange: {
    addedHook: function (updateStore, doc, id, store) {
      console.log('added',doc);
      console.log(store)
      // updateStore(doc);
    },

Is there any batch support?

[mesqueeb]

Yes if you give me your username I can add you on discord.

_{Sent with GitHawk}

[Eronne]

Hi @mesqueeb & @riksnelders

Do you have any update on this? We are currently working on an application that handles sometime big datasets (around 2 or 3k results for one query that takes around 6s). Opening Vue devtool after query fetch crashes the tab.

[riksnelders]

Hi @Eronne.

Yes vuex and devtools crashes with bigger amount of records. I had the same issue. Doesn't seem to be an issue with vuex-easy-firestore though.

My only concern right now is that all the docs are added seperatly into the store. Seems to eat up some cpu, instead of batch mutation. @mesqueeb This might be somerthing to add later on :)?

concerning the complete freeze up with 100% cpu, it seemed to be a combination with another package (datatables) and rendering striped animated progress bars

edit: Application handles 2-3k records fine now. Takes 4-5 seconds to get and render everything . But the rendering is the slowpart.

[mesqueeb]

@Eronne I do not recommend using Vuex at all with large amounts of data. Vuex isn't performant enough for this I think. I'm currently working on a complete rewrite for v2.0 that will not rely on Vuex anymore. Vuex will be optional through a plugin, but the core of the library will just be JavaScript (TS). I'm sure you'll see much better performance if you switch away from Vuex for the local data storage, which is something my next version will support.

@riksnelders I don't see any way a "batch mutation" could add all documents at once without looping once over the documents. If you somehow retrieve all docs at once (which firebase's onSnapshot listener doesn't do anyway) and then "replace" the local "data" with the array or object with all docs, then this won't even be recognised by the front end because Vue 2.0's reactivity system doesn't support replacing the whole thing while keeping reactivity.

Besides, as you already found out:

it seemed to be a combination with another package (datatables) and rendering striped animated progress bars

And I'll repeat it again for @Eronne, Javascript looping over 1000s of records and just placing them in Vuex is probably not the bottleneck. I'm 99% sure that the reason there even might be slow performance is either because Vuex dev tools is open or because of how the front-end is implemented.

Reading about how to use the "performance tab" in google chrome is your best bet. I am usually able to find out what performance bottlenecks are through this.

I'll close this issue now, since it's a non-issue in my eyes, but feel free to keep the conversation going! 😄 Also, feel free to reach out on discord if you wanna discuss more. _{--
Vuex Easy Firestore was made with ♥ by Luca Ban.
If you use this library in your projects, you can support the maintenance of this library by a small contribution via Github 💜.
You can also reach out on twitter if you want a one-on-one coding review/lesson. 🦜}