Performance issue when changing entites in a loop

[sinasita]

I currently have a performance issue while adding data to a Vuex ORM model in a loop. Each time I add a new entry to the Vuex ORM model, the reactivity forces the attached Vue components to re-render. Of course, I could change the loop and collect all the entities in an array first and update them all at once, but in my case, there are complex checks involved and the entity-types have several relations to other entites, so it would be great to keep the transactions, as they are.

My idea is to start recording a transaction just before running the loop and afterwards committing the whole recorded transaction. The transaction would internally collect all changes and flush them to the Vuex ORM at once. This way there would be only one re-render and the whole process hopefully speeds up a lot. Or maybe interrupt reactivity somehow before starting the loop and turn it on again, when the loop is finished.

Do you know about such a feature or a plugin? Do you have such a feature planned for the future? If not, can you give me a hint how I could start e.g. by writing a plugin?

[kiaking]

@sinasita Thank you for the feedback! Yeah, the Vue will re-render everything when there's a change to the store.

My idea is to start recording a transaction just before running the loop and afterwards committing the whole recorded transaction. The transaction would internally collect all changes and flush them to the Vuex ORM at once.

This is a pretty nice idea I think. Currently, we don't have the plan to add this feature to the core, but maybe it's nice to have it. I think we need to do some PoC for this, but I would like to discuss how to implement this.

I think we should have method something like prepare and resolve (just an idea. We should come up with a better name).

User.prepare({
  data: {},
  method: 'insert'
})

User.resolve()

Now, the problem is, where do we store that data...? If we could store them inside the store, maybe it would be good...? We need to check that Vuex will not re-render any data change if the state is not referenced from anywhere.

Is it possible for you to start some level of design on this feature? Like how should the API be, and how data will be stored, and where, and such. Then, I think it's easier for me to guide where those methods should live.

[sinasita]

I think we have to look at the API and the storage separately.

About the API:

1. Go with your proposal and add prepare() and resolve() methods to each model
2. Add methods to the Vuex ORM instance, e.g.

   VuexORM.startTransaction() 

   and

   VuexORM.commitTransaction(). 

   This way you don’t have to think about the individuals models which are affected during the transaction

About the storage:

1. Create a clone of the store (or of all the entites used withhin vuex-orm) once startTransaction() is called and doing all the changes to the clone. On calling commitTransaction() the store is replaced with the cloned store. Because the cloned Data is not in use on the page itself, it's reactivity (hopefully) doesn't matter, because nothing gets rerendered on update. Or if it matters, it gets unreactive with Object.freeze();
2. Create a log of all the changes and replaying them when committing to the store. But doing it like that, the question is, wether the changes can be committed from Vuex ORM to Vuex at once without triggering a reactivity update each time or if it won't change anything and the problem remains.
3. Create a log of all the changes and merge them before committing to the store. With this option, the hard part is propably the merging, especially if inserts and deletes are involved

Me personally, I'd prefer the second API option with global transactions and the first storage option with a cloned store. What do you think about it? Does it make sense for you? Or do you have other ideas? Thank you!

[kiaking]

Thanks for the insight!

About the API

I like your API. It's straight forward and easy to understand. However, not so sure if we should have Global State or not. There's an issue requesting to make Vuex ORM able to have multiple database instances. Vue 3 is also moving toward cutting globals.

I think it's really nice to have global methods like startTransaction. But I would like to think more deeply about where and how to add that functionality.

I think at first, we can start tackling this issue for Model apis. Then think about what we can do about globals.

About the Storage

I think (1) is the easiest to implement? But also costly to create that clone when you have a big state because we must perform a deep copy of the entire state. (2) and (3) could be more effective, but yeah, it's going to be tough to do merging right.

So I think we should go with (1), and see how much performance impact it has when copying the whole state.

[sinasita]

Hello and sorry for answering so late!

I understand the problem about the global state, maybe with VuexORM.startTransaction(), I didn't choose a good name for what to do. My thought was, when the method gets called, the whole vuex-orm entities get cloned (or a parameter is added to know, which entities need to be cloned from that point on), maybe that can be done with a RootAction/RootMutation?

  store.dispatch['entities/startTransaction'](payload, { root: true });

or maybe another naming:

  store.dispatch['entities/createClone'](payload, { root: true });

The entities with a clone get a flag, and as long as the flag exists, their clone is used for further transactions.

You might have an idea of how to check that and then, use the clone instead? Maybe the lifecycle-hook beforeUpdate is the correct starting point?

    Query.on('beforeUpdate', function (model) {
        if (_hasClone) { 
             ...do something to the clone;
             return false;
        }
    });

And on calling a RootAction called commitTransaction or mergeClones, all the flags get reset and the real entities get updated with the cloneState, maybe with a create and all the connected clones get deleted.

Do you see any obvious problems coming up? Do you think, the clones should be on the store module "entities"? Or should there be another one ("clonedEntites")? Would it be possible anyhow to create a second module?

[kiaking]

Thanks for your insight! Yeah... I too think VuexORM.startTransaction() is nice as API. The only thing what bothers me is the global state problem. Let me open up another issue for that and see how we can address it. Then, I might think it's easier for us to come back here and re-think about the API.

[Juice10]

Any progress made here? Would love to have transactions in Vuex-ORM