Way to telling pinia orm that the underlying data of a repo has changed.

[tintin10q]

Describe the feature

It would be great if there was a way to force pinia-orm to reload the data from storage.

In #1212 I talked about using indexedDB and saving data in a web worker. I have now implemented this myself by just saving data into the indexeddb object store where the pinia orm data will end up eventually in the correct format.

However, pinia orm only knows about this data in the storage after a full page refresh.

Is there a way I can tell pinia orm that the underlying data has changed and that it has to just read the data from the storage again and then update in vue? Does this already exist?

If not, I suggest perhaps reload() function that takes a model class as input. So like reload(Posts) would load the data of the Posts model again from the storage and tell vue that things have changed.

Something like this reload function would allow to make very performant code.

Additional information

• [X] Would you be willing to help implement this feature?

Final checks

• [X] Check existing discussions and issues.

[karsvaniersel]

I love this idea, I have actually been looking for something similar now that Laravel Reverb is released. I've created a rough MVP of a sync engine (like Linear has), where everything is stored in IndexedDB and is being updated by receiving data from web sockets.

Besides, if you are not even using indexedDB, it would be amazing to have a plugin, similar to the axios one, where the data is coming in from a web socket to reload the data.

This could be extended to "sources", where a source can be:

• Axios / regular API calls
• Web Sockets
• IndexedDB

Ideally, for IndexedDB you would subscribe to the changes of the IndexedDB stores.

For web sockets the event name could match a pattern of Model:Hook, so for example a web socket pushes an object that contains: { event: "Post:Created", data: {...} }

This would in turn result in useRepo(Post).save(data)

Anyway, just some of my ideas, seems like pinia-orm has a lot of potential, especially using local storage (IndexedDB or event SQLite). However, is this repo even maintained? Cause the last updates have been longer over a year ago..

[tintin10q]

@karsvaniersel I am glad to read that your also exited about this idea. Hopefully we can bring @CodeDredd back to add something like the reload function. However, what does not help in this is pursuit is complain that the repo has not had an update for over a year ago.... Who knows what he has had going on.

[vesper8]

Last major update was 1.7.0 on September 17th so not quite that long... however I am also very concerned by the absence of development following that release.. especially considering how critical this piece of software is to any app that decides to use it. I too also love the idea presented in this issue and hope that maybe someday it can be added as a plugin.

[CodeDredd]

@tintin10q @vesper8 ... 1.8.0 is in progress. Working the issues from the bottom. So to get this started...i don't know if this more a pinia plugin then a pinia-orm stuff. Try to get some insights and reporting back...

[CodeDredd]

@tintin10q There is the revive function. Have you tried that out? It is what you have described with reload

[tintin10q]

Sounds promising. Is there documentation about this function?

[CodeDredd]

No there is no documentation about it 😞 .... But it exactly does what you are describing. Anyways i am now going to implement a worker feature. About indexdb...this looks complicated. I am yet not sure how to think about it. But maybe you can give my some hints if there are some stones on the way. It would be good to have a an example to work on to see where the problems are and if they are solved by revive and worker.

Here is an example for revive. i took it from the unit test 😉

    const userRepo = useRepo(User)

    fillState({
      users: {
        1: { id: 1, name: 'John Doe' },
        2: { id: 2, name: 'Jane Doe' },
        3: { id: 3, name: 'Johnny Doe' },
      },
    })

    const schema = [{ id: 3 }, { id: 1 }]

    const users = userRepo.revive(schema)

    expect(users.length).toBe(2)
    expect(users[0].id).toBe(3)
    expect(users[1].id).toBe(1)

[tintin10q]

Thanks for giving the unit test. That is very helpful.

Just to clarify, I use a persistent store pinia plugin. So that is a pinia plugin, not a pinia orm plugin. So when I mean reload from the store, I mean load the data again from the persistent storage. Does revive do that?

I don't know how aware pinia orm is of the persistent storage.

But given that you make it work with indexdb it seems that pinia orm is aware of the storage. You could also leave the storage to a plugin, but this could also be a first-party plugin, of course. But if you plan to implement saving in workers, you might want to go for indexed db anyways because I think that's the only storage that is available in workers.

About indexdb...this looks complicated.

Yep, you're right it is complicated 😄

I am yet not sure how to think about it. But maybe you can give me some hints if there are some stones on the way.

I have worked with index db quite a bit now and from my experience I can say that one major advantage of indexed dB is that it can store any javascript thing directly (including maps and sets) instead of only json strings. This way, you don't have to serialize and deserilize things and that can save time. It would also, in theory, allow any type of javascript value to be put in a model.

In my opinion the best way to use indexed db is to just make datastores that have a single object in them. This object would then be a javascript map that goes from the primary keys to the objects. I think this is the fastest and easiest way to work with it. With that setup, you only need to load the map, and then you can do what you want. This setup allows to avoid all these fancy schema things with primary keys because loading data using the indexdb db key fields and stuff is actually quite slow in my experience because every iteration of the cursor makes a new promise So, I think those types of datastore only makes sense if your data doesn't fit into memory. But pinia orm needs to kinda fit into memory anyways.

So my advice for indexed db is to make a single datastore for each repo that stores a javascript map from the primary key to the data.

It would be good to have an example to work on to see where the problems are and if they are solved by revive and worker.

I can give you an example when I am back from my holiday next week.

Here is an example for revive. i take it from the unit test 😉

I have some questions about this.

Does this actually reload the data from storage? As I am on holiday I can only test this next week.

Also, it seems revive needs a list of id. So, in theory, I could call this with the list of ids that were saved into the storage?

That would be useful because then I can store in a worker using indexed db using the same format it would be and then revive the changed records.

I hope this is useful.

[CodeDredd]

Does this actually reload the data from storage? As I am on holiday I can only test this next week.

Yes it does reload the data from the storage.

Also, it seems revive needs a list of id. So, in theory, I could call this with the list of ids that were saved into the storage?

Correct. You just need the ids to reload the data completly.

[karsvaniersel]

To add to the Index db debate, I have been using Pinia ORM quite a lot at the moment with a Laravel backend (API, front end is a standalone Nuxt app).

Before using Pinia ORM I was using Index DB, but I was basically creating my own entire layer of an ORM (very similar to Pinia ORM). Hence my switch. In the near future, I still want to add a sync engine to keep the index DB and the backend in sync (using web sockets).

Personally I feel like it would not be super complicated to hook Pinia ORM into Index DB. What I was thinking was the following, once the data comes in and Pinia ORM serialises this data it would add a record in an index DB, where the table name is equal to the entity. Pinia ORM would still create the regular Vue stores / useRepo(), but underneath it all it stores the data in Index DB.

Then on the model hooks, created / updated / deleted etc, I would literally just store it in the Index DB as well. So in a way it kinda seems redudant, storing data both in Pinia ORM and in an underlying Index DB store, but the upside is that this data can now be stored in a persistant way, and you wouldn't even need to hit up your server anymore to get this data.

Of course not every type of web app would like this behaviour, like apps that rely heavily on permissions etc. But if you take a look at Linear or any other simple "To do / Task" app, you do not really care about a lot of items. What you do care about is not having to retrieve a list of "Labels" on every page request.

I made the switch from Inertia + Vue + Laravel to just Vue + Laravel as API and I have almost 0 request to my server, only the request (which gets all the data and fills up Pinia ORM) and when a user creates a new item. Using index DB, I could reduce the load time for the user from 20/40ms to zero, since the app will be local (index DB) first, ironically enough this also comes with the added benefit of offline first.

If you are interested more in Index DB check out https://www.npmjs.com/package/idb, it is a very lightweight package but it does everything you want it to do.

Currently I'm pretty swamped with work but soon when I have some downtime I want to see if I can make a Pinia ORM plugin that functions as a bridge between Laravel + Vue. Would be pretty sweet if I could just make it in a way that Laravel sends some code that defines my Pinia ORM models to keep everything in sync in combination with Index DB :D

[CodeDredd]

@karsvaniersel Here is some insperation for you https://github.com/dev1437/pinia-orm-generator

[CodeDredd]

I am closing this issue now. Please feel free to move it to a discussion if there is still something open to discuess or to make a feature request.

[bgervan]

@tintin10q do you mind sharing the code for the persistent storage with pinia-orm?