search

dexie / Dexie.js

A Minimalistic Wrapper for IndexedDB
https://dexie.org
Apache License 2.0
11.74k stars 642 forks source link

Dexiecloud upgrade() may have a race condition #1422

Open dusty-phillips opened 3 years ago

dusty-phillips commented 3 years ago

Just fubar'd my db kind of on purpose (I wanted to see what happened so I could report). Here's what I did

  1. Made a .version() of the db with a (unindexed) field that contained an object as content
  2. Made a new .version() of the db that changed that field's type to a string
  3. added a .upgrade() function on the new version that iterated all the objects and converted them to strings
  4. opened the site in a new browser that ran the .version() and .upgrade() on load
  5. waited for dexie-cloud to sync the objects over from the other browser
  6. boom The site is expecting strings because .upgrade() ran successfully before the old objects were synced into the db.

I don't know that this is a generically solvable problem in a cloud sync world. I think the only realistic solution is to have two fields on the model (one for the old object, one for the new string) and have client code dual read for the foreseeable future (if so, this pattern is worth documenting somewhere).

Couple other ideas:

dfahlander commented 3 years ago

Yes, migrations needs to be server side for dexie cloud. And they need to be per-object so they can run on demand when an old client syncs. I designed a solution for this some time ago and it will go into dexie cloud at some point. For now, dexie cloud does not handle migrations - and it's a limitation in need of documenting.

dusty-phillips commented 9 months ago

Hi @dfahlander, wanted to circle back on this as you are approaching release… and so am I. I have a rather tricky db migration I need to queue up and wondered if you have more concrete thoughts or plans yet.

dfahlander commented 9 months ago

Hi! Yes, everything is on place: API:s, management app and the service. I'm just trying to get dexie 4.0 into stable since Dexie Cloud depends on it and we're working on the public message, website updates and newsletter.

Regarding upgrade - it needs to be done server side. The best way right now is to use the CLI to export your database to a JSON file, create a node script that migrates it and then import it back to a new database where you can test that the migrated data looks good.

Export your entire database

npx dexie-cloud export exportfile.json

Migrate the JSON file

node yourMigration.js < exportfile.json > upgradedDB.json

Import it back to a new DB

npx dexie-cloud create
npx dexie-cloud import exportfile.json

Let me know if this would work in your case

dusty-phillips commented 9 months ago

Should work in this case since my users aren’t very active, but if they wrote to the db while the migration was happening I would have a problem, right? I guess that can be mitigated with a dual read strategy for some period of time.

I previously had some client side infrastructure that effectively did a “migrate on read” strategy but it was too slow because it never knew if a migration was necessary so it had to check each time. So I can imagine a solution where you store the db version on all server stored objects, and when you read an object from the server you check the version and perform the versioned migration client side of it needs updating.

dusty-phillips commented 9 months ago

Thought about this some more and realized that because I am using a pwa, I have no guarantee that the client has updated to the latest client code, so I need to guarantee backwards compatibility even for new objects sent to the server after I migrate.