Looking into using Dexie Cloud for my new app.

[nlaurie]

Couple of questions

My primary keys are uuid and named uuid not id , is that ok ?

I'm using dexie-encrypted for sensitive data, will that be maintained when syncing to the cloud. Need to ensure the users private info is not recoverable if the cloud source gets compromised.

Ultimately I will most likely run my own hosted version, but for the beta , I would like to use your cloud.

[dfahlander]

My primary keys are uuid and named uuid not id , is that ok ?

Yes, you can specify any property name and you can either use your own ID generation or let Dexie Cloud auto-generate the keys.

To manage the keys using your own format, such as GUIDs or similar, declare the primary key as so:

db.version(N).stores({yourTable: 'uuid, index1, index2, ...'})

To let Dexie Cloud generate the keys for you, do:

db.version(N).stores({yourTable: '@uuid, index1, index2, ...'})

The latter will generate globally unique keys for you with timestamp and table baked into the ID. These auto-generated keys have some benefits over random GUIDS, in case any of those feature would be important:

1. They sort in the same order they are generated - the format is a lexically sortable version of base64 with timestamp and counter baked into it, making them behave almost as auto-generated numbers. If you create three objects, they will always be naturally sorted in the same order as they were created.
2. They have the table name baked into the key making it possible to determine which table they belong to.
3. Their string representation is more compressed than GUIDs as it uses a 64 base instead of 16 and also doesn't contain dashes.

I'm using dexie-encrypted for sensitive data, will that be maintained when syncing to the cloud. Need to ensure the users private info is not recoverable if the cloud source gets compromised.

I'm currently investigation interoperability between dexie-encrypted and dexie-cloud but it seems that dexie-encrypted puts its encryption layer on level 0, which is below dexie-cloud's mutation-tracking level 1. I think it is crucial that dexie-cloud works with encryption addons so this need to be patched soon. I will leave a comment on this issue once it has been patched and verified.

Ultimately I will most likely run my own hosted version, but for the beta , I would like to use your cloud.

Great, you may use https://dexie.cloud already by running npx dexie-cloud create from a command prompt, to create a database and then connect to it in your app. If you're not in the private beta yet, you will be prompted for applying in the CLI. To see examples of how to connect dexie-cloud in a web application, you could look at the sample app at https://github.com/dexie/Dexie.js/tree/master/samples/dexie-cloud-todo-app. I wish you the best with your app and don't hesitate to reach out if you have more questions!

[nlaurie]

Thank you for your answers, and I'm already part of the beta, one last question .

I have a fairly complex crypto application utilizing dexie.js for all data storage. it seems like the dexie-cloud-addon is requiring 4.0.0-alpha.3, but dexie-encrypted is asking for ^3.0.0.

What versions are required so I can trial dexie-cloud with dexie-encrypted. as encryption is absolutely mandatory in my case.

Woops, just re-read your response to dexie-encrypted, seems like I'm out of luck until you patch.

[dfahlander]

Yes. I will look at it asap next week!

[dfahlander]

Yes. I will look at it asap next week!

Unfortunately I didn't find a time spot last week, sorry for the delay. My plan is to fix it this week.

[dfahlander]

Hi! At last, I've POCed this and found a solution for dexie-cloud with dexie-encrypted and they finally works well together, given that:

1. Make sure you use dexie@^4.0.0-alpha.4 even if dexie-encrypted expect dexie@3.x. The only incompability is the typings so you need a // @ts-ignore line before the call to applyEncryptionMiddleware().
2. You list dexie-encrypteds internal table "_encryptionSettings" in dexie-cloud's unsyncedTables [configuration](https://dexie.org/cloud/docs/db.cloud.configure()) option.
3. If you don't just want local encryption but encryption also in the cloud servers (which I assume was your requirement) you need to change the level at which the encryption occur, so it occurs before sync. See snippet below on how to do it.
4. Update dexie-cloud-addon to version ^4.0.0-beta.22. It contains a bugfix in syncing ArrayBuffer (the encrypted data).

Making dexie-encrypted work with dexie-cloud

db.cloud.configure({
  ...,
  unsyncedTables: ["_encryptionSettings"]
});

Applying encryption layer above sync layer:

function reorderDexieEncrypted (db: Dexie) {
  // @ts-ignore
  const mw = db._middlewares.dbcore.find(mw => mw.name === 'encryption');
  if (!mw) throw new Error("Dexie encrypted not applied");
  db.use({
    name: "encryption",
    stack: "dbcore",
    level: 2,
    create: mw.create
  });
}

Call this function after having called dexie-encrypted's applyEncryptionMiddleware(). This function forces dexie-encrypted to be invoked above the sync layer so that encrypted fields keeps being encrypted in dexie-cloud servers.

The full source of the POC can be found here. You may clone it and run yarn install, yarn start. It will connect to a live database. To verify encryption and sync, make sure to hit the login button and login with your email address + OTP. Do that in two different browsers (such as one chrome and one firefox for example) and verify data is encrypted also when syncing (by looking at the network tab in devtools).

[nlaurie]

David

Awesome , I will try it out this week. Excited to see if DexieSync will work for our application

Nick

On Mon, Sep 5, 2022 at 9:49 AM David Fahlander @.***> wrote:

Hi! At last, I've POCed this and found a solution for dexie-cloud with dexie-encrypted and they finally works well together, given that:

1. You list dexie-encrypteds internal table "_encryptionSettings" in dexie-cloud's unsyncedTables configuration https://dexie.org/cloud/docs/db.cloud.configure() option.
2. If you don't just want local encryption but encryption also in the cloud servers (which I assume was your requirement) you need to change the level at which the encryption occur, so it occurs before sync.
3. Update dexie-cloud-addon to version ^4.0.0-beta.22. It contains a bugfix in syncing ArrayBuffer (the encrypted data).

Making dexie-encrypted work with dexie-cloud

db.cloud.configure({ databaseUrl: "...", unsyncedTables: ["_encryptionSettings"]});

Applying encryption layer above sync layer:

function reorderDexieEncrypted (db: Dexie) { // @ts-ignore const mw = db._middlewares.dbcore.find(mw => mw.name === 'encryption'); if (!mw) throw new Error("Dexie encrypted not applied"); db.use({ name: "encryption", stack: "dbcore", level: 2, create: mw.create });}

Call this function after having called dexie-encrypted's applyEncryptionMiddleware(). This function forces dexie-encrypted to be invoked above the sync layer so that encrypted fields keeps being encrypted in dexie-cloud servers.

The full source of the POC can be found here https://github.com/dfahlander/poc-dexie-cloud-encryption. You may clone it and run yarn install, yarn start. It will connect to a live database. To verify encryption and sync, make sure to hit the login button and login with your email address + OTP. Do that in two different browsers (such as one chrome and one firefox for example) and verify data is encrypted also when syncing (by looking at the network tab in devtools).

— Reply to this email directly, view it on GitHub https://github.com/dexie/Dexie.js/issues/1604#issuecomment-1237065115, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOW63YESWP4Q4QM3TT2MN3V4X3AFANCNFSM56L6GYWQ . You are receiving this because you authored the thread.Message ID: @.***>

[nlaurie]

David

I started to implement the changes into my application and had one question. Does Dexie Cloud require the alpha 4. of Dexie or can I use it with 3.2.2. I held up when I got npm warnings. It dosent seem like 4.0 is a trivial upgrade.

Regards Nick

On Sat, Sep 10, 2022 at 8:00 AM Nick Laurie @.***> wrote:

David

Awesome , I will try it out this week. Excited to see if DexieSync will work for our application

Nick

On Mon, Sep 5, 2022 at 9:49 AM David Fahlander @.***> wrote:

Hi! At last, I've POCed this and found a solution for dexie-cloud with dexie-encrypted and they finally works well together, given that:

1. You list dexie-encrypteds internal table "_encryptionSettings" in dexie-cloud's unsyncedTables configuration https://dexie.org/cloud/docs/db.cloud.configure() option.
2. If you don't just want local encryption but encryption also in the cloud servers (which I assume was your requirement) you need to change the level at which the encryption occur, so it occurs before sync.
3. Update dexie-cloud-addon to version ^4.0.0-beta.22. It contains a bugfix in syncing ArrayBuffer (the encrypted data).

Making dexie-encrypted work with dexie-cloud

db.cloud.configure({ databaseUrl: "...", unsyncedTables: ["_encryptionSettings"]});

Applying encryption layer above sync layer:

function reorderDexieEncrypted (db: Dexie) { // @ts-ignore const mw = db._middlewares.dbcore.find(mw => mw.name === 'encryption'); if (!mw) throw new Error("Dexie encrypted not applied"); db.use({ name: "encryption", stack: "dbcore", level: 2, create: mw.create });}

Call this function after having called dexie-encrypted's applyEncryptionMiddleware(). This function forces dexie-encrypted to be invoked above the sync layer so that encrypted fields keeps being encrypted in dexie-cloud servers.

The full source of the POC can be found here https://github.com/dfahlander/poc-dexie-cloud-encryption. You may clone it and run yarn install, yarn start. It will connect to a live database. To verify encryption and sync, make sure to hit the login button and login with your email address + OTP. Do that in two different browsers (such as one chrome and one firefox for example) and verify data is encrypted also when syncing (by looking at the network tab in devtools).

— Reply to this email directly, view it on GitHub https://github.com/dexie/Dexie.js/issues/1604#issuecomment-1237065115, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOW63YESWP4Q4QM3TT2MN3V4X3AFANCNFSM56L6GYWQ . You are receiving this because you authored the thread.Message ID: @.***>

[dfahlander]

David I started to implement the changes into my application and had one question. Does Dexie Cloud require the alpha 4. of Dexie or can I use it with 3.2.2. I held up when I got npm warnings. It dosent seem like 4.0 is a trivial upgrade. Regards Nick

It really requires 4.0.0 but so far there are very few breaking changes so it should be trivial to upgrade. If using dexie-export-import or dexie-observable, their 'next' version are compatible: npm install dexie-export-import@next.

[nlaurie]

I thought so too, but in 4.0 I hit this typescript circular dependency issue with the new way Dexie types table objects. In the old version my setup worked beautifully. I have an object model (rather sophisticated) that I take care of the serialization from the POJO entity in a custom mapToClass mainly using class-transformer for nested objects. The objects are all responsible for generating their POJO for Dexie when needed to add or update. The result was amazing, I could query and get an Instantiated Class Object from the DB and everything was serialized to the contract specs. We spent all day yesterday trying to get 4.0 to accept our classes with no luck. ( mostly because there are a lot of properties that exist that shouldn't be serialized or be taken into account with Dexie.

Ideally it would be amazing if there was a mode that the user of the lib (me) could specify the Class and the POJO to the Table and hook into the system to take over serialization in this case. It takes the pressure off the core library from the assumption of serialization, since not all modes are perfect and generically traversing an unknown classes can be tricky at best

Couple Methods 1) Have the user Use a New Table Subclass that takes the extra type and exposes serialization hook , that takes over the automatic serialization (easiest for redirecting that automatic designtime typing) 2) Pulling from your 'Entity' Thread. Maybe Have the user implement an interface for serialization on their class ( may not help the typing issue unless you can pull the POJO type from the interface and detect the interface at runtime/devtime ) 3) Add an options param to the Table Table<Type, Key, options> that indicates the extra serialization info and ignoreproperties you introduced in 4

I have many hours invested in Dexie and would love to help make it better or at least handle this case where one wants to control the types and serialization. My app really depends on this and Dexie Cloud in order to launch at this point. Maybe with some guidance we can add this feature, I totally understand where you were going with the Entity , but once you start using realistic Classes the generic typing goes nutz with classes that do real work and have non serialized properties and lots of dependencies. Happy to show you a demo of our app to see our object model being used in Dexie.

I'll start a fork to see if my team can do an elegant POC Nick

On Tue, Oct 18, 2022 at 2:09 AM David Fahlander @.***> wrote:

It really requires 4.0.0 but it should be trivial to upgrade. Very few breaking changes.

— Reply to this email directly, view it on GitHub https://github.com/dexie/Dexie.js/issues/1604#issuecomment-1281858938, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACOW633W4MRM6ZD3H4NLSJDWDY5KJANCNFSM56L6GYWQ . You are receiving this because you authored the thread.Message ID: @.***>

[nlaurie]

I Updated the changes above with a PullRequest #1632. I think the solution is pretty solid and aids in keeping the runtime typing simple. I also upgraded dexie-cloud-addon with the changes.

That being said , I finally was able to run dexie-cloud. for the first time and ran into the next snag. It seems if I use Zero Config Auth , the database connects and starts syncing, but the case we are using is the "[auth integration]"(https://dexie.org/cloud/docs/db.cloud.configure()#example-integrate-custom-authentication)

When configuring this setup we have the endpoint working (Tested with Postman) returns Dexie Cloud Access tokens fine.

It's just the client is throwing an error when starting the DB.

Failed to execute 'put' on 'IDBObjectStore': async tokenParams => { loglevelWEBPACK_IMPORTED_MODULE1default().warn('Requesting Token', dexie...... } could not be cloned.

I have logging in the fetchTokens() callback and the method doesn't even get to that point of calling it.

Any answer is appreciated as we are trying to determine if cloud with encrypting will work for our platform. Thanks Nick

[dfahlander]

Thanks for your PR, I will dive into it as soon as possible. I have seen cloning bug also in Firefox only and it seems to happen sporadically when storing the private key, especially often when running in a service worker. In the normal DOM, this does not happen, or happens once but heals after a retry.

I've come to the conclusion that this is a bug in Firefox and for the reason, I disable using the service worker for Firefox.

[nlaurie]

I am running My App in Edge as a Service Worker. Let me know if its something I can debug.

[dfahlander]

Ok. The issue I got was due to storing a CryptoKey in indexedDB. For me it bugged out when running from a service worker on Firefox. The cryptoKey is stored in a property nonExportablePrivateKey in authenticate.ts that the caller later on puts into the $logins table (using await context.save(); in login.ts, a method that does db.table("$logins").put(this);.