How to persist store to indexedDB (idb)

[geoidesic]

Hi,

I'm trying to persist the stores to indexedDB. There are some libraries for this but they aren't compatible with jsonapi-vuex. There are two problems I'm having with writing my own code for this purpose:

1. I can't simply take the store object and save it to indexedDB because it chokes on certain nested objects like relationships.
2. So instead I'm making a shallow clone of the object using lodash's _.clone and storing that. However when I then try read items from the indexedDB back into the store I can't seem to get them into a format that jsonapi-vuex's mutators will accept.
3. The other thing that wasn't clear is for a store which contains different collections, how to use a mutator to fill a single collection from that store?

Here's what the data looks like in the jsonapi-vuex store:

{
  "_jv": {
    "id": "3",
    "isAttr": {
      "_custom": {
        "display": "<span>ƒ</span> isAttr(name)",
        "type": "function"
      }
    },
    "isRel": {
      "_custom": {
        "display": "<span>ƒ</span> isRel(name)",
        "type": "function"
      }
    },
    "links": {
      "self": "/api/users/3"
    },
    "relationships": {
      "customer": {
        "data": {
          "id": "1",
          "type": "users"
        },
        "links": {
          "self": "/api/users/1"
        }
      },
      "entity-type": {
        "data": {
          "id": "1",
          "type": "entity-types"
        },
        "links": {
          "self": "/api/entity_types/1"
        }
      },
      "role": {
        "data": {
          "id": "3",
          "type": "roles"
        },
        "links": {
          "self": "/api/roles/3"
        }
      }
    },
    "type": "users"
  },
  "active": true,
  "address1": null,
  "address2": null,
  "created": "2019-11-20T15:11:37+00:00",
  "email": null,
  "modified": "2019-11-20T15:11:37+00:00",
  "name": "Geoid",
  "phone": null,
  "position": null,
  "postcode": null,
  "surname": "Esic"
}

Here's the code I'm using to write from the store to the indexedDB (note I'm using the idb library, which is a vuejs plugin for indexedDB):

// `result` is the result of a jsonapi-vuex dispatch message
Object.keys(result).forEach(async key => {
            let clone = _.clone(result[key]);
            delete clone._jv;
            await db.instance.put(dbStorageName, {
              id: key,
              ...clone
            });
          });

What I end up with is an indexedDB record that looks like this:

{
  "active": true,
  "address1": null,
  "address2": null,
  "created": "2019-11-20T15:11:37+00:00",
  "customer": {},
  "email": null,
  "entity-type": {},
  "id": "3",
  "modified": "2019-11-20T15:11:37+00:00",
  "name": "Geoid",
  "phone": null,
  "position": null,
  "postcode": null,
  "role": {},
  "surname": "Esic"
}

When I try store that back into the jsonapi-vuex store, then I get the errors, using this code:

if (!empty(records)) {
          store.commit(
            `${dbStorageName}/addRecords`,
            records
          );
        }

The error is:

TypeError: Cannot destructure property 'type' of 'item[jvtag]' as it is undefined.

Maybe there's a better way? I've seen you have some utils for serialising / deserialising but the usage instructions are a bit terse: i.e. the descriptions of the functions don't really tell one anything that you wouldn't already grock from the name of the function – it would be helpful to have some examples of how and when to use these functions. So I haven't figured out how to make use of them in this context (if they are indeed useful in this context).

I tried wrapping my records with various combinations of your helper functions without success.

[mrichar1]

Hi - thanks for reporting these problems.

To look at the issues:

1. "chokes on certain nested objects like relationships."

The objects in the store should have no recursive functions in them. Those in _jv_ (isAttr and isRel) only return True or False. The relationships section in _jv is really a mirror of the original JSONAPI section, which has no recursive code in it.

Can you send me the actual errors from the code when it fails due to recursion? I'd be interested in knowing which specific bits of the store it is working on when it fails.

2. "making a shallow clone ... I can't seem to get them into a format that jsonapi-vuex's mutators will accept."

You are deleting _jv when cloning. This contains the type and id metadata of the object, which is how it is all keyed in the code - hence the errors about item[jvtag]. You can delete some of the child elements of this object, but the object, type and id must always be present as a minimum.

3. "for a store which contains different collections, how to use a mutator to fill a single collection from that store?"

I'm not 100% sure what you mean by this. Do you mean "how do I add multiple items to a single collection in the store?" or "How do I POST/PATCH a whole collection to the API?" or something else?

[geoidesic]

1. Ok, so if I do let result = await store.dispatch('clients/get') and then try to store that result directly to indexedDB like so: await db.instance.put('clients/get', result) then I get an error like this:

   Uncaught (in promise) DOMException: Failed to execute 'put' on 'IDBObjectStore': isRel(name) {
     return hasProperty(lodash_get__WEBPACK_IMPORTED_MODULE_1___default()(obj, [jvtag, 'relation...<omitted>... } could not be cloned.

2. I tried a different approach now. I decided I would convert the result back to JSONAPI format and then store that as a string to indexedDB and then possibly reverse that process when trying to populate the Vuex store from indexedDB, but I get other problems.

   Object.keys(result).forEach(async key => {
   let clone = utils.deepCopy(result[key]);
   clone._jv = JSON.stringify(result[key]._jv);
   await db.instance.put(message.endpoint, {
   id: key,
   jsonapi: JSON.stringify(utils.normToJsonapi(clone))
   });
   });

   I take the result from the jv/get, process each row, so as to provide an id key (which indexedDB needs for each record) and then a stringified JSONAPI object for each record is stored. (Where records is as per the screenshot below – a set of records representing one of the collections from the Vuex store as it has been saved to the indexedDB store in JSONAPI format and now retrieved from there):

Then when I try to reverse the process I did this:

let obj = {};
records.forEach(record => {
  return (obj[record.id] = utils.jsonapiToNorm(record.jsonapi));
});
store.commit(`${message.vuexstore}/addRecords`, obj);

This doesn't cause any errors but the Vuex store is a mess (see screenshot):

3. So the store I have, in its correct form, looks like this screenshot:

The store is clients and it contains two collections, users and organisations. These collections come from different endpoints, i.e. 2 different Ajax calls. When I store these to the indexedDB database they are each a distinct data store. My question was to do with not understanding how to use the addRecords mutator to process these separate datastores so as to achieve the desired Vuex store result. I'm confused by the fact that the store can have these multiple collections, each with multiple records, but looking at the code for addRecords it's not clear to me how to structure my payload for the mutators

[geoidesic]

I think part of the problem is that the helper functions normToJsonapi and jsonapiToNorm do not seem to be complimentary. I.e. if I convert my get result to a JSONAPI object through the use of normToJsonapi (which seems to work as expected) and then run jsonapiToNorm on that, I end up with gobbledigook, instead of what I would expect which is that it should reverse the process.

Perhaps I'm not understanding the helpers correctly.

[geoidesic]

Ok. I managed to get it working. These were the issues:

1. I wasn't parsing my stored JSON string on the way out.
2. It seems that jsonapiToNorm wants to operate on the contents of the JSONAPI data.

Once I did that it's all hanging together :)

Final code for idb:

Object.keys(result).forEach(async key => {
  let clone = utils.deepCopy(result[key]);
  await db.instance.put(endpoint, {
    id: key,
    jsonapi: JSON.stringify(utils.normToJsonapi(clone))
  });
});

Final code for converting that back into a jsonapi-vuex store:

let obj = {};
records.forEach(record => {
  obj[record.id] = utils.jsonapiToNorm(
    JSON.parse(record.jsonapi).data
  );
});
store.commit(`${vuexstore}/addRecords`, obj);

[mrichar1]

Great - you've beaten me to an answer!

Yes, the conversion functions aren't quite complimentary, since one takes data, while the other produces the full document. This should at least be documented, and I'll consider overhauling it if it's easy to make them properly complimentary without affecting the codebase too much.

Saving the store as JSONAPI is a good idea, since then it's guaranteed to be 'safe' JSON. In theory you might not even need to do JSON.stringify on it, since it should be a simple object with no functions etc, but I'm not sure what idb expects.

Glad you got it all sorted! :smiley:

[geoidesic]

You're right, I was able to simplify it further to:

Object.keys(result).forEach(async key => {
  let clone = utils.deepCopy(result[key]);
  await db.instance.put(endpoint, {
    ...utils.normToJsonapi(clone).data
  });
});

and

let obj = {};
records.forEach(record => {
  obj[record.id] = utils.jsonapiToNorm(record);
});
store.commit(`${vuexstore}/addRecords`, obj);