[jsinterop] How to serialize / deserialize JavaScript object in IndexedDB using Dart interop

[poirierlouis]

Environment

Dart SDK version: 2.18.5 (stable) (Tue Nov 22 15:47:29 2022 +0000) on "windows_x64" OS: Windows 10 Browser: Chrome

Web environment only, regarding DDC and dart2js.

Problem

I fail to serialize a JavaScript object into IndexedDB when using js-interop. Object is a FileSystemDirectoryHandle from the experimental File System Access API. Object is defined as [Serializable] in web IDL and storing it works fine with plain JavaScript (see example).

I define js-interop classes in Dart to implement the File System Access API such as:

// Only to keep track of API's types definitions.
typedef Promise<T> = dynamic;

// FileSystemHandle and *Options are declared in the same way.
@JS()
class FileSystemDirectoryHandle extends FileSystemHandle {
  external Promise<FileSystemFileHandle> getFileHandle(String name, [FileSystemGetFileOptions? options]);
  external Promise<FileSystemDirectoryHandle> getDirectoryHandle(String name, [FileSystemGetDirectoryOptions? options]);
  external Promise<void> removeEntry(String name, [FileSystemRemoveOptions? options]);
  external Promise<List<String>?> resolve(FileSystemHandle possibleDescendant);
}

What I'm trying to achieve

Here is the use-case and minimal code example below:

1. Ask user to pick a directory.
2. Returned directory's handle can be cast-ed to my FileSystemDirectoryHandle, thank you js-interop.
3. Store handle in IndexedDB.
4. Read handle back from IndexedDB.
5. Use directory's handle.

What I'm doing

// (1)
final handle = await js.promiseToFuture(window.showDirectoryPicker());

// (2)
final directory = handle as FileSystemDirectoryHandle;

print("name: ${directory.name}");

// (3)
// storage use dart:indexed_db in a homebrew implementation (tested and works fine with 
// primitive types).
await storage.set("dir", handle);

Reload page:

// (4) Dynamic only
final directory = await storage.get("dir");

print(directory.name);

// (5) Typed
FileSystemDirectoryHandle dirHandle = directory as FileSystemDirectoryHandle;

print(dirHandle.name);

What I'm expecting

I should be able to manipulate the FileSystemDirectoryHandle (4) & (5) like the object I get at point (1). Therefore printing the name of handle should work.

What happens

It throws an error (4, dynamic):

Uncaught (in promise) Error: NoSuchMethodError: 'name' method not found Receiver: Instance of 'LinkedMap<dynamic, dynamic>'

and (5, typed):

Error: Expected a value of type 'FileSystemDirectoryHandle', but got one of type 'LinkedMap<dynamic, dynamic>'

Here a screenshot of the IndexedDB in DevTools, to compare value between plain JavaScript and Dart js-interop.

From the look of it, I suspect it sends a Dart object to IndexedDB which serialize using Dart mechanism and might explains the Symbol declarations in the screenshot.

Anyway, it makes sense to me that I cannot obtain a FileSystemDirectoryHandle back. All information is lost during serialization. I'm looking for a way to send the native JavaScript object, such as browser's [Serializable] feature can trigger on the FileSystemDirectoryHandle.

As I have been searching for this without success, the only issue I found which might be related somehow is #48188.

Any guidance would be much appreciated.

More code

See more in this repository, usage in example here and js-interop here.

[srujzs]

I see, the return type makes sense, since dart:indexed_db serializes the handle that you're placing in storage (I believe ObjectStore.put is the culprit). I'm not entirely sure why we serialize there as I'm not familiar with the JS APIs, and it's entirely possible this may not work to directly use the handle. I also don't think it's possible to unserialize back to the original handle with our conversion APIs. However, a workaround to use the handle instead could be to use interop to avoid the dart:indexed_db serialization. Perhaps try this (disclaimer: I haven't tried it myself):

@JS()
@staticInterop
class MyObjectStore {}

extension on MyObjectStore {
  external Request put(value, key);
}

...
// In `LightStorage.set`:
final store = twx.objectStore(storeName) as MyObjectStore; // Use your own interop type here.
// Calls `MyObjectStore.put` since that's the static type.
final request = store.put(value, key);
// Copy `_completeRequest` from `dart:indexed_db`.
Future<T> _completeRequest<T>(Request request) {
  var completer = new Completer<T>.sync();
  request.onSuccess.listen((e) {
    T result = request.result;
    completer.complete(result);
  });
  request.onError.listen(completer.completeError);
  return completer.future;
}
_completeRequest(request);

It's a bit hacky, but it might resolve your use case.

[poirierlouis]

I've tried your workaround and it works fine, thanks !

I just needed to do the same to get the value back:

extension on MyObjectStore {
  external Request get(key);
  external Request put(value, key);
}

With this solution, I can put / get a FileSystemDirectoryHandle without errors. Here's how it looks in DevTools (mix of JavaScript object and Dart Symbols):

I guess however it is not a definitive solution. Ultimately, Dart Symbols should not be stored / serialized along with native JavaScript object.

While searching in this repository, I saw usage of @Native and keyword native along with Interceptors. Out of curiosity, declaring with Native internally provides some kind of performance improvements over js-interop? Why is the Native approach only internal / private to Dart SDK and not exposed to developers, like js-interop is?

While my use-case is solved with your workaround @srujzs, shall I close this issue or it shall continue to live on to discuss / fix issue in dart:indexed_db ?

[srujzs]

Glad it worked!

While searching in this repository, I saw usage of @Native and keyword native along with Interceptors. Out of curiosity, declaring with Native internally provides some kind of performance improvements over js-interop? Why is the Native approach only internal / private to Dart SDK and not exposed to developers, like js-interop is?

@Native is a special annotation. It binds the type (the string within the annotation) to the class it defines. This means whenever you get a JS object, it gets intercepted as its complementary @Native type (hence the word "interceptor") if it exists. This is a really powerful feature that we don't want to expose. For an example of where this goes wrong, imagine if two users in different libraries defined a @Native type for the same underlying JS type. It also gets complex when we start talking about user-defined JS types and not just the DOM types. In terms of performance, it's probably faster to use JS interop since we don't need to do any checking to inspect the underlying type, and we don't need an interceptor if you're using interop that's purely static (like @staticInterop).

As for this API, we should still serialize Dart objects, but for some JS objects (https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm#supported_types), we don't need to since they're already serializable. The complexity around this though is being able to determine that. I suppose we could keep around a list of types and check at runtime before trying to serialize? Not sure.

We're working on introducing new interop features and a newer set of web libraries that are more direct, so it's likely we won't end up fixing this in favor of that. That being said, I'll leave this open so we have something to come back to when we're thinking about serializing already serializable JS objects.

[poirierlouis]

Thank you for your explanation.

As for this API, we should still serialize Dart objects, but for some JS objects (https://developer.mozilla.org/en-US/docs/Web/API/Web_Workers_API/Structured_clone_algorithm#supported_types), we don't need to since they're already serializable. The complexity around this though is being able to determine that. I suppose we could keep around a list of types and check at runtime before trying to serialize? Not sure.

In IndexedDB package, ObjectStore.put use convertDartToNative_SerializedScriptValue from html_common:conversions. There is indeed a forest of if to test for Dart types and return them as-this. It also serializes to JsMap / JsObject when appropriate. It ends up throwing when type of object is unknown which is naturally the case with the FileSystemHandle interface.

Could a list of types be aggregated directly from Web IDL using the list of supported types as mentioned in your link?

Or could the algorithm not throw and let object pass through after all known Dart types have been tested?

I guess it might lead to errors in some cases, but it might be ideal as new [Serializable] interfaces gets defined on W3C. I don't know if there is a way to generally test an object marked [Serializable] at runtime? (I guess not)

We're working on introducing new interop features and a newer set of web libraries that are more direct, so it's likely we won't end up fixing this in favor of that. That being said, I'll leave this open so we have something to come back to when we're thinking about serializing already serializable JS objects.

Great! If I were to contribute, say a native file_system_access package for example, is there resources / documentations I could look into to guide me through Dart / JS / js-interop code structure? Or it would basically be to learn from existing code?