search

fairDataSociety / blossom

Browser Extension based on Fair Data Protocol
Apache License 2.0
5 stars 4 forks source link

fix: serialization #130 #131

Closed tomicvladan closed 1 year ago

tomicvladan commented 1 year ago

Fixes the serialization issue.

tomicvladan commented 1 year ago

Using blob, bytes can be converted to base64. But in any case, large files can't be transferred. If a file is too large, it will cause this error:

Error in invocation of runtime.sendMessage(optional string extensionId, any message, optional object options, optional function callback): Message length exceeded maximum allowed length.
IgorShadurin commented 1 year ago

@tomicvladan I can suggest other approaches. I don't know if they will work in this case. 1 - IndexedDB - can be used as an intermediate storage location. It seems to support big data. 2 - Message chunking - file can be read the in parts and transfer pieces of 1-5 mb to the extension, collecting a whole file in memory. Then upload the assembled file via fdp-storage.

IgorShadurin commented 1 year ago

@tomicvladan I can suggest other approaches. I don't know if they will work in this case. 1 - IndexedDB - can be used as an intermediate storage location. It seems to support big data. 2 - Message chunking - file can be read the in parts and transfer pieces of 1-5 mb to the extension, collecting a whole file in memory. Then upload the assembled file via fdp-storage.

3 - Store data inside SharedWorker and read them peace by peace https://developer.mozilla.org/en-US/docs/Web/API/SharedWorker

tomicvladan commented 1 year ago

Thanks @IgorShadurin, these approaches would work for web pages, but they can't be used for extension.

  1. IndexedDB is not available in service worker script, where the file is actually uploaded.
  2. Message chunking - because of transient nature of service worker, we can't rely on any buffer. The process can be terminated at any point, and the buffer would be lost.
  3. SharedWorker - file must be uploaded from a process that contains user's private key. We can't share private key in any place where other applications can access. Additionally the SharedWorker is not available in service worker script, so this approach can't be used as well