Closed donl closed 1 year ago
Hi Don,
I'm trying to run your code snippet, but I get a TypeError: Cannot read properties of undefined (reading 'digest')
on the line const hashBuffer = await crypto.subtle.digest("SHA-256", arrayBuffer);
Also, I had to install and import node-fetch
to get the fetch
on the image to work.
Can you edit the code snippet to isolate the exact issue?
I'm running node v18.7.0 -- it warns that Fetch is experimental but runs and I believe the crypto module is rather new in node as well. I had to specify "type": "module"
in package.json
for the import syntax.
The gist of the problem is when trying to store binary data from a png for instance, the result ends up a handful of bytes larger with the bytes mismatching from the original after a certain (random?) offset it seems.
It has worked ok with data from a small textual svg file and even small png (<2000 bytes?) data.
I'll see if I can rework the example for you.
Thanks!
... the hashing was just an "easy" way to visually compare the binary data results. The console.log
lines with the hex()
of the original binary data object would show the actual data.
Thanks, I switched to Node 18.8.0 to test
It looks like the ascii85 encoding used for transport is causing an issue. I'll have to investigate this further.
Here is an updated snippet, includes a working example in the comments.
import { AceBase } from "acebase";
import { AceBaseClient } from "acebase-client";
import { AceBaseServer } from "acebase-server";
const options = {
path: "./server",
host: "localhost",
port: 5758,
authentication: {
enabled: false,
},
};
const areEqual = (first, second) =>
first.length === second.length && first.every((value, index) => value === second[index]);
const server = new AceBaseServer("default", options);
server.db.ref("*").on("child_added", async (snap) => {
const val = snap.val();
if (val.binary) {
const eventUint8Array = new Uint8Array(val.binary);
console.assert(areEqual(eventUint8Array, blobUint8Array),
'eventUint8Array !== blobUint8Array')
} else {
console.log("Event - child_added: ", val);
}
});
let settings = {
host: "localhost",
port: 5758,
https: false,
dbname: "default",
logLevel: "verbose",
};
//const db = new AceBase(); // seems to work as expected...
const db = new AceBaseClient(settings);
await db.ready();
// this file fails between server <-> client
const url = "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a0/Circle_-_black_simple.svg/480px-Circle_-_black_simple.svg.png";
// this file works between server <-> client
// const url = "https://upload.wikimedia.org/wikipedia/commons/thumb/a/a0/Circle_-_black_simple.svg/240px-Circle_-_black_simple.svg.png";
const response = await fetch(url);
if (!response.ok) {
throw new Error(`HTTP error, status = ${response.status}`);
}
const blob = await response.blob();
const blobArrayBuffer = await blob.arrayBuffer();
const blobUint8Array = new Uint8Array(blobArrayBuffer);
const newFileRef = db.ref("files").push();
await newFileRef.set({
name: new Date().toString(),
type: blob.type,
binary: blobUint8Array,
});
const snap = await db.ref(newFileRef.path).get();
const snapVal = snap.val();
const snapUint8Array = new Uint8Array(snapVal.binary);
console.assert(areEqual(snapUint8Array, blobUint8Array),'snapUint8Array !== blobUint8Array')
const aceBlob = new Blob([snapUint8Array], { type: snapVal.type });
const aceBlobArrayBuffer = await aceBlob.arrayBuffer();
const aceBlobUint8Array = new Uint8Array(aceBlobArrayBuffer);
console.assert(areEqual(snapUint8Array, aceBlobUint8Array), 'snapUint8Array !== aceBlobUint8Array:')
db.close();
server.shutdown();
I just keep muddling my way through... It seems the crux of the issue is a missing continue
here during encode
:
if(!n){
result.push('z');
continue;
Upon research, I stumbled upon the Z85 encoding (z85-codec). Interesting in that no escaping is necessary for json (eg"
). A quick test with a random jpeg had about .5% of the total space used due to escape characters. There might be some computation tradeoffs for and against to consider-- let alone backwards compatibility.
While going down this rabbit hole it made me wonder about packing/squashing data into the high bits of the UTF16 encoding used by IndexedDB.
I think you nailed it with that continue
, I'll perform some more tests and release a new version
I published acebase-client v1.17.1, which will fix this issue from the client side, server side is up next!
I just published scebase-server v1.13.0 which fixes this issue on the server side.
Thanks for your help @donl, really appreciated! 💯
Using regular AceBase seems fine, however in a client/server setup there appears to be an issue with binary data.
Here is an example that hopefully illustrates the issue: