search

TBD54566975 / web5-js

Monorepo for the Web5 JS TypeScript implementation
https://tbd54566975.github.io/web5-js/
Apache License 2.0
132 stars 53 forks source link

Tags for Records does not support UTF-8 #946

Open sondreb opened 3 weeks ago

sondreb commented 3 weeks ago

When attempting to set a UTF-8 character as the value of a Tags on a Record, like an emjoii, I get this error:

Failed to read the 'headers' property from 'RequestInit': String contains non ISO-8859-1 code point."

Since the tags are indexed, perhaps that is the reason? This happens in the browser, so maybe related to IndexedDB.

LiranCohen commented 1 week ago

Adding a little more context to what's happening.

Currently our JSONRPC mechanism JSON encodes the full DWN Message without the data and attaches that to a header dwn-request, however headers cannot contain non-ascii characters and must be encoded.

Solution should be to base64url encode the request before adding it to the header.

It should be noted that http-headers are potentially capped at size, and this is not standardized. Base64URL encoding will increase the payload size, so this should be at least considered.

Some potential caps/limits provided by GPT (should be verified):

Common Limits:

Apache: Default limit is 8KB for the total size of all headers. Nginx: Default limit is 4KB to 8KB for the total header size. IIS: Default limit is 16KB for the total header size. Cloudflare: Limits total header size to 16KB.

Browser Limits:

Chrome: Around 256KB total header size limit. Firefox: About 450KB total header size limit. Safari and IE: Lower limits, around 80KB-100KB.