Arweave Integration Brainstorming

[neruthes]

Main Issue	Documentation
https://github.com/DimensionDev/Maskbook/issues/1057	https://confluence.dimension.chat/x/3IEf

[neruthes]

Attachment Service

Introduction

Maskbook Post can employ Arweave to support publishing attachments.

Steps

Publishing

• Alice clicks "Arweave" in "Plugins (Experimental Feature)" section.
• Arweave plugin opens a PluginScreenModal where Alice may upload a file (or multiple files).
• Arweave plugin renders a PACPreviewView in the Maskbook Post Composition View (MbPCV).
• Alice publishes the post.
• Arweave plugin uploads the files in the background.

Viewing

• Bob sees the decrypted post.
• Bob sees the PAC (PostAffixingCanvas) of Arweave plugin.
• Bob clicks the link (or file icon).
• Browser opens Arweave file URL.

[septs]

Redefinition Wallet feature set:

Basic feature set

1. show name (settable)
2. show wallet logo (or coin icon?)
3. show wallet address
4. show current balance
5. show private key
6. transaction history (basic fields)
7. remove this wallet

Extended feature set

1. transfer balance 1.1. Send AR for Arweave 1.2. Send Red Packet for Ethereum
2. transaction history (more fields)
3. provider specific feature set

[neruthes]

Moderate Issue:

Arweave does not distinguish MetadataPreviewPage and DirectDownloadUrl. It is difficult to allow arbitrary binary attachments.

[septs]

Arweave metadata in tags

• https://viewblock.io/arweave/tx/9h5BzarJ_XhSh1k8GjDK16bN5zHEBG4wdHaO9XBnHm4
• https://viewblock.io/arweave/tx/ILGdGJh9jq4HNC-m5ve27FTeboAkmDdVhKw7eZfkDbs
• https://github.com/ArweaveTeam/arweave-deploy/blob/71c99a7bebac0e5d9c12497100453e6c309da04f/src/commands/deploy-dir.ts#L143-L144

[septs]

Predefinition Tags

Content-Type: [mime type]
User-Agent: Maskbook/[version]
X-Action: [action format definiton]
X-Hint: [hint text]

action format definition (URN based)

# attachement
user-action:com.maskbook:1:plugin.org.arweave:1:attachment:1
# masbkook backup with encrypted (archived)
user-action:com.maskbook:1:plugin.org.arweave:1:maskbook-backup:1

[Jack-Works]

I found you two are inventing strange standard 🤔

[neruthes]

Join the brainstorming and offer your ideas please 🎉

[septs]

sample application/pdf: https://arweave.net/d_6yFXJGH52rYID8H9-ettLqe-Lknv4YQOWpu3wXGVw https://viewblock.io/arweave/tx/d_6yFXJGH52rYID8H9-ettLqe-Lknv4YQOWpu3wXGVw

sample audio/mp3: https://arweave.net/Lf-cEVOMqUCDoPqyp2H594YxIGkrN5Ih-CAGwsyVmLY https://viewblock.io/arweave/tx/Lf-cEVOMqUCDoPqyp2H594YxIGkrN5Ih-CAGwsyVmLY

[septs]

Related: https://github.com/DimensionDev/Maskbook/issues/1131 https://github.com/ArweaveTeam/arweave/issues/240

[neruthes]

Related: DimensionDev/Maskbook#1131 ArweaveTeam/arweave#240

15 confirmations... How long does it usually take? 🤔

[septs]

Related: DimensionDev/Maskbook#1131 ArweaveTeam/arweave#240

15 confirmations... How long does it usually take? 🤔

12 confirmations at 10 MiB blob ≈ 10 minutes This result is NOT CREDIBLE (i need to repeated observe)

[septs]

arweave network has many small blob (not more then 1 MiB transaction) (for webpage archive, maybe is official browser extension designed usage?)

[yisiliu]

We don't really need to focus on the web page archiving at first. All we need is to support 0-15MB file uploading onto arweave network directly from our extension.

[neruthes]

Plugin: Feature: File Manager

Proposed by @OiCkilL. We want to include a file manager in the plugin.

A user can:

• View list of files
• See transaction detail
• Go to *.arweave.net HTTPS online

[neruthes]

Wallet: Feature: WalletType-Specific Transaction History Interpretation

Coverage

• UIWalletTransactionHistoryListCellView
• UIWalletTransactionHistoryDetailView

Changes

These views should be able to render rich content for the wallet type.

[neruthes]

UI Wireframe: Upload Attachment When Composing Post

[neruthes]

UI Wireframe: Plugin Main View (Not For Stage 1)

[neruthes]

Proposal: Move Arweave Wallet Into Arweave Plugin

Proposed by @OiCkilL.

We previously wanted to add Arweave to the list of supported wallet types. However, it is possible that the Arweave wallet can exist inside the Arwaeve plugin, since its third-party applications (within Maskbook) will remain rare in the upcoming future.

We can greatly reduce technical complexity if we let the Arweave plugin manage the Arweave wallet. No need to hurry on designing wallet operation request API, etc.

However, this proposal comes with certain challenges.

Challenge 1: Plugin Deletion & Data Lifecycle

Although we are not going to really implement plugin deletion feature right now, it is necessary to inspect the meaning of deletion. If the user deletes the Arweave plugin, shall the Arweave wallet remain in the Maskbook database? The answer seems to be false.

We can have a safer data lifecycle, like the idea employed by Dropbox. Apps can store their data in Dropbox, and the data is not deleted upon uninstallation, and the user will be able to purge junk data manually in the Dropbox storage management panel. This kind of data management could be a bit overkill for Maskbook.

Minimally, we may have a "Purge remaining data of deleted plugins" button.

Challenge 2: Cognitive Inconsistency

It may have been well recognized that all Wallets should be managed by Maskbook instead of the Red Packet plugin. If now the Arweave wallet is managed in the Arweave plugin, a user may be confused.

It remains uncertain whether this is a big matter, but we have to be prepared.

We previously discussed the idea of wallet virtualization (similar to #40). Maskbook can include a Virtual Wallet in the list of Wallets in the Wallets tab in Dashboard. When the user tries to make some operation, it is actually the plugin which handles the requests of the user, instead of native Maskbook wallet operation panel. However, this will make it harder to understand where a wallet is actually stored. Some text explanation can help, but uncertainty remains.

[neruthes]

Discussion Summary

Arweave Wallet Keygen can be complicated. My wish is that the keygen can happen inside Maskbook and the user can skip Arweave official website for this process. Otherwise, the steps will be too frustrating for a normal Maskbook user who merely wants to upload an attachment.

And @yisiliu confirmed that it will be possible that an Arweave wallet generated in Maskbook will get free file uploading eligibility per our agreement with Arweave.

It may be a good choice to keep keygen automated and within Maskbook.

[neruthes]

Note

Since the major purpose is to upload encrypted attachments, we will not be able to employ the native HTTPS online distribution service of Arweave official website.

This imposes several design restrictions:

• The user can only upload an attachment when composing a Post so a PostKey can encrypt the file; otherwise the file will be unencrypted.
• When displaying the attachment, Maskbook cannot simply open a URL like \https://ovd753swy5w7ntippwytucubwjrldkjmdqxaxf2zvumtqe4ytcwq.arweave.net/dUf-7lbHbfbND32xOgqBsmKxqSwcLguXWa0ZOBOYmK0 and has to download and decrypt it locally.

When designing the file-viewing experience, Maskbook should not open arweave.net. Instead, Maskbook shall open a in-the-extension webpage which is managed by Maskbook where necessary parameters are passed. For example, chrome-extension://nbajgjnadncebloipcedoannhljjkiel/plugin/com.maskbook.provide.net.arweave/encryptedFile.html?id=dUf-7lbHbfbND32xOgqBsmKxqSwcLguXWa0ZOBOYmK0&postKey=12345678.

[neruthes]

Note

Although Arweave offers high flexibility for file size, it may be good to have a file-size limit. If the file is too large, the usability of the attachment service will be in danger.

My proposal is that the limit should be 20 MB. Looking forward to seeing proposals for other numbers.

[yisiliu]

I proposed 15MB but 20 should be fine too. Another purpose of this is to later upload user's encrypted posts onto the Arweave network and provide them a dashboard for them to view.

[neruthes]

Proposal: Independent FileKey

In the long run, we want to have a comprehensive access control strategy and reading the PostKey of a Post can be considered dangerous. It may be necessary to prevent the Arweave plugin from accessing the PostKey of a Post.

With this in mind, it might be a good idea to split the PostKey and the FileKey. The FileKey is used to encrypt the file and it is put in the Post payload as a field of plugin-specific private metadata.

This will allow the Arweave plugin to better manage the list of files without endangering the confidentiality of PostKeys.

[neruthes]

Note

I just summarized recent fruits of discussions into the documentation. Looking forward to feedbacks.

[neruthes]

Discussion Summary

• Arweave server cannot offer files list of current wallet.
• We will not implement File Manager in the current stage.
• When composing Post, the list of my files should be changed to the list of unshared uploads.
• Consider allowing changing filename before uploading.

[neruthes]

Note: Blockchain-Based Attachment Blob Data Structure (Overall)

Encrypted Blockchain-Based Attachment Description

Mounting Point:

• Post
  • Post Metadata
  • com.maskbook.provide.net.arweave
    • EncryptedFileDescription

Data Fields:

• TransactionId
• RawFileHash
• EncryptedFileHash
• FileKey
• ~FileKeyHash~ (?)
• ContentType
• FileName
• FileSize

Blockchain-Based Attachment Binary Blob

Data Fields:

• Format Version (Current: 0)
• Metadata
• ~FileKeyHash~ (?)
• RawFileHash
• EncryptedFileHash
• OtherMetadataForAES
• AES.encrypt(RawFile, FileKey)

Binary Serialization:

// To be designed by @septs.

[septs]

Attachment Blob Magic header:

MASKBOOK-ATTACHMENT as ASCII encoding

to identify maskbook attachment blob file

[septs]

@yisiliu apply https://msgpack.org to maskbook attachment blob format OR design your own binary layout?

[yisiliu]

apply https://msgpack.org to maskbook attachment blob format

Actually in the design of ntge, we are also considering moving to msgpack so yes go ahead with it.

[neruthes]

Discussion Summary

Proposed by @yisiliu.

The FileKeyHash field should be removed from on-chain payloads.

[neruthes]

Discussion Summary

Proposed by @septs.

The file limit should start at 1 MB and we can raise it gradually according to the feedbacks from the users.

[neruthes]

Discussion Summary: Upload clearfile attachment without encryption

It can be a common scenario where a user merely wants to take advantage of the attachment feature of Maskbook without encrypting the file.

Interaction Design

We can have a checkbox to toggle off "Encrypted Attachment" or another button "Upload Without Encryption". The user can use this to access the basic file uploading feature of Arweave.

Data Structure

The returned data from the "Attachment by Arweave" plugin (Plugin-Specific Post Metadata) should contain 2 fields:

• EncryptedFileDescription
• ClearFileDescription

If the file is encrypted, the EncryptedFileDescription should be a valid Encrypted Blockchain-Based Attachment Description object (as defined above), and the ClearFileDescription should be null.

If the file is unencrypted, the ClearFileDescription should be a string of Arweave Transaction ID, and the EncryptedFileDescription should be null.

Error Handling

• If a valid EncryptedFileDescription is found, the ClearFileDescription field shall be ignored.

[yisiliu]

Due to the limit of the expected confirmation time to be ~15-20 minutes, users cannot retrieve the data in that waiting time, which would cause user's misunderstanding of our production. I've received some suggestions on how to improve this experience and here is the proposal:

Let's maintain a cache server to hold the file for the users temporarily and set the lifetime of these temps as 30 minutes. Each time a user would like to access a file, Maskbook would first query arweave.net/ID. If it doesn't work, then Maskbook would query maskbook.com/temp/ID to access it as a temporary solution.

What do you guys think?

[neruthes]

Let's maintain a cache server to hold the file for the users temporarily and set the lifetime of these temps as 30 minutes. Each time a user would like to access a file, Maskbook would first query arweave.net/ID. If it doesn't work, then Maskbook would query maskbook.com/temp/ID to access it as a temporary solution.

A good way to enhance user experience, indeed. I would like to suggest using domain name arweave.net.provide.maskbook.com, to conform to the package name of the plugin, com.maskbook.provide.net.arweave.

[neruthes]

Discussion Summary: Image Preview

For images, the plugin can store a small preview image (e.g. 20*20px) upon uploading or upon downloading. When browsing the list of files, if the preview image can be found in the local database, the preview image shall replace the file type icon.

The preview images shall not be included when exporting plugin data or creating Maskbook database backup.

[neruthes]

Discussion Summary: Go to Post

When browsing the list of files, the user should be able to jump the URL of the Post which contained the attachment, if the file is from a Post.

When witnessing a Post which contains Plugin-Specific Post Metadata for "Attachment by Arweave" plugin, the plugin is notified. Upon notification, if the TransactionId is not associated with a PostLocator, the plugin shall record an event to associate the PostLocator with the TransactionId.

File browsing is not included in Stage 1.

[neruthes]

Discussion Summary: Dashboard Plugin Panel

The Plugin panel in Dashboard is not included in Stage 1.

[septs]

@neruthes

1. Require a MIME Type for Content Type (Encrypted payload)

Content-Type: application/x.dimensiondev+encrypted-v1

Reference: https://en.wikipedia.org/wiki/Media_type#Registration_trees

2. Raw file
   1. Prefer original MIME Type
   2. Use application/octet-stream when don't know data type

[yisiliu]

Looks good

[neruthes]

Idea: Reuse Landing Page

We can reuse the same landing page for all these files since the only varying element in the landing page would be the File Key, which can be easily passed to the landing page by location.search or location.hash.

Additionally, if we can solve cross-origin problems, the landing page can be editable, allowing better revamping and debugging.

Certain metadata should be passed to the landing page from the Plugin "Attachment by Arweave":

• FileKey
• FileKeyHash
• File Name
• File Size
• Content-Type

Pros

• Better waiting experience: The viewer can see the "waiting transaction confirmation" notice in the landing page instead of Maskbook Post UI.
• Saving time slightly for reducing a serial confirmation sequence of 2 files into confirming 1 file only.
• Less risk with extension store review policy.

Cons

• Technically, this new solution makes the Attachment by Arweave plugin less independent.

[neruthes]

Question

Should it be prohibited that sharing the landing page URL can result in allowing the recipient to decrypt the file?

[septs]

Roadmap

• [x] File Format Builder
• [x] File Format Parser
• [x] Landing Page (Preview Edition)
• [x] Maskbook integration