search

metaplex-foundation / metaplex

A directory of what the Metaplex Foundation works on!
https://metaplex.com
Apache License 2.0
3.33k stars 6.25k forks source link

Architecture improvement: headless Metaplex #90

Closed etodanik closed 2 years ago

etodanik commented 3 years ago

Would be willing to contribute towards this goal.

In order to make Metaplex truly an empowering platform, it could be useful to extract as much of the logic (both the part that interacts with things like wallets, auctions and also some of the UI logic that's unique to Metaplex) into a headless format. The benefits would be:

There's more to Metaplex than just UI, but some great examples of headless UI libraries in the wild: https://headlessui.dev/ https://www.radix-ui.com/primitives/docs/overview/introduction

josezy commented 3 years ago

How do you imagine the route/roadmap to accomplish this?

etodanik commented 3 years ago

@josezy I'll prepare a rough design suggestion of what my approach would be.

etodanik commented 3 years ago

The perfect Metaplex SDK from a DX perspective

This is a collection of thoughts that I had while trying to work a bit with Metaplex, on how I would make the SDK better if I had a magic wand.

This would bring Metaplex developer experience to par with the most used API's out there like Stripe, Twilio, Sentry, Sendgrid/Mailgun, Shopify and many others that captured lots of developers into their ecosystem.

It would also rocket Metaplex into a light-speed pace of innovation from the developer base.

I can't stress enough how crippled and absolutely frustrated I am right now at the length of time it takes to make Metaplex into anything useable beyond the reference store implementation

General Structure & Layering

First and foremost, don't force the whole might of the Lerna monorepo on me. Let me pick and choose, so I can build my own sexy frontend on Metaplex.

Also, is Lerna really necessary here? There's the package @oyster/common.

Recommendation for a good workflow that doesn't depend on Lerna: https://github.com/wclr/yalc

And of course, there's good ole' npm/yarn link.

And if you do use lerna, still, properly layer and for pete's sake - publish everything to npm so that I don't have to try to rebuild oyster just to play around with some alternative frontends.

Ideally the SDK will have something like the following layers (sorted by low level to high level, each one a package, and as we go up the levels of abstraction, we depend on the lower level packages):

1. @metaplex/protocol

The Rust contract and the underlying types and messages of the protocol that are useful for low level Solana transactions.

2. @metaplex/storage-adapter-arweave

A layer for each underlying storage solution (which I gather right now is Arweave only, but isn't necessary so as the storage is just a URL. It could be nice to build one Arweave implementation but let the community do more. They could come up with adapters for IPFS, Sia e.t.c)

3. @metaplex/core

A layer that calls those low level Solana transactions, some of the initial protocol work happens here. This layer could also include some useful primitives like error handling (nonexistent at the moment). The errors should be well documented outside the Rust codebase.

This layer would be Metaplex only and not be polluted with lots of utilities for Arweave, Coingecko e.t.c - Those should live separately form the Metaplex codebase. For example, for Coingecko - you can publish a little utility npm package.

For Arweave - you can publish something like @metaplex/storage-adapter-arweave

This layer could have stuff like createAssociatedTokenAccountInstruction and all that lower level stuff:

// this is all great code, but 99% of the end-user developers don't give a crap
// about how the deepest internals are structured, so this is why code like this should live
// in a separate core package
import { Keypair, TransactionInstruction } from '@solana/web3.js';
import { Token } from '@solana/spl-token';
import {
  createAssociatedTokenAccountInstruction,
  createMint,
  findProgramAddress,
  programIds,
  StringPublicKey,
  toPublicKey,
} from '@metaplex/core';
import { WalletNotConnectedError } from '@solana/wallet-adapter-base';

export async function createMintAndAccountWithOne(
  wallet: any,
  receiverWallet: StringPublicKey,
  mintRent: any,
  instructions: TransactionInstruction[],
  signers: Keypair[],
): Promise<{ mint: StringPublicKey; account: StringPublicKey }> {
  if (!wallet.publicKey) throw new WalletNotConnectedError();

  const mint = createMint(
    instructions,
    wallet.publicKey,
    mintRent,
    0,
    wallet.publicKey,
    wallet.publicKey,
    signers,
  );

  const PROGRAM_IDS = programIds();

  const account: StringPublicKey = (
    await findProgramAddress(
      [
        toPublicKey(receiverWallet).toBuffer(),
        PROGRAM_IDS.token.toBuffer(),
        mint.toBuffer(),
      ],
      PROGRAM_IDS.associatedToken,
    )
  )[0];

  createAssociatedTokenAccountInstruction(
    instructions,
    toPublicKey(account),
    wallet.publicKey,
    toPublicKey(receiverWallet),
    mint,
  );

4. @metaplex/client

A vanilla javascript client that can batch the lower level Solana transactions into useful and simple to document and understand things like: metaplex.mintNft(options); or metaplex.bid(options); This layer of abstraction should NOT require the developer to understand every single, tradeoff, limitation and design choice of the Metaplex smart contract.

I'm aware that Metaplex processes can get "stuck" because a transaction timed out. This is where you make the DX work: Make sure everything failed is retry-able, cancelable e.t.c

**Error handling should be good, throw good errors for everything possible.

This layer also Has no UI dependencies.

Using it could look something like:

import metaplex from "@metaplex/client";
import {
  Connection,
  PublicKey,
  Account,
} from '@solana/web3.js';

// .. create some basic primitives like a web3 connection

const options = {
    // the various options you need to create an nft
}

try {
    const result = await metaplex.mintNft(options);
} catch(e) {
    console.error(e);
    // something went wrong.. do we need to rewind and replay / retry the whole thing? 
    // maybe there aren't enough SOL in the wallet? let me know stuff like that :)
}

5. @metaplex/react, @metaplex-vue, @metaplex-angular

On top of all that, it'd be ideal to then have a headless layer for React / Vue / Others.... I gave a few good examples of headless libraries: https://github.com/metaplex-foundation/metaplex/issues/90

By Headless, I mean they would consist mostly of hooks (with the hook abstracting away the Context/Statefulness needed). Here is a good headless UI flow on React (some of it already looks like it on the Metaplex repo, but is hopelessly intertwined with the rest of the code and non extractable): 

import { useAuction } from "@metaplex/react";
const AuctionCard = (props: AuctionCardProps) => {
    // this is a bit oversimplified, and some of the variable names may be wrong
    // but I wanted to show the general idea
    const { art, bid, winningBid, minimumBidIncrease } = useAuction(props.auctionId);
    const { title, previewUrl } = art;
  return (
        <div className="auction-card">
            <h1>{title}</h1>
            <img src={previewUrl} alt="Auction image preview" />
            <button onPress={() => {bid({ amount: winningBid.amount + minimumBidIncrease })}}>Bid</button>
        </div>
    ); 
};

6. @metaplex/ui

Some CSS stuff to power your various UI implementations

7. @metaplex/ui-react, @metaplex/ui-vue e.t.c

And finally, on top of all that you might want to make a component library that can be composed with the auction (but does not depend on it internally). Also, a good thing would be if it in turn builds upon a separate set of CSS files that are either vanilla or built on some tree-shakeable and purge-able CSS library like Tailwind, or just from-scratch set of CSS. Here's how it may look:

import { AuctionHeroCard, AuctionCard } from "@metaplex/ui-react";
import { useAuction, useCreator } from "@metaplex/react";

const MyPage = (props: MyPageProps) => 
    const { activeAuctions } = useCreator(props.creatorId);
    const { art, bids } = useAuction(activeAuctions[0]);
    return (
        <>
            <AuctionHeroCard art={art} bids={bids} />
            {activeAuctions.slice(1).map(auction => 
                <AuctionCard key={auction.id} art={auction.art} bids={auction.bids} />
            );
        </>
    );
}

8. @metaplex/storefront

And, only on top of all of those many layers, would a reference store be useful. Yes, it can be done on antd, it should probably be done with next.js, but it would contain maybe 20% of the current code (the rest will go live in various parts of this stack), and would be very readable, maintainable and simple to fork & edit. (fun fact: took me about 5-6 hours after forking to get the monorepo to a workable state because I wanted to rip out antd and do my own design for the frontend)

Solana Issue 1 - Error Handling

It would be cool if Metaplex pioneers a sane way to do errors in Solana. Right now a bunch of mega-cryptic stuff appears in the console.log (which non-tech people would never read, so when an auction fails to list, is effectively game over for them until the programmer wakes up).

Solana Issue 2 - Bandwidth

This plagues Serum and other projects as well, Solana apps download insane amounts of data continuously because each account data change sends the whole state down the WS subscription. I can only see it be solved with proper diff-ing on the RPC level:

web3 / json-rpc: accountNotification always fetches full account data · Issue #17496 · solana-labs/solana

Here is a Network tab for a typical Metaplex storefront:

Screen_Shot_2021-09-06_at_12 30 34

15-50 MB per load is absolutely insane for the web.

Solana Issue 3 - Latency

The RPC isn't great on latency right now, to levels unacceptable for web browsing (sometimes the auctions take more than a minute to load). Perhaps this could be solve with a reference-implementation of a caching server that has a ready-to-deploy k18s / docker-compose recipe for a server that has redis, a WS and REST API and can serve all the data the blockchain would serve but centrally cached for a store ID.

Non Coding - Documentation

This is of course mega important. You should have a "Quick Start" (Metaplexin' in under 5 minutes) for each client type (Vanilla / React / Vue e.t.c). Your documentation should look familiar and friendly to someone who codes well on web2.

Good documentation should have:

Good examples of docs: https://www.twilio.com/docs/sms/quickstart/node

https://stripe.com/docs/development/quickstart

Non Coding - The underlying protocol in an RFC-like format

It would be nice to eventually have the protocol documented as neatly and thoroughly as some RFC's out there. Here's an example, RFC6455:

https://datatracker.ietf.org/doc/html/rfc6455

I once had to re-implement the websocket protocol on a bit-by-bit level on a 8-Bit microcontroller in C language. Because of that RFC, I could.

Something like this could be too big of an undertaking for a young protocol like Metaplex, but eventually it'd be great to have every bit, function, nook and cranny documented.

Non Coding - Community forums or another non-realtime communication channel

Discord is great for crypto community building but is absolutely terrible for development. It isn't well searchable, it's not thread-able and the attention span on a chat is very short. This can be frustrating for serious development help.

It would be crucial for Metaplex to have some sort of an async, non-realtime threaded communication channel like a forum. It would allow:

Here is an example of a community for an API / SDK. They're not doing the best job at always replying, but it has helped me a few times: https://community.auth0.com/c/apiv2/12

Another option is "GitHub Discussions", but then the devs should be very active there (which they're not).

Conclusion

I know that what I laid out is a lot, and could be a lot of work, but the very best DX I ever had on projects followed this kind of format. Even on the blockchain, the best and most adopted protocols (see the imperfect, but well layered IPFS) follow this.

But it's common-place in the most adopted web2 centralized APIS like Twilio, Stripe, Sendgrid/Mailgun e.t.c

What this would allow is insane speed of development for the community (people like me). It took me a whole wasted weekend to try to make sense of Metaplex as it is for the purposes of doing my own little frontend for a store. This greatly prevents Metaplex from becoming a true platform.

MarshallCB commented 3 years ago

Agree with @israelidanny here. Smooth and beginner-friendly DX is crucial for building out a strong ecosystem. Would also be willing to contribute. I'm new to Solana but familiar with web2 JS libraries

github-actions[bot] commented 2 years ago

This Issue has received no activity for 30 days. We will close it in 2 days, please reopen if you are still experiencing this issue.