Partitioning Open Token Registry

[bitjson]

Just want to open discussion on how to partition Open Token Registry to reduce the required download/embedded size. (Thanks to @bchguru and @joemarct for some discussions here already.)

My current thinking:

As OTR grows, we'll want to continue to build our tooling around a process in which pull requests continue to modify the single, merged registry file. Then downstream tooling can partition that larger registry into whichever shapes and formats are most useful to applications. This keeps the update workflow simple: diffs occur in only one place and consistency across all built registry "artifacts" is guaranteed by the tooling.

I think OTR should be distributed in at least 2 shapes:

• single, merged registry file: the complete, raw registry. This could get very large as more NFT collections are added. (For reference, @bchguru's metadata for the first 2000 NFT's weigh in at 2MB.) For embedded and more privacy-sensitive applications, this is the most useful format, as usage does not require continued network access or asynchronous resolution.
• externalized sequential NFTs: the raw registry is parsed to remove at least nft.types of all token definitions in which parse.bytecode is not set. This will generally exclude the art collection type of NFTs while retaining NFT information for most decentralized applications, so the top-level registry may be able to remain under 1MB for much longer.

I don't have a strong opinion yet on how best to externalize portions of registries, but some background: @bchguru initially suggested adding URI support to the IdentityHistory type in the BCMR standard.

I'm hesitant to introduce an asynchronous resolution requirement to the standard itself: (My comment from a direct message)

RE adding URI support for IdentityHistory: I can certainly see us wanting some way for a registry to say "I trust this other registry URI completely" (and an IPFS URI would ensure integrity); I think we would want to add that at a higher level than the identities map (since the other registry can include many identities). That's certainly possible to add just as an extension too (and probably the best way to begin standardizing new things). In practice, that makes registries quite a bit more complex to "resolve", since resolution isn't just a parsing thing, but might include timeouts for fetching data (and then you need to deal with allowed levels of recursion, e.g. some limit like the SPF recursive limit in the DNS world)

Anyways, just opening this issue to collect discussion – I'd like to figure out a set of BCMR extensions that let us partition OTR into sets of smaller, more specifically-focused registries.

[bitjson]

Additional requirement: lets automatically emit a script for pinning all IPFS URIs contained in each registry partition. E.g. (current for the partition that excludes sequential NFTs):

ipfs pin add bafkreig32k6kowcldesjbytktvsh47wnr72qp67ig3aldcjkkkgkx4jmnq
ipfs pin add bafybeigqogqx3n4cldk6nizl5vihopi2dkgvw4yqamc5rhleyqu25soe4e

Would make it easy for projects relying on OTR to ensure they have fully replicated all registry content. Bonus points if we can setup a GitHub Action to automatically pin everything with nft.storage (using the IPFS CLI for flexibility to switch pinning services): https://nft.storage/docs/how-to/pinning-service/

[bitjson]

To revise this:

As OTR grows, we'll want to continue to build our tooling around a process in which pull requests continue to modify the single, merged registry file [...]

It would be much nicer to partition the source data for the registry into a unique file per identity, maybe at src/[authbase].json, and each file would contain only the IdentityHistory for that identity. Then the CI process could merge all identities as appropriate, set a proper registry version for OTR, create the partitioned registries, etc.