Closed rafaelcr closed 1 year ago
Really happy to see this!
My first thought around this is the concept around having to specify a list of asset id's for the NFT section
(define-public (nft-metadata-update-notify (contract <nft-trait>) (token-ids (list 100 uint)))
what if we wanted to refresh the megapont robot components (tens of thousands of assets)? I feel like we need an option to do all, or a range, or some other syntax. What do you all think?
I think this is a great proposal - I like the approach of standardizing print
events which a schema.
My only real feedback is the same as Thomas's - the most likely use case is going to be "update all metadata", in the case of reveals. This should definitely be able to handle that. Secondarily would be ranged updates, which would be nice, but my guess is that it's not a hugely common use case.
Another thought - is including the contract-id
in the payload necessary? My first thought is that this can't really be verified - which means it could be abused as a kind of DDoS vector. Maybe not an issue, because you'd have to pay TX fees. Also, isn't the contract principal included in the payload of the print
event by default? Maybe this doesn't work, because we want this to work for existing NFT contracts.
Overall I'd say 👍🏼
My only real feedback is the same as Thomas's - the most likely use case is going to be "update all metadata"
@hstove I agree that this is a very real and valid use case. A simple solution would be to make the token-ids
argument optional. If it's not provided, then indexers would refresh metadata for all token IDs. cc @aulneau
Another thought - is including the contract-id in the payload necessary? My first thought is that this can't really be verified - which means it could be abused as a kind of DDoS vector. Maybe not an issue, because you'd have to pay TX fees. Also, isn't the contract principal included in the payload of the print event by default? Maybe this doesn't work, because we want this to work for existing NFT contracts.
That's right, including the contract-id is necessary if you want to send notifications for tokens declared by old contracts. We designed it this way so any contract may send notifications on behalf of any other contract. As you say, this could be abused but we decided transaction fees were enough to discourage that. Besides, the side effect of abuse is that contracts will just have their metadata refreshed.
Thank you for your submission @rafaelcr! I think this SIP looks pretty good. One thing I would add before advancing it would be a section about the API contract this creates for indexers and metadata servers. Because in the legacy case anyone can trigger a refresh event, I think at the very least you'll want to point this out in this SIP that indexers should be prepared for things like gratuitous, delayed, and out-of-order notification delivery (since you don't know the order txs will be mined in, or that they even correspond to an actual metadata change), and all metadata URL queries should be idempotent.
Couple thoughts in response to the feedback:
I agree with @aulneau and @hstove about usefulness of bulk updates for NFT metadata. That said, I'd like to dig more into the scalability implications (e.g. DDoS possibilities) that have been brought up. We should encourage NFT-class owners to only issue update notifications for tokens which actually need updated, as opposed the easy path of "update all", which would likely be (ab)used if available. Perhaps having a token limit per contract-call tx would actually be a helpful limiter here.
As an example, a request to refresh metadata for all tokens in a given NFT class results in DDoS-like traffic patterns, where an indexer ends up issuing tens of thousands of http requests. Common hosts like cloudflare or public ipfs-http gateways have high likelihood of rate-limiting / IP-banning the indexer service.
We should disallow the ability for anyone to trigger tens or hundreds of thousands of NFT metadata resolution processes with a few relatively cheap contract-calls. This could be done in a way that is backwards-compatible with existing token contracts by requiring a signed message that matches the principal of the contract in the update event payload.
After thinking through the above, it's clear that there are significant interface and scale difference between FT and individual-NFT metadata operations.
I think it would be reasonable to separate this into two different SIPs: one for FT metadata, another for NFT metadata.
In my opinion, this first SIP should be limited to FT metadata updates, and a later followup SIP can be related to NFT metadata updates and tackle issue around scalability, DDoS, and developer-friendly interfaces. That seems like a logical progression:
Thanks for your review @jcnelson , I've addressed your observations and I've also added a new section covering considerations for metadata indexers. Looking forward to your comments.
@zone117x and myself were discussing abuse vectors for NFT metadata updates, here's a brief summary on some conclusions we reached:
We should disallow the ability for anyone to trigger tens or hundreds of thousands of NFT metadata resolution processes with a few relatively cheap contract-calls. This could be done in a way that is backwards-compatible with existing token contracts by requiring a signed message that matches the principal of the contract in the update event payload.
A simple solution to this problem could be to modify this SIP so that notifications are treated as valid only when initiated by the same principal who deployed the contract declaring the tokens to be refreshed.
For example, if the Megapont developers wish to notify metadata was updated for the entire Wasteland-Apes
NFT collection, they would need to make the notification contract call transaction from the SP2KAF9RF86PVX3NEE27DFV1CQX0T4WGR41X3S45C
address, while specifying SP2KAF9RF86PVX3NEE27DFV1CQX0T4WGR41X3S45C.wasteland-apes-nft::Wasteland-Apes
as the contract_id
in the notification payload. Metadata indexers would then look at the sender_address
value in the incoming transaction and compare it against contract_id
from the notification payload looking for a match in the principal value before refreshing metadata.
Making this change will get rid of any worries on third-party attackers constantly issuing "update all" notifications for large NFT collections, i.e. it will remove the ability to perform gratuitous updates.
Nevertheless, there are remaining worries around other kind of attacks like a malicious agent deploying an NFT contract, intentionally making metadata refresh calls really heavy (in size and/or computation required for nodes), and then making constant "update all" notifications on their contract. This could potentially still successfully DDoS metadata indexers and some nodes.
I'd love to get some thoughts on this @aulneau @hstove
@zone117x SIP-016 - NFT metadata is not yet activated.
It's in the process of being activated.
Left a few more comments. Feel free to rename this tip SIP-019 and set its status to Accepted (and add me to the Signed-off
section).
Let me know when you want the technical CAB to give it a formal review.
@jcnelson this SIP is ready for a formal review. Looking forward to their findings.
Thanks for addressing everything! @kantai whenever you're ready, can you give this a review?
Implementations of marketplace functions for NFTs https://github.com/stacksgov/sips/issues/51 use also events to notify indexers with list
, unlist
, buy
notifications. Instead of notification
, the label action
is used, so that it is easier to filter a database that stores the raw events. action
will be always the first part of the hex representation of the event content.
For metadata update, notification
comes before payload
, so the database search works as well.
Can this SIP be a specification for notifications in general and define more notification types and payloads in the future?
Can this SIP be a specification for notifications in general and define more notification types and payloads in the future?
@friedger while this SIP wasn't meant to be a general definition for contract notifications, the proposed payload structure (notification
, payload
) was definitely meant for them to be expandable in the future. Do you believe it's worth expanding on this topic a bit inside this SIP?
@rafaelcr If you had the intention, I would mention that the generalisation is out of scope but intended to be adopted in future SIPs as well.
This looks great to me! You can add me to the sign-off
section of the SIP now as well.
Cool, the technical CAB has formally approved this SIP. It can now be advanced to Recommended
status.
Also, because there's nothing blocking activation, you can go ahead and advance the SIP directly to Activation-in-Progress
. Great work everyone!
Done, thanks @jcnelson @kantai !
Hi all! Love the idea of this SIP and I think it can bring value to those who rely on accurate metadata. As mentioned in today's community call, I think there could be value in having some kind of (potentially optional) flag to add a bit of color as to what is happening with this particular change.
There could potentially be a single parameter that is passed that would cover each of these three statuses, which I believe would be mutually exclusive:
This could essentially all fit into one additional parameter like uri-persistence
or similar. And by accounting for instances like 1 and 3 above, different processes could be triggered or avoided without having to monitor for completely separate transactions occurring independently like freeze-metadata
calls.
Thanks for your input @nickgamma , based on your suggestions I think we can introduce a new enum property on the notification payload tuple called update-mode
with the following possible values:
standard
: The new metadata will be valid until the next notification comes (default mode if none is specified)dynamic
: The new metadata is expected to change very quickly and many times in the future. Indexers should not expect to receive explicit notifications for each of these changes and should consider refreshing this token's metadata frequently.
This could also be accompanied by an optional ttl
property that makes a suggestion to indexers on how long they should consider the metadata valid until a new refresh is required (specified in seconds).frozen
: This token's metadata will never change again, ever. Indexers should ignore new notifications for this token, even if valid.(I think the property name update-mode
works better than uri-persistence
because changes could also happen inside the JSON, not necessarily the URI)
What do you think about this?
I will also take this opportunity to add a SIP-013 notification spec
Just curious what the reasoning behind this is, as opposed to gating the function call itself with some permissions.
I think this makes it a bit too restrictive by constraining the indexer to follow (2). Sometimes, the deployer address belongs to a developer, or some third party, for example. If the contract wants to restrict triggering metadata updates to the contract deployer, then just gate the function with (is-eq tx-sender contract-owner)
I guess there could be cases where the contract writer forgets to add checks and the notification function could get 'spammed', but the rigidity of the second condition seems a bit too much to me.
EDIT: ah I see, I misunderstood by thinking that these functions would be included in the NFT contract itself, rather than in a separate contract. Is that just for backwards compatibility or a more general solution?
(I think the property name
update-mode
works better thanuri-persistence
because changes could also happen inside the JSON, not necessarily the URI)What do you think about this?
I will also take this opportunity to add a SIP-013 notification spec
That sounds great!
@Jamil
Is that just for backwards compatibility or a more general solution?
Exactly. For example, the reference contract allows a token owner to still emit a notification for their old contract that was deployed before this SIP was proposed.
For newer contracts, you have the option to emit the print
s directly e.g. inside a function that changes the metadata URI for a token.
Updated meta data via contract call: https://explorer.stacks.co/txid/0xfc81a8c30025d7135d4313ea746831de1c7794478d4e0d23ef76970ee071cf20?chain=mainnet And will do so whenever a new exhibition is inaugurated.
Is there any use for these transactions on-chain? If not, the argument against off-chain communication is not convincing.
I suggest to investigate the matrix protocol, create an open room, where signed messages indicate updates of token meta data. Indexers can verify the signatures and act upon the message content.
No centralized service involved.
This SIP could be reduced to the definition of which matrix room to choose for notifications and a process to update the room.
Hi @friedger, thanks, that is a good question. I really believe these notifications should be on-chain primarily for the following reasons:
1) For newer token contracts, emitting a notification on a metadata URI update function is trivial.
The transaction you provided is a great example of this, actually, as it piggy-backs on the inaugurate
function call you intended to do anyway and emitted a notification consistent with this change immediately. Older contracts (before this SIP) will have an additional notification step to perform, of course, but this would be slowly outpaced by the deployment of newer token collections. Adding the print
to the function's Clarity code also serves as self-explanatory documentation, and ultimately I think this will encourage more devs to emit these notifications.
1) It does not force a dependency on a third party protocol.
Even though the Matrix protocol is free, open source and decentralized, it still adds a layer of friction for developer adoption of this SIP (both for creators and indexers) and creates a dependency to something outside of the Stacks ecosystem, which IMO is not in the best interest of a SIP.
1) The notification can be clearly displayed in the explorer (and any other similar/relevant apps) alongside the transaction.
This also reinforces the intent of updating the metadata and makes it explicit to users that this actually happened in the contract. The explorer UI can be improved to accommodate this in a better way, of course, but the effort to do something like that would be minimal.
1) For chain indexers, consuming this event is practically free if they already process transactions.
They would also have an immediate association of the notification with the actual transaction that made the change if they emitted the event within it.
1) Adding notifications through print
is consistent with current practices and opens up new use cases in the future.
Print events are already heavily used by BNS, for example, and I think adding to that practice is a good idea for consistency and standardization. If there is a new use case that comes up in the future, a newer SIP can propose an additional print structure and indexers would be quick to adopt these if they need to.
To be fair, these advantages would not present themselves entirely if the metadata update was done without the need for a transaction (e.g. updating a JSON file in Amazon S3), however, calling one of the provided reference contracts to emit the notification would make this task simple for developers or creators in this case.
After writing this response I realize that these reasons should also be included in the SIP text, so I will fix that. Nevertheless, if you have any other thoughts on this I'll be happy to discuss.
Cheers
Thank you for your thoughts. It makes sense to have a standard when looking at tx like inaugurate
.
I still believe that we should not promote notifications for off-chain data if the ordering of events do not matter. Maybe that can be clarified in the SIP. Probably, high gas fees will prevent this use case anyway in the long run.
Re dependencies, we have already dependencies on http, p2p protocols, gaia,.. Does the Stacks ecosystem have closed borders?
I now approve :-)
- For chain indexers, consuming this event is practically free if they already process transactions. They would also have an immediate association of the notification with the actual transaction that made the change if they emitted the event within it.
Conversely, client-side applications now have an extra dependency/cost of running an indexer. Pushing more centralized databases seems counter-productive in the long term.
@dantrevino what I tried to say is that if there is already any client app that listens to node RPC messages to process token transactions and events, they only need to add an extra check to see if the print events they already receive (in the events
key in /new_block
messages) represent a SIP-019 notification. No extra dependency required.
Adoption, so far, is not that big yet: https://github.com/boomcrypto/clarity-deployed-contracts/search?q=token-metadata-update
Is it just about increasing this SIP's awareness to target audiences like the marketplaces? If so we (or I) can help keep relevant people updated on this
Hiro's Token Metadata Service could also help increase adoption once it's deployed to production and it gets some user traction. We aim to publish a beta before year end.
I'll be hosting a demo this week if you're interested: https://discord.com/channels/621759717756370964/623217767356694547/1049746072840454214 @friedger @Hero-Gamer
We'll be adding this to Gamma's Create Portal contracts by default within the next week or so.
Hi @Jamil I wonder if has Gamma added SIP-019 support?
Adoption of this SIP has picked up considerably in the last few months. We're now at ~1,600 contracts using these notifications: https://github.com/boomcrypto/clarity-deployed-contracts/search?q=token-metadata-update
As far as I can tell from these contracts, in terms of indexer support these notifications are emitted by and/or used by:
I think we can now safely consider all activation criteria are met for this SIP 🎉 cc @Hero-Gamer As such, I hereby kindly request the Steering Commitee to review these results and move SIP-019 to Ratified
Marvin here from the Steering Committee. I think it is safe to say that all activation criteria have been met 😁. You can increment the status to Ratified
and append me to the sign-off list Marvin Janssen (http://github.com/MarvinJanssen)
. Thanks!
Done @MarvinJanssen , thanks!
🎉
hi there, i deployed a token yesterday, but metadata are not yet indexed. there is something I can do to fix it? should i launch a metadata update to see if it works? with the same contract on testnet it tooks about 8hours to index, but now are more than 24h on mainnet and still wrong decimals..
Hi @eriqferrari you can try a metadata update. Usually IPFS takes a while to reflect a file across nodes so Hiro's API (and other indexers) could have missed the data when you first deployed the token. However, to keep this conversation about this SIP only, please open an issue in https://github.com/hirosystems/token-metadata-api if you still believe there is a problem. Thanks.
Preamble
SIP Number: 019
Title: Notifications for Token Metadata Updates
Author: Rafael Cárdenas (rafael@hiro.so), Matthew Little (matt@hiro.so)
Consideration: Technical
Type: Standard
Status: Activation-in-Progress
Created: 17 May 2022
License: GPL-3.0
Sign-off: Jude Nelson (jude@stacks.org), Aaron Blankstein (aaron@hiro.so)
Layer: Traits
Abstract
As the use of tokens (fungible and non-fungible) has grown in popularity, Stacks developers have found novel ways to define and use metadata to describe them. This rich data is commonly cached and indexed for future use in applications such as marketplaces, statistics aggregators, and developer tools like the Stacks Blockchain API.
Occasionally, however, this metadata needs to change for a number of reasons: artwork reveals, media storage migrations, branding updates, etc. As of today, these changes do not have a standardized way of being propagated through the network for indexers to refresh their cache, so the display of stale metadata is a very common problem.
This SIP aims to define a simple mechanism for developers to notify the Stacks network when metadata for a token has changed, so interested parties can refresh their cache and display up-to-date information in their applications.
Introduction
Smart contracts that declare NFTs, FTs and SFTs conform to a standard set of traits used to describe each token (see SIP-009, SIP-010 and SIP-013). One of these traits is
get-token-uri
, which should return a URI string that resolves to a token's metadata usually in the form of a JSON file. There is currently no defined structure for this data, and it is not considered to be immutable.To illustrate a common use of
get-token-uri
, we'll look at theSPSCWDV3RKV5ZRN1FQD84YE1NQFEDJ9R1F4DYQ11.newyorkcitycoin-token-v2
contract which declares the NewYorkCityCoin fungible token.At the time of writing, the value returned by this contract for
get-token-uri
is the string:When this URI is resolved, it returns a JSON file with the following metadata:
Even though the URI string is fixed, this file lives off-chain so it is conceivable that its contents could change at any point in the future. Additionally, this contract includes a way for its owners to change this URI via a
var-set
function call:This setup is very flexible for administrators, but it creates a complex problem for metadata indexers which now need to figure out if (and when) they should re-index token contracts to avoid displaying stale metadata in their applications.
Metadata staleness
Within the Stacks ecosystem, there are a number of applications that need to index token metadata and struggle with specific challenges caused by changed metadata. For example:
get-token-uri
on-chain variable could change, the off-chain JSON file could change, and/or the image served by the URL could change.For indexing, developers usually run and maintain a background process that listens for new token contracts deployed to the blockchain so they can immediately call on their metadata to save the results. This works for new contracts, but it is insufficient for old ones that may change their metadata after it has been processed.
To avoid staleness, some indexers resort to a cron-like periodic refresh of all tracked contracts, but while this may work for individual applications, it does not provide a consistent experience for Stacks users that may interact with different metadata-aware systems with different refresh periods. This workaround also adds unnecessary network traffic and creates extra strain on public Stacks nodes due to aggressively polling contract-read RPC endpoints.
Metadata update notifications
To solve this problem reliably, contract administrators need a way to notify the network when they have made changes to the metadata so any indexers may then perform a refresh just for that contract.
The proposed mechanism for these notifications leverages the
print
Clarity language function. When used, its output is bundled inside an event of typecontract_event
:This event is then attached to a transaction object and broadcasted when the same transaction is included in a block or microblock.
This SIP proposes a standard message structure (similar to a notification payload) that would be used through
print
. Existing metadata indexers would receive this event through the Stacks node event-emitter interface, parse and validate its contents, and refresh any contracts that were updated.print
was also selected for the following reasons:print
notifications in the Stacks ecosystem: the BNS contract, for example, uses it to notify the network when a change to a name or its zonefile has occurred. The PoX-2 contract for Stacks 2.1 will make heavy use of it to record stacking state changes across addresses. This SIP aims to continue this trend.print
notification to a function's Clarity code also serves as self-explanatory documentation.print
structure and indexers would be quick to adopt these if they need to. See Notification structure reusability.Specification
Notification messages for each token class are specified below. Token metadata update notifications must be made via a contract call transaction to the deployed reference contract or from a call to
print
within any other contract, including the token contract itself.Fungible Tokens
When a contract needs to notify the network that metadata has changed for a Fungible Token, it shall call
print
with a tuple with the following structure:notification
"token-metadata-update"
payload.token-class
"ft"
payload.contract-id
payload.update-mode
payload.ttl
payload.update-mode: dynamic
Non-Fungible Tokens
When a contract needs to notify the network that metadata has changed for a Non-Fungible Token, it shall call
print
with a tuple with the following structure:notification
"token-metadata-update"
payload.token-class
"nft"
payload.contract-id
payload.token-ids
payload.update-mode
payload.ttl
payload.update-mode: dynamic
If a notification does not contain a value for
payload.token-ids
, it means it is requesting an update for all tokens.Semi-Fungible Tokens
When a contract needs to notify the network that metadata has changed for a Semi-Fungible Token, it shall call
print
with a tuple with the following structure:notification
"token-metadata-update"
payload.token-class
"sft"
payload.contract-id
payload.token-ids
payload.update-mode
payload.ttl
payload.update-mode: dynamic
Notifications for SFTs must include a value for
payload.token-ids
.Metadata update modes
Applications may use tokens for very different purposes. Some of these could require none or very few metadata updates ever (e.g. digital artwork that never changes except for reveals), while others could need to alter it several times a day (e.g. NFTs for in-game items that are traded and modded continuously).
This use-case variety also affects how developers decide to host their metadata JSON files. For example, they could choose to use IPFS for low-frequency updates and finality, versus Amazon S3 for high-frequency off-chain updates.
In order to allow creators and app developers to specify how token metadata should be treated by indexers, notifications support an optional
payload.update-mode
key that may contain one of the following values:standard
: The new metadata will be valid until the next notification comes.This is the default mode if none is specified.
frozen
: This token's metadata will never change again, ever.Indexers should ignore new notifications for this token, even if valid.
dynamic
: The new metadata is expected to change very quickly and many times in the future (even off-chain).Indexers should not expect to receive explicit notifications for each of these changes and should consider refreshing this token's metadata frequently. Token developers may suggest a reasonable amount of time between refreshes by adding an estimated value (defined in seconds) to the
payload.ttl
notification property.Considerations for metadata indexers
For a token metadata update notification to be considered valid by metadata indexers, it must meet the following requirements:
contract_identifier
field of the contract event must be equal to thepayload.contract-id
(i.e., the event was produced by the contract that owns the metadata) or the transaction'stx-sender
principal should match the principal contained in the notification'spayload.contract-id
(i.e., the STX address that sent the transaction which emits the notification should match the owner of the token contract being updated).Notifications that do not meet these requirements must be ignored.
Other implications
Given these constraints the notifications this SIP proposes should be taken as hints to metadata indexers. Metadata indexers are not obliged to follow them.
Notification structure reusability
Even though establishing a generalized smart contract notification standard is out of scope for this SIP, the proposed
print
message structure was designed for reusability by future SIPs that wish to standardize other events.For example, developers could vary the
notification
andpayload
values to notify the network when an NFT collection has been fully minted or another important milestone is reached.Related work
An alternative considered for token metadata update notifications is for them to be transmitted via an off-chain notification service that indexer developers may subscribe to, such as:
While these channels would have several advantages like being simpler to update, faster to propagate, and easier to moderate, they have key disadvantages that make them inadequate for this SIP's intended use:
Backwards compatibility
Developers who need to emit metadata update notifications for tokens declared in older contracts (that were deployed before this notification standard was established) could do so by either calling the contract described in Reference Implementations or by first deploying a new separate contract containing a public function that prints this notification and then calling it to have it emitted.
Activation
This SIP will be activated when the following conditions are met:
If the Stacks blockchain reaches block height 170000 and the above has not happened, this SIP will be considered rejected.
Reference implementations
A reference contract has been deployed to mainnet as
SP1H6HY2ZPSFPZF6HBNADAYKQ2FJN75GHVV95YZQ.token-metadata-update-notify
. It demonstrates how to send notifications for each token class and it is available for developers to use for refreshing any existing or future token contract. If the SIP evolves to require a change to this contract pre-activation, a new one will be deployed and noted here.The Stacks Blockchain API will also add compatibility for this standard while this SIP is being considered to demonstrate how indexers can listen for and react to these notifications.