hicetnunc.xyz open dataset and parsers

The project development is suspended, parsers will fail to launch on current data

Hic et nunc is a new eco-friendly NFT marketplace, built on top of Tezos blockchain.

It is especially popular in generative graphics and data viz community, so I've decided to share data and all scripts that I've made for https://hashquine.github.io/hicetnunc rating.

It is published under CC BY license, so that it is even possible to sell NFTs that use that data (or modified scripts) as long as there is the following phrase somewhere in the token description: based on @hashquine dataset.

Since hicetnunc servers are already under an extreme load due to quick growth, I've reorganized code, so that all data is taken from Tezos blockchain and IPFS without any calls to the hicetnunc.xyz website or API.

Data sources

• Blockchain transactions by TzStats API (better-call.dev was not used in order not to interfere with hicetnunc backend).
• IPFS by cloudflare-ipfs.com and ipfs.io depending on mime type (same sources as in hicetnunc frontend).
• Wallet address owner metadata (name, Twitter etc.) from api.tzkt.io (same source as in hicetnunc frontend).

What data is available

• Money data: list of all purchases, prices and commissions.
• All NFTs raw files, their previews and thumbnails, although 3d files and interactive SVG/HTML files are not yet processed properly.
• Authors metadata verified via tzkt.io like Twitter account address.
• Token transfers: list of changes of tokens owners including burns and direct transfers.
• All metadata available for tokens.
• Swaps and mints.

Data not available:

• Everything connected with hDAO tokens and hDAO feed. Although all related transactions are already being collected, they are not analysed yet.
• Twitter statistics like the number of followers.
• Direct money transfers between users, when NFT tokens are not transferred in the same transaction.

Dataset schema

The goal was to simplify data analysis and visualization with a wide range of existing tools, so there are lots of redundant fields, which contain precalculated aggregations and different representations of the same data.

All files have two equivalent versions: JSON and CSV.

• JSON files are dictionary of dictionaries with rows of CSV files are indexed by the *_id field.
• CSV files have commas as delimiters.
• Fields values are ether numbers or strings, empty values represented by -1 or "".
• All identifiers are strings.

Any field, which references some event in the blockchain (for example, mint time) have 4 representations:

• mint_iso_date — string with UTC date and time: "2021-03-01T15:00:00Z",
• mint_stamp — integer Unix timestamp in seconds: 1614610800,
• mint_hash — string with transaction hash, where event occurred: "oom5Ju6X9nYpBCi...",
• mint_row_id — integer with global unique operation id (internal to TzStats) with that event: 42181049

Any field, which references a set of values (like the set of prices of sold works), have following aggregations:

• sold_count — values count,
• sold_nonzero_count — number of positive values,
• sold_zero_count — number of zeros,
• sold_price_min — minimum value (excl. zeros),
• sold_price_max — maximum value,
• sold_price_sum — sum of values,
• sold_price_avg — average value (sum divided by count excl. zeros).

tokens.json and tokens.csv — of all NFTs tokens

There is a confusing fact, that in hicetnunc each NFT can have multiple identical instances, which are fungible. In this document term "token" refers to the set of all that instances.

There are following invariants:

mint_count = author_owns_count + available_count + available_zero_count + other_own_count + burn_count
author_sent_count <= other_own_count + available_count + available_zero_count

addrs.json and addrs.csv — of all hicetnunc users

All users, who ever created or owned NFT token.

sells.json and sells.csv — of all purchases via "official" hicetnunc swaps

There is the following invariant:

price * count = total_royalties + total_comission + total_seller_income

transfers.json and transfers.csv — all token transfers

swaps.json and swaps.csv — all "official" hicetnunc swaps ever created

Essential information about contracts logic

There are 3 Tezos addresses, which are common to most of hicetnunc transactions:

• "NFT" contract: KT1RJ6PbjHpwc3M5rw5s2Nbmefwbuwbdxton.
  • This is the registry of all owners of all NFT tokens.
    • This is the typical way how most of NFTs works on Ethereum or on Tezos.
    • Click on "Bigmap #511" tab in Tezos explorer to see registry of token owners.
    • This contract is the single "source of truth" about current owners of all NFT tokens issued by hicetnunc. If there is no information about token owner in the registry, than that person doesn't own any tokens.
  • This contract also the registry of tokens infos (metadata).
    • Token info is a small JSON structure stored on on IPFS.
      • Here is an example of such structure.
      • It contains link to IPFS with NFT binary contents (some image, for example).
      • It also contains title, description, creator and tags.
      • It does not contain price or related information.
    • Only link to IPFS is stored on the blockchain.
    • Note, that, however, there is no way to alter token metadata after minting.
    • Click on "Bigmap #514" tab in Tezos explorer to see mapping from tokens to IPFS urls.
  • Every token owner can call "transfer" method of the contract to send tokens to other address.
    • This contract can't do any money related operations. Money logic should be implemented in other contracts, which call "NFT" contract as a part of transaction operation.
  • There is also a "mint" method in this contract, but it can only be called by the "Art house" contract.
• "Art house" contract: KT1Hkg5qeNhfwpKW4fXvq7HGZB9z2EnmCCA9.
  • This contract implements money related operations on hicetnunc.
  • It's main structure is a swap. It is some amount of tokens, which are available of sale for specific price.
    • Click on "Bigmap #523" tab in Tezos explorer to see all current swaps.
  • Note, that there may be other contracts implementing swap mechanism. These contracts may decide not to pay comission or royalties.
  • Objects can be minted only with this contract by calling "mint_OBJKT" method.
    • This contract keeps track of royalties and assigns tokens ids.
• Comission wallet: tz1UBZUkXpKGhYsP5KtzDNqLLchwF4uHrGjw.
  • 2.5% of every purchase via "Art house" contract swaps is sent to this wallet.

There are several other contracts related to curation and hDAO mechanisms, which are independent from the contracts mentioned above.

Actually, hicetnunc was not created just as another NFT marketplace, it has much broader mission as hDAO (hicetnunc DAO). You can get the idea of the creators vision on hicetnunc blog. As a result, only a small subset of contract's logic is actually used during hicetnunc website operation.

Official and external swap mechanisms

• Official swap mechanism by hicetnunc "Art house" contract.

  • Any token bought on hicetnunc website is a part of some swap.
  • Swap is just some amount of tokens, which are offered for sale by specific price.
  • When swap is created, the seller sends all offered tokens to the "Art house" smart contract.
  • Then anybody can send the required amount of money to the "collect" method of contract and get tokens in return.
    • 2.5% of comission is transferred to hicetnunc comission wallet.
    • 10% of royalties (this parameter is configurable in general) is transferred to token author.
    • rest of money is sent to swap creator.
    • proportional amount of hDAO tokens are also sent to buyer, seller, token author and comission wallet.
  • Seller can cancel swap any time and get unsold tokens back.
  • Swaps can be created by any token owner any number of times.
  • In this dataset official swaps are treated as author->swap, other->swap, swap->author, swap->other transfers.

• External swap mechanisms.

  • Since "transfer" method of "NFT" contract can be called by any token owner directly, it is possible to make custom smart contracts, which implement any desired logic.
  • These custom contracts are not required not pay comission or royalties to hicetnunc.
  • In general, swap contracts can be used to exchange any types entities.
  • Example: https://quipuswap.com/swap
  • In this dataset external swaps are treated as other->other transfers. The related price is guessed heuristically (as half of money transferred in all operations) and may not be always correct.

Token lifecycle

In contrast to NFT definition, each NFT artwork in hicetnunc can have multiple copies, which are fungible. The NFT contract only tracks the amount of copies owned by each address. This means, that there is no way (even in theory) to track history of single copy like it can be done on OpenSea, for example.

It is possible, however, to track history of token groups to some extent. Here is a list of possible owner types in this dataset:

• author — the person, who created the tokens during mint.
• user — any other hicetnunc user.
• ext — any external contract (external swap mechanism, for example).
• burn — reserved address for burning tokens.
• swap — when tokens are offered on sale in official swaps.

List of possible transitions:

• mint->author, mint->user First, every token should be minted.
  • For each token type there may exist only single mint operation. It is impossible to mint additional tokens later.
  • The only way to mint a token is to call "mint_OBJKT" method in "Art house" contract.
  • Here is a typical mint transaction
  • Internally it calls "mint" method in "NFT" contract.
  • In dataset the sender is empty for mint operations.
  • As result of mint operation, all tokens are transferred to some address. In 99% of cases this is the transaction sender, but sometimes it is different.
  • Royalties are always sent to the mint transaction sender.
• author->swap, user->swap Any token owner can create official swap.
  • Hicetnunc swap is created by calling "swap" method in "Art house" contract.
  • Here is a typical swap creation transaction by author.
  • Internally tokens are transferred to the "Art house" address.
• swap->author, swap->other There are two situations, when tokens may be transferred from a swap.
  1. Purchase
     • When token is purchased on hicetnunc website, it is transferred to the buyer. This is the main operation on hicetnunc.
     • Buyer should call "collect" method of "Art house" contract and send required amount of money with it.
     • Here is an example of "collect" transaction.
       • First 3 internal operations send money to token creator (royalties), hicetnunc wallet (comission) and to the seller (which is the same as token creator in some cases) in that order.
       • Fourth operation creates hDAO tokens and sends them to the buyer, seller and hicetnunc wallet. These tokens have special meaning and are not tracked in this dataset.
       • Last internal operation does the actual token transfer.
     • Note, that case of zero price is handled differently.
       • Here is an example of purchase with zero price.
  2. Swap cancel
     • When swap creator decides to cancel swap, all remaining tokens are transferred back to him.
• author->user, user->user Any token owner can transfer tokens directly to other users for free by calling "transfer" method of "NFT" contract.
  • Here is an example of direct transfer transaction from author to other user.
• author->ext, user->ext, ext->user, ext->author — external swaps
  • Here is an example of external swap.
• author->burn, other->burn Any token owner can transfer tokens to burn address tz1burnburnburnburnburnburnburjAYjjX.
  • Here is and example of burn transfer from author.
  • Tokens can never be transferred from burn address since it is impossible to retrieve its private key (similar to how it is impossible to reverse hash containing all zeros).

Details about edge cases

How to define the author of the token

1. mint_sender The address of the sender of the "mint" transaction.
   • This is the person, who receives royalties in hicetnunc.
   • In this dataset token author this is equivalent to token author.
2. issuer The address of the receiver of tokens after the mint transaction.
   • It is also the first parameter of the "mint" call in "Art house" contract.
   • Here is an example of mint, where transaction sender and token issuer are different.
3. info_creator Field "creators" in JSON in token metadata.
   • Here is an example of mint, where metadata creator field is different from transaction sender and issuer.
   • As of 4th of April, it always has single entry.
   • Sometimes it is empty.
     • Here is an example of mint with empty metadata creator field.
     • Note, that corresponding token page has a bug, that it shows token owner controls on token page.

Hicetnunc core addresses can own NFTs as regular users

• Any user can send any NFT tokens to "NFT" or "Art house contract"
  • Technically, it has the same effect as sending this tokens to burn address, since contracts were not programmed to send their own NFTs (except from swap mechanism) under any circumstances.
• Comission wallet sometimes mint NFTs and buys them from other users.
  • Since it is not a contract and manipulated by a real person (hicetnunc creator).

Void transactions

• It is possible to send 0 tokens. Example.
• Sender and receiver can be the same. Example

Strange issue with negative swap count:

• https://tzstats.com/opYoTN4LUvNq7F5oiq7a6frW1Q5UubQuXSrDmrqrJGJzHnKaJQQ/47335724
• https://tzstats.com/oo7Sr2daVXamKTDtajRRvJGmTmYAantjJeLoPkTD6hraVHXQR1B/47335862

How to update dataset

Note, that the code is still experimental and may require substantial changes to make it run. But you can read introduction in HACKING.md