TODO: replace fetching of the floor price with new method

Theres a faster way of getting floor prices I just found. When you have a collection endpoint with query params (meaning filtering is enabled), opensea exposes a __wired__ variable to the window with a lot of information, also including the floor prices. Simply run:

Object.values(__wired__.records)
  .filter(o => o.__typename === "AssetQuantityType")
  .filter(o => o.quantityInEth)
  .map(o => o.quantity / 1000000000000000000);

Some Remarks

✅ this does work on pages that have filtering enabled (BUY NOW). Example
❌ does not work on the /collection/{slug} route without query params. example
this new method improves speed because you don't need to scroll
only the top 32 offers are availible, so if you need more than that you would still need to scroll

Leaving this here for the moment because inside the __wired__ variable there is a lot of information that I currently cannot make sense of. I think there might be more hidden such that we can maybe also find the tokenId or name in this object. If someone wants to do some digging feel free @Jethro87 @robertistok @Androz2091 .

I implemented a method in the branch alternative-method-to-get-floor-prices but not sure yet whether to replace the old way of scraping that data (through the cards), or leave both methods coexist so the developer can try both functions (not ideal) or implement a fallback machanism (try method 1, if it fails try method2). https://github.com/dcts/opensea-scraper/blob/136886477220c821d24aed3f3690f60e2bce625b/src/functions/floorPrice.js#L76

Will leave open for discussion for now :)

@dcts This is really interesting. I did a bit of digging and came up with some more information. I haven't yet been able to combine the price and asset data, though.

"__typename": "CollectionType" Gives the top 100 collections by volume, as shown on the homepage

  {
    "__id": "Q29sbGVjdGlvblR5cGU6MjM5ODM2MA==",
    "__typename": "CollectionType",
    "assetCount": null,
    "imageUrl": "https://lh3.googleusercontent.com/pyNATu7ZX-iQ-6njFABGsUNfL_hQFJOSh3f5Q7Mbl3JCx-8dova1j0wbZS1xQKpL737005T7v1Hp3qwnIszi5qFiERiTz1iAhrtJ=s120",
    "name": "Fluffy Polar Bears V2",
    "slug": "fluffy-polar-bears-v2",
    "isVerified": false,
    "id": "Q29sbGVjdGlvblR5cGU6MjM5ODM2MA=="
  },

"__typename": "StringTraitType" Is the trait_type that precedes the values of each trait_value

  {
    "__id": "client:Q29sbGVjdGlvblR5cGU6MjI4NTAyMw==:stringTraits:0",
    "__typename": "StringTraitType",
    "key": "visor",
    "counts": {
      "__refs": [
        "client:Q29sbGVjdGlvblR5cGU6MjI4NTAyMw==:stringTraits:0:counts:0",
        "client:Q29sbGVjdGlvblR5cGU6MjI4NTAyMw==:stringTraits:0:counts:1",
        "..."
      ]
    }
  },

"__typename": "StringTraitCountType" Gives the count of each trait value (without the trait type)

 {
    "__id": "client:Q29sbGVjdGlvblR5cGU6MjI4NTAyMw==:stringTraits:0:counts:1",
    "__typename": "StringTraitCountType",
    "count": 70,
    "value": "rubies"
  },

"__typename": "AssetType" Gives the asset details (without price, as far as I can tell)

  {
    "__id": "QXNzZXRUeXBlOjc4NTA2Mjk0",
    "__typename": "AssetType",
    "assetContract": {
      "__ref": "QXNzZXRDb250cmFjdFR5cGU6MzQxMzE4"
    },
    "collection": {
      "__ref": "Q29sbGVjdGlvblR5cGU6MjI4NTAyMw=="
    },
    "relayId": "QXNzZXRUeXBlOjc4NTA2Mjk0",
    "tokenId": "7790",
    "backgroundColor": null,
    "imageUrl": "https://lh3.googleusercontent.com/fBbfSAtc8q-KE6ORHITNjg4l0qDYbo9PtTy6flcVNHqM4hYmidI8xHbwnC-2TibjpkoSKM-GjVyTYKARJc-XaO0uiB_mwPXkWAz0SHo",
    "name": "BOONJI #7790",
    "id": "QXNzZXRUeXBlOjc4NTA2Mjk0",
    "isDelisted": false,
    "animationUrl": null,
    "displayImageUrl": "https://lh3.googleusercontent.com/fBbfSAtc8q-KE6ORHITNjg4l0qDYbo9PtTy6flcVNHqM4hYmidI8xHbwnC-2TibjpkoSKM-GjVyTYKARJc-XaO0uiB_mwPXkWAz0SHo",
    "decimals": 0,
    "favoritesCount": 0,
    "isFavorite": false,
    "isFrozen": false,
    "hasUnlockableContent": false,
    "orderData": {
      "__ref": "client:QXNzZXRUeXBlOjc4NTA2Mjk0:orderData"
    },
    "assetEventData": {
      "__ref": "client:QXNzZXRUeXBlOjc4NTA2Mjk0:assetEventData"
    },
    "isEditable": {
      "__ref": "client:QXNzZXRUeXBlOjc4NTA2Mjk0:isEditable"
    },
    "isListable": true,
    "ownership(identity:{})": null,
    "creator": {
      "__ref": "QWNjb3VudFR5cGU6OTg2NTA2MDY="
    }
  },

Thats great, thanks for the diggin @Jethro87! Especially the "__typename": "AssetType" !!! I noticed on your comment that it has the tokenId, and it seems like its also exactely 32 obejcts, so it might be referencing to the same first 32 items than scraped with the above method. Not sure if this works but bringing the pieces together would look like this:

// get all floorPrices in ETH
const floorPrices = Object.values(__wired__.records)
  .filter(o => o.__typename === "AssetQuantityType")
  .filter(o => o.quantityInEth)
  .map(o => o.quantityInEth / 1000000000000000000);

// get additional info
const offers = Object.values(__wired__.records).filter(o => {
  return o.__typename === "AssetType" && o.tokenId;
}).map(o => {
  return {
    name: o.name,
    tokenId: o.tokenId
  };
});

// merge information together:
floorPrices.forEach((floorPrice, indx) => {
  offers[indx].floorPrice = {
    amount: floorPrice,
    currency: "ETH",
  };
});

// check offers
console.log(offers);

If I check this in my browser it seems to work, although it might be error prone because we are not binding the data specifically, just hoping that both arrays reference the same items.

So all in all if this works it implies that we now can get up to 32 offers (which I think is more than enough) without the scrolling logic, thus making the scraping significantly faster!!

@dcts Nice catch! I just manually tested this on three collections and your solution appears to work. The 32 floor prices corresponded perfectly to the 32 offers available in the __wired__ variable.

I think we should "accept" this as a faster/better solution for the time being, but I'm wary of removing the old scrolling method from the package. Who knows if/when OS will remove or change the wired variable. What do you think?

Yes totally agree @Jethro87. I am working on a V5 that switches to this new method. I had the same concern as you and thought lets keep both methods, but now I prefere to just drop the "old" method in favor of the new one. This repository should just have the best current working solution, and when it breaks, ideally someone will notice and open an issue. Also the old solution is always usable by reverting to V4.

Also I figured out how to grab the offers currency, so the offers that are in an alternative currency (like SAND or ASH) are also grabbed. Leaving the full solution here in case someone needs it prior to V5 release:

// EXAMPLE TO SCRAPE SANDBOX LAND FLOOR PRICE
// => https://opensea.io/collection/sandbox?search[sortAscending]=true&search[sortBy]=PRICE&search[stringTraits][0][name]=Type&search[stringTraits][0][values][0]=Land&search[toggles][0]=BUY_NOW

function _extractOffers(__wired__) {
  // create currency dict to extract different offer currencies
  const currencyDict = {};
  Object.values(__wired__.records)
    .filter(o => o.__typename === "AssetType")
    .filter(o => o.usdSpotPrice)
    .forEach(currency => {
      currencyDict[currency.id] = {
        id: currency.id,
        symbol: currency.symbol,
        imageUrl: currency.imageUrl,
        usdSpotPrice: currency.usdSpotPrice,
      }
  });
  // create contract dict to generate offerUrl
  const assetContractDict = {};
  Object.values(__wired__.records)
    .filter(o => o.__typename === "AssetContractType" && o.address)
    .forEach(o => {
      assetContractDict[o.id] = o.address
    })
  // get all floorPrices (all currencies)
  const floorPrices = Object.values(__wired__.records)
    .filter(o => o.__typename === "AssetQuantityType")
    .filter(o => o.quantityInEth)
    .map(o => {
      return {
        amount: o.quantity / 1000000000000000000,
        currency: currencyDict[o.asset.__ref].symbol,
      }
  });
  // get offers
  const offers = Object.values(__wired__.records)
    .filter(o => o.__typename === "AssetType" && o.tokenId)
    .map(o => {
      const assetContract = _extractAssetContract(o, assetContractDict);
      const tokenId = o.tokenId;
      const contractAndTokenIdExist = Boolean(assetContract) && Boolean(tokenId);
      return {
        name: o.name,
        tokenId: tokenId,
        assetContract: assetContract,
        offerUrl: contractAndTokenIdExist ? `https://opensea.io/assets/${assetContract}/${tokenId}` : undefined,
      };
    });
    // merge information together:
    floorPrices.forEach((floorPrice, indx) => {
      offers[indx].floorPrice = floorPrice;
    });
    return offers;
}

function _extractAssetContract(offerObj, assetContractDict) {
  try {
    return assetContractDict[offerObj.assetContract.__ref];
  } catch (err) {
    return undefined;
  }
}

_extractOffers(__wired__);

dcts / opensea-scraper

TODO: replace fetching of the floor price with new method #20

Some Remarks