adding testnet

[jack75015]

Hey What could be nice is adding testnet option to scrap opensea testnet website :)

[dcts]

Is it this one: https://testnets.opensea.io

Or are there more than 1 sotes?

[jack75015]

yep this one they use Rinkeby eth network

[dcts]

@jack75015 is this still relevant to you? Because I tested and you can simply do this (as a "hack"):

// TESTNET
const res = await OpenseaScraper.offersByUrl("https://testnets.opensea.io/collection/cyberbrokers-v3",options)
// => this gives you testnet results

// MAINNET
const res2 = await OpenseaScraper.offers("cyberbrokers-v3",options);
// => this gives you production results (empty) 

[dcts]

closing as testnet scraping is already supported when using offersByUrl method.

[0xMarsRover]

Hi dcts,

I tried to use the same code to get results from opensea testnet, but it failed with the error message below.

It seems that the page is not actually scrapered so that there is no "html" content for further use. Any suggestions for that?

Thanks. Kai

[dcts]

@kaiqiangh I could replicate the issue, the problem is the following:

• the testnet sites usually have a weird page that is displayed in between the actual result page
• just waiting for ~2 seconds before starting the scraping soloves the issue
• unfortunately there is no feature implemented to wait before scraping

I am working on a feature additionalWait where you can simply add to the options the number of milliseconds to wait before starting to scrape, which should fix the issue.

will post soon once the feature is published on npm.

[0xMarsRover]

Thank you very much!

[dcts]

new version with additionalWait just published (opensea-scraper v6.5.0)

You should be able to overcome this issue like this:

1. upgrade to OpenseaScraper v6.5.0 (by running npm install opensea-scraper@6.5.0) EDIT: bugfix added, please update to 6.5.1 instead: npm install opensea-scraper@6.5.1.
2. add an additionalWait of ~2-4 seconds:

   // TESTNET
   const options = {
   additionalWait: 4000, // 4 seconds should work, you can try lowering that number
   }
   const res = await OpenseaScraper.offersByUrl("https://testnets.opensea.io/collection/cyberbrokers-v3",options)
   // => this gives you testnet results

let me know if it worked :)

[0xMarsRover]

Hi @dcts,

Much appreciated for your quick update.

It works, but there is a minor bug.

the offerURL should be llike "https://testnets.opensea.io/assets/rinkeby/NftAddress/nftID".

[dcts]

Good catch, just fixed it! Please update to newest version:

npm install opensea-scraper@6.5.1

[0xMarsRover]

Thanks! It perfectly works now.

[dcts]

great! Closing...!