search

diadata-org / diadata

DIAdata.org platform
https://diadata.org
Other
233 stars 129 forks source link

Write a Trades Scraper for Sorare NFTs #393

Closed jppade closed 3 years ago

jppade commented 3 years ago

We are looking for a Go developer who is experienced in writing applications interfacing with the Ethereum blockchain. Basic knowledge in NFTs is helpful but not necessary. Your task is to write a trades scraper for the Sorare NFTs. The data must be fetched directly from the blockchain (using for instance infura API). The structure for the scraper is already laid out. Your job is basically to implement the scraping logic in: https://github.com/diadata-org/diadata/blob/master/internal/pkg/nftTrade-scrapers/sorare.go The structure is just a suggestion. Feel free to extend/improve it. Any questions don't hesitate to ask!

gitcoinbot commented 3 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 350.0 USDC (350.0 USD @ $1.0/USDC) attached to it as part of the diadata-org fund.

gitcoinbot commented 3 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 265 years, 6 months from now. Please review their action plans below:

1) controlcpluscontrolv has been approved to start work.

I've worked with you before on NFT's I would be more than happy to do this project -I've done many webscrapers before, this could also be done much easier in python if you're open to that, but if not I can use Go. You already laid out the work so this wouldn't take too long to implement and I would be happy to do it for you! 2) hhio618 has applied to start work _(Funders only: approve worker | reject worker)_.

I already done some of Diadata tasks and I'm familiar with the code. 3) dmcclung has applied to start work _(Funders only: approve worker | reject worker)_.

Seems very straightforward, I am very interested in knocking this out for you!

Learn more on the Gitcoin Issue Details page.

ControlCplusControlV commented 3 years ago

I expect this will take 2 more days (1 day to build the scraper of the Ethereum blockchain, and a day for testing), right now I have a data scraper that gets its info from from https://coinranking.com/dapp/sorare/tokens API, that gets contract addresses and basic info about them. I then plan on using this data to query the Infura API to get Ethereum blockchain data on each contract, and detect if a transaction relating to it is in a recent block as well as build a history of all past transactions for each NFT. Let me know if you have any questions or concerns!

jppade commented 3 years ago

@ControlCplusControlV Thanks for the update! That sounds perfect. If you're already done with the data scraper fetching the addresses that's cool. Otherwise, we can also fetch these from our database later on. So you can focus on how to get the transaction data from on-chain.

ControlCplusControlV commented 3 years ago

@jppade oh ok, I am already done, at least this way when they release new cards it should still be able to track those new cards. Moving onto the on-chain data right now, working through how to fetch all past transaction history, then scan blocks as they are created. Is there any output format you prefer for sending into your database? Was planning on exporting a 2d array, for each contract, containing tx hash of their trade and a blocknumber (I know thats implied in the hash but makes it so much easier for sorting I'd imagine), wasn't sure if you wanted a USD/ETH value in there too

Edit - Also wanted to ask where you were on doing this in python, its like half a day to complete if so (Python libraries hugely supportive of parsing javascript responses, and querying the blockchain). Otherwise I can continue doing it on Go along the expected timeline.

jppade commented 3 years ago

@ControlCplusControlV The output format should be given by the struct dia.NFTTrade in the template. Although somewhat redundant, I think it's a good idea to extend it by the tx hash. I am aware that Python makes your life easier at some points, but we really prefer Go. By the way, in order to minimize your work load and clear out quick questions in between you can also contact me on telegram. My handle over there is the same as here.

gitcoinbot commented 3 years ago

@controlcpluscontrolv Hello from Gitcoin Core - are you still working on this issue? Please submit a WIP PR or comment back within the next 3 days or you will be removed from this ticket and it will be returned to an ‘Open’ status. Please let us know if you have questions!

Funders only: Snooze warnings for 1 day | 3 days | 5 days | 10 days | 100 days

ControlCplusControlV commented 3 years ago

Still working on this, in contact with @jppade on how to proceed and work through this

gitcoinbot commented 3 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 350.0 USDC (350.0 USD @ $1.0/USDC) has been submitted by: