[Bounty] Headless browser for fetching Proof Post

[hominluo]

Catalogue

• Platform
  • [x] Gitcoin
• Experience
  • [x] Intermediate
• Category
  • [x] Next.ID ProofService
• Time Commitment
  • [x] 2 weeks
• Bounty Size
  • [x] $3000

tl;dr: add a headless browser mechanism alongside current API-based proof post fetching channel.

Bounty Description

For now, proof_server performs proof_post fetching by API or single HTTP GET. But in real world, there may be no way for some platforms to do this way. Including:

• No APIs at all provided by platform
• API needs each user to do an OAuth to fetch their own data
• API has very strict quota and will finally exceed if proof_service users grows.
• User data cannot be fetched by single HTTP GET (e.g. JavaScript-driven XHR is performed in that page)
• HTML DOM struct is changed frequently, can't define a fixed selector to query the data from DOM tree.
• Remote HTTP server has a strict anti-bot mechanism, single HTTP GET will be blocked in a short amount of time.

To make proof_server supports more and more services later, we need a headless browser to simulate normal user's page-browsing flow.

Design goal

• The headless browser module can be kind of independent from current proof_server's code structure. With a simple wrapper, proof_server can communicate with it in different processes.
• The headless browser module will find user's proof post using single regexp, instead of parsing and css-style selecting DOM.

Winner Info

We will select up to 2 winners and each will receive at least $3000 as the winning prize.

Submission Info

Please make sure you read through our docs before starting your hacking. You will need to create a pr to this repo to submit your work.

Contact Info

@hominluo @nykma here or Telegram for any question regarding how to work on Next.ID.

[gitcoinbot]

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

---

This issue now has a funding of 1452.5352 MASK attached to it as part of the nextdotid fund.

• If you would like to work on this issue you can 'start work' on the Gitcoin Issue Details page.
  • Want to chip in? Add your own contribution here.
  • Questions? Checkout Gitcoin Help or the Gitcoin's Discord
  • $1,003,888,910.50 more funded OSS Work available on the Gitcoin Issue Explorer

[unJokers]

cool

[lyledavids]

Hi @hominluo @nykma

Is it possible to integrate this with Puppeteer? What repo must the PR be in?

[nykma]

You can make this as an independent module alongside "proof_service" . If you want to involve in, create a repo under your namespace first.

[lyledavids]

@nykma Do I just have to show the json response?

[nykma]

How to design the response API structure is up to you. Just to make sure proof_service side can get the proof post content posted by user.

[lyledavids]

Going to keep the same response structure. Just want to know if that is all that must be returned?

[lyledavids]

only json

[lyledavids]

@nykma for https://proof-service.next.id/v1/proof endpoint Always getting screen - mismatch message

Is this normal?

[tommy-a]

Hi @hominluo @nykma

Just stumbled upon this bounty and started working on it a few days ago; I'm close to finishing but don't think I'll make it in time for the submission deadline, would it be possible to submit it tomorrow, or is this a hard cutoff?

Thanks!

[lyledavids]

@tommy-a there was a hackathon extension for a extra day

[synycboom]

Hi @hominluo

I'm not sure whether or not you guys have selected the winners. Could you tell me the estimated announcement date?