search

Livepeer-Community-Node / Grant-Program

Livepeer Community Transcoder Node's Grant Program
10 stars 5 forks source link

Unified Livepeer API #19

Open rbcp18 opened 5 years ago

rbcp18 commented 5 years ago

Give a 3 sentence description about this proposal. This proposal looks to build a unified API for Livepeer. The API will act as a point of contact for Livepeer developers to perform queries, connect to websockets, run smart contract functions, and interact with nodes, similar to the TzScan API (https://tzscan.io/api#tag/Bakings-Requests) and Cosmos API (https://cosmos.network/rpc/#/ICS0/get_node_info).

Describe the problem you are solving. For development of many web3 applications - from analysis tools to protocol-level applications - developers turn to APIs to begin their build out. Currently, only a few options exist for developers to integrate the Livepeer platform. These options are (1) running your own node, (2) interacting with the Livepeer JS SDK, and (3) scanning data from available Livepeer tools. While one can privately run a node, gather data, and build out functionality themselves, this process significantly increases development time for the integration of Livepeer services.

Describe the solution you are proposing. With the "Unified Livepeer API", developers will interact with a single interface to query data, subscribe to events, run smart contract based functions (bonding & delegation), broadcast to a specific node, and perform transcoder interactions.

The API will consist of two layers:

Layer 1: Data and WebSockets

Layer 1 will integrate all major Livepeer tools and access points. Platforms which are planned for integration include: the Livepeer SDK (for GET request functionality), Livepeer subgraph (for queries on transcoders, rewards, and rounds, and websockets on events), Livepeer studio (for staking alerts), Scout.cool (if possible, for mainnet market data), and OpenMarketCap (or other token data feeds).

In addition, custom data-based API calls will be integrated, such as (1) retrieval of all Livepeer smart contracts on mainnet, (2) JSON data of transcoders (by address) with their campaign information (name, team, about), and (3) events from specific LPT exchanges (eg. Poloniex, Uniswap, IDEX) and wallets (eg. 0x6941627cba3518385e75de75d25a189185672bfe, which holds 6.5% of the network).

Layer 2: Nodes and Smart Contracts

Layer 2 will constitute the launch of full nodes running on both Mainnet and Ropsten with HTTP access. This will enable developers to immediately start building on a Livepeer node, without going through the potential roadblocks faced when launching a private node. In addition, this will enable broadcasters to use the Testnet nodes (and Mainnet nodes, if desired) as a way to quickly get up and running with the OBS integration.

Alongside the node integration, all major smart contract actions, including bonding and delegation functions, will be directly integrated into the API. To do so, the API will utilize the Livepeer Minter contract and other available contracts. This smart contract functionality will be integrated with web3 wallet support, initially with Metamask and, subsequently, with Fortmatic.

Describe the scope of the project including a rough timeline and milestones

Layer 1: Data and WebSockets, 3 weeks

Week 0 - 1.5: (1) Main network data, (2) alerts, and (3) market information integrations. Integrated platforms will be the Livepeer SDK, Livepeer subgraph, OpenMarketCap (pricing and market activity data feeds), Livepeer Studio, and Scout.cool (if possible).

Week 1.5 - 3: (1) Custom data integrations, specifically API calls around smart contracts, transcoders, and broadcasters. (2) Webscoket functionality on transcoder, reward, and round events. (3) DevOps function testing on API system infrastructure.

Layer 2: Nodes and Smart Contracts, 3 weeks

Week 3 - 4.5: (1) Node integration on Testnet (Ropsten) and Mainnet. (2) Node function testing for developer and broadcaster usage. (3) Maintenance set up for node availability and uptime.

Week 4.5 - 6: (1) Contract integration, focused on delegation and bonding. (2) Unit testing of contract functions on Testnet and Mainnet. (3) Supplementary contract exploration and integration (eg. Websockets based on LPT token contract events). (4) Web3 wallet integration for Metamask and Fortmatic.

Team We are the guys behind Fabrx (www.fabrx.io)

Please estimate hours spent on project based on the above 20 hours per week for 6 weeks

dob commented 5 years ago

Nice. There are lots of requests for a REST based API to interact with the LP protocol and to fetch data, so I'm sure there's a user base for this that would be excited to see it exist.

One thing I wonder about is the sustainability of these sorts of services. Would you be hosting this API yourself, and offering some sort of loose SLA on performance, scale, and reliability? Would users be paying to access it? If they're not paying, and it goes down, what's the incentive to fix it and get it up and running quickly?

rbcp18 commented 5 years ago

Hi @dob,

Glad to hear that you've also seen demand for a Livepeer REST based API. To answer your questions:

  1. Yes, we will be hosting the API ourselves. We propose to offer a loose SLA on those metrics, utilizing an agreement similar to the Twilio SLA (https://www.twilio.com/legal/service-level-agreement).

  2. Access to the API will be free.

  3. To align our incentives, we've explored several revenue generating models:

We would currently need ~ 0.5% of the network to become an active transcoder. ~ 50% of circulating supply is not yet delegated (based on https://stakingrewards.com/asset/lpt).

Overall, we believe that solution (a), launching our own transcoder, is most ideal for both the API platform and ecosystem at large.

dob commented 5 years ago

Great. I like that you're thinking in terms of crypto-native models. I think a combined with b seems reasonable since it might be overly complicated to enforce rules around scaled applications at with high usage, who would likely be happy to pay anyway for use beyond certain limits. Tagging @yondonfu, @adamsoffer, @iameli as well who may have some input on a unified API on top of the smart contracts, the graph, etc.

RaffiSapire commented 4 years ago

Thanks @rbcp18 - please keep us posted on the first milestone once its completed, share back a link or update here! I think many community members would be excited about this and happy to stake towards this transcoder node. I've also pinged Adam, Yondon and Eli to provide their feedback.

rbcp18 commented 4 years ago

Update from the Fabrx team - the first milestone of the Livepeer API is live! Version 1.0.0 unifies various Livepeer access points, including the Livepeer subgraph for onchain data, Livepeer SDK for direct functionality, Livepeer.Studio for transcoder alerts, and Livepeer token (LPT) for market data.

Version 1.0.0 of the Livepeer API can be found here: https://app.swaggerhub.com/apis-docs/rfabrx/livepeer-api/1.0.0

adamsoffer commented 4 years ago

@rbcp18 this is amazing! Awesome work. One thing I want to point out is that there is a long-running PR with a bunch of updates to the Livepeer Subgraph that I'm hoping to eventually get merged and deployed in the near future. It introduces a few new entities such as Shares, Delegators, and UnbondingLocks and some breaking changes. I'm in the process of testing it on The Graph's hosted explorer under the name adamsoffer/livepeer (the deployment is pending) but I just want to get that on your radar since this would introduce breaking changes to your REST API when we update the official Livepeer Subgraph.

Sidenote: This is out of scope, but it would be great to offer a unified GraphQL API down the line in addition to REST - something that combines all the disparate Livepeer related GraphQL schemas. Currently, the onus is on the developer to combine the Livepeer Subgraph, the graphql-sdk, and any other relevant schemas such as 3box and the ENS subgraph. Apollo just released a new feature called "Federation" that would make something like this fairly trivial. Something to think about!

iameli commented 4 years ago

Looks like a lot of solid work so far! I'm absolutely in favor of developing REST APIs and OpenAPI schemas for as much of the Livepeer ecosystem as possible. A couple thoughts:

@adamsoffer:

This is out of scope, but it would be great to offer a unified GraphQL API down the line in addition to REST - something that combines all the disparate Livepeer related GraphQL schemas. Currently, the onus is on the developer to combine the Livepeer Subgraph, the graphql-sdk, and any other relevant schemas such as 3box and the ENS subgraph. Apollo just released a new feature called "Federation" that would make something like this fairly trivial. Something to think about!

Honestly, I'd sooner deprecate @livepeer/graphql-sdk in favor of @livepeer/sdk for contract interactions and the subgraph for read/data operations. The GraphQL mutators don't accomplish much beyond being wrappers around the SDK, and the subgraph provides a much more consistent data backend. But we'll talk more about it this week 😀

rbcp18 commented 4 years ago

@adamsoffer Thanks for the heads up on the Livepeer Subgraph. The new editions sound exciting! In regards to a unified GraphQL API, that would be awesome. I love the idea of expanding into GraphQL and combining various schemas and contract data into one, easy to run interface.

@iameli Glad to hear that we’re thinking along the same lines. To answer your questions:

RaffiSapire commented 4 years ago

To provide a grant amount, it would be helpful for you to scope out the amount of time it will take you. @rbcp18 . I see you have a # of weeks, but not clear to us if it would take you a full 40 hour work week for each milestone. Understanding the true time estimates would be helpful here. If you could do that, I can pretty easily gut check with engineers in the community on timing estimates, and provide a grant amount! thanks and so excited to help get this going!!

rbcp18 commented 4 years ago

@RaffiSapire That is exciting!! We have mapped out a total of 120 man-hours for the project, 20 hours per week for 6 weeks (same as described at the bottom of the original proposal). This would average out to be ~ 30 hours per milestone (1.5 weeks each). Let us know anything else you’d need from us to get this going 😀

RaffiSapire commented 4 years ago

Approved, granting 1900 LPT, disbursed by milestone. Please post progress for each milestone when its complete. If you need help, Eli or Adam can help you on Discord. Could we organize a demo for the community at the end of the 6 weeks, and maybe brainstorm ways to get people to use the API some time this month? @rbcp18

rbcp18 commented 4 years ago

@RaffiSapire Awesome!! We're pumped to build the unified api for the ecosystem! We'll be back soon with the first milestone.

We'd love to organize a demo for the community, maybe the Week of September 16th?

And definitely, it would be great to brainstorm ways to get people to use the API. Does sometime next week or the following week work for you? Happy to coordinate through email (ryan@fabrx.io).

rbcp18 commented 4 years ago

Hi @RaffiSapire

Update from the Fabrx team - the first milestone of the Livepeer API is complete (with the open sourced GitHub repo below)! We host the API on AWS and have tested each endpoint. The repo includes both the Python and Node servers which we run. We have simplified the repo, such that a developer can easily run both servers from their local machine.

We’d love to get feedback from the community on the API and how it can be improved.

Version 1.0.0 of the Livepeer API can be found here: https://app.swaggerhub.com/apis-docs/rfabrx/livepeer-api/1.0.0. The GitHub repository can be found here: https://github.com/rbcp18/livepeer-api.

For the grant proceeds, you can send the LPT to this address: 0x4d65151cD05f43F9aCbEdBe4182b02445A93D7CF. We look forward to brainstorming use cases of the API with you!

adamsoffer commented 4 years ago

@rbcp18 this looks awesome! Thanks for this great contribution :) A few questions:

Great work!

rbcp18 commented 4 years ago

Hi @adamsoffer

Thanks for the love! To answer your questions:

Out of curiosity, what was the thinking behind exposing some endpoints in Node (the livepeer sdk), and others (the subgraph/coingecko) in python?

Could you provide instructions on how one might go about spinning up the unified REST endpoint locally in the README?

How/where are you exposing the staking alerts endpoint? I didn’t see that one in the GitHub repo.

Thanks again for the feedback, and helping to push this project forward - cheers!

chrishobcroft commented 4 years ago

Nice work. Do you have plans to add Livepeer to fabrx.io homepage?

image

rbcp18 commented 4 years ago

@chrishobcroft done! Livepeer is live on the fabrx.io homepage :)

rbcp18 commented 4 years ago

Hi all,

As part of milestone 2, we have launched a .json file to list all available transcoder campaigns (name, address, team, about). We will be updating the file with the campaigns available in https://forum.livepeer.org/. If you have info on a transcoder which is not on the list, please add it to the list on the .json file by submitting a PR 😀

We hope to use this file as a main Livepeer Transcoder Registry.

The .json file can be found here: https://github.com/rbcp18/livepeer-api/blob/master/livepeer-api-open-source/src-python/resources/json_files/livepeer_transcoder_campaigns.json.

In addition, the API has been updated to v1.0.1 (https://app.swaggerhub.com/apis-docs/rfabrx/livepeer-api/1.0.1#/). The update contains both the GET all_transcoder_campaign_info and GET all_contract_addresses calls).

rbcp18 commented 4 years ago

Hi @RaffiSapire Update from the Fabrx team - the second milestone of the Livepeer API is complete!

  1. Custom data integrations are publicly available!
  2. Events from specific LPT exchanges and wallets.
  3. Websocket functionality on transcoder, reward, and round events.
    • The Livepeer Websocket is live! Developers can subscribe to the Websocket here:
    • The Websocket returns events on:
      • Transcoders
        • Subscribe to a specific transcoder and receive per block info on:
          • Active Status
          • Address
          • Fee Share
          • Last Reward Round
          • Price Per Segment
          • Pending Reward Cut
          • Pending Fee Share
          • Reward Cut
          • Status
          • Total Stake
      • Rewards
        • Get reward projections (approximations) per block for a specific transcoder on:
          • Total LPT Staked to Transcoder
          • Inflation Per Round Percent
          • Reward Cut Percent
          • Transcoder Per Round Rewards
          • User Per Round Rewards
          • Total Per Round Rewards
      • Rounds
        • Get round info per block on:
          • Round ID
          • Round Initialized
          • Last Initialized Round
          • Length
          • Start Block
  4. DevOps function testing on API system infrastructure.
    • We’ve added AWS CloudWatch and ran tests on the API to verify performance.

We’d love to get feedback from the community on the API and how it can be improved.

Version 1.0.1 of the Livepeer API can be found here: https://app.swaggerhub.com/apis-docs/rfabrx/livepeer-api/1.0.1.

The GitHub repository can be found here: https://github.com/rbcp18/livepeer-api.

For the grant proceeds, you can send the LPT to this address:

0x4d65151cD05f43F9aCbEdBe4182b02445A93D7CF

adamsoffer commented 4 years ago

@rbcp18 So much data! The WebSocket support looks great. Love the idea of seeing realtime per-block reward projections.

Re: the exchange data API, some of us are big fans of uniswap; any chance we can add that to the exchange list as well? Looks like it has an exchange api.

Thank you for the awesome update 🙏

rbcp18 commented 4 years ago

@adamsoffer thanks again for the love! Uniswap sounds like an awesome addition - we'd be happy to integrate the data feed 😀

RaffiSapire commented 4 years ago

Hey @rbcp18 I just disbursed LPT for M 1 and M 2 here https://etherscan.io/tx/0x6758ee54b6f7f1f2625f7095081ecfba35b7695669779fc5d21f19c24821e7ee

Let us know when you think M3 will be delivered, please note - with streamflow coming up, there are couple changes that will need to be made to the API. After Yondon releases documentation for interacting with Streamflow on testnet in the next week or so, that is a good time to play around with the new contracts.

dob commented 4 years ago

@rbcp18 Hey, we've heard some reports of the fabrx API being down. Any update?

rbcp18 commented 4 years ago

Hey @dob it appears one of the node modules is throwing an error, causing the express app to not recognize methods - we're looking into it. The rest of the api appears to be running. Will report back once the bug is patched 👍

adamsoffer commented 4 years ago

@rbcp18 - if you're using the livepeerjs sdk, it might be related to this issue. If so, just make sure you pass in a default gas value to the sdk.

rbcp18 commented 4 years ago

@adamsoffer This was exactly the issue. Thanks for help! @dob The API is back up and github project updated 😀