search

diadata-org / diadata

DIAdata.org platform
https://diadata.org
Other
233 stars 129 forks source link

Scraper for FTX Futures API #185

Closed kaythxbye closed 4 years ago

kaythxbye commented 5 years ago
  1. Define your data need specifications within this github issue template
  2. Fund the ticket on https://gitcoin.co
  3. Approve or reject work submitted

We are looking for a Go developer that can provide scrapers of a websocket API.

We reliably want to get futures data in realtime from FTX exchange. This trade information is then written into our databases using a specific format. We have an existing framework for this which allows us to parametrize the scraper. We work with Go and expect that the scrapers will work concurrently and efficiently. The URL of the API can be found here: https://docs.ftx.com/#websocket-api

The submitted work has to include the API scraper as well as a pull request with the respective documentation of the added code.

The DIA documentation on how to contribute can be found here: https://github.com/diadata-org/diadata

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

This issue now has a funding of 1.0 ETH (186.07 USD @ $186.07/ETH) attached to it as part of the diadata-org fund.

gitcoinbot commented 5 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Workers have applied to start work.

These users each claimed they can complete the work by 4 weeks, 1 day from now. Please review their action plans below:

1) ultraman95 has applied to start work _(Funders only: approve worker | reject worker)_.

I'm going to use the gorilla/websocket framework to scraper future data from FTX . but I have a question about "This trade information is then written into our databases using a specific format". That is to say, I also write the retrieved data to the database in a specific format. Can you explain what the specific format is?

Learn more on the Gitcoin Issue Details page.

gitcoinbot commented 4 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work has been started.

These users each claimed they can complete the work by 1 week, 3 days from now. Please review their action plans below:

1) ultraman95 has applied to start work _(Funders only: approve worker | reject worker)_.

I'm going to use the gorilla/websocket framework to scraper future data from FTX . but I have a question about "This trade information is then written into our databases using a specific format". That is to say, I also write the retrieved data to the database in a specific format. Can you explain what the specific format is? 2) nazariyv has been approved to start work.

Hi, I have completed the work on the Huobi futures exchange scraping for you. I would be happy to assist you with this task as well.

I will create a unified API for the futures markets scrapers via interfaces that each scraper will adhere to. I am planning on adding some robustness checks to the Huobi and any other scrapers that I work on that will ensure data consistency and integrity in the case of network outages and any other reasons that may disrupt the normal work of the threads.

I prefer to work on a few scrapers at a time, so I will apply for FTX and BitMex. It shouldn't take me longer than two-four days to finish them. We can then agree on the work done and I will move on to the next scraper if you wish.

Learn more on the Gitcoin Issue Details page.

nazariyv commented 4 years ago

Just to give you the heads up, I have been refactoring the Huobi bot a bit, as per the better-unified proposal. It is pretty much done. I might not be able to work tomorrow, as I am on the work trip to Amster atm.

nazariyv commented 4 years ago

looked at FTX, it will require gorilla websocket, since FTX expects a JSON message.

nazariyv commented 4 years ago

subscribed to FTX. need to unify the scrapers

kaythxbye commented 4 years ago

Nice work, I think unifying will probably also speed up other bounties in that context.

Am Fr., 22. Nov. 2019 um 19:39 Uhr schrieb Nazariy <notifications@github.com

:

subscribed to FTX. need to unify the scrapers

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/diadata-org/diadata/issues/185?email_source=notifications&email_token=AA6YCJBE75JUGDR6LRMBXJTQVARNRA5CNFSM4JKTP2R2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEE6PZBQ#issuecomment-557644934, or unsubscribe https://github.com/notifications/unsubscribe-auth/AA6YCJB5ZKDECGELJGLCJDDQVARNRANCNFSM4JKTP2RQ .

nazariyv commented 4 years ago

Thanks! I am back in London now. I will try to finish this up today in the evening. Would be good to also add a bunch of tests for this, but this is for the future scrapers that I will work on. Current scrapers are very robust anyway.

P.S. I am using the Go 1.13 built-in modules dependency mechanism for the repo where I develop these scrapers before I make a PR, and I love it so far.

nazariyv commented 4 years ago

just the heads up. I have finished FTX + refactors to Huobi. I will make the PR tomorrow.

gitcoinbot commented 4 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

Work for 1.0 ETH (146.89 USD @ $146.89/ETH) has been submitted by:

  1. @nazariyv

@kaythxbye please take a look at the submitted work:

gitcoinbot commented 4 years ago

Issue Status: 1. Open 2. Started 3. Submitted 4. Done

The funding of 1.0 ETH (151.0 USD @ $151.0/ETH) attached to this issue has been approved & issued to @nazariyv.

nazariyv commented 4 years ago

the latest work (as of 12/02/2019) is on PR: #189

nazariyv commented 4 years ago

latest work on PR: #190