candle limit in get history

[mikepumpkineater]

Hello, ftx has a limit on the number of candles returned when receiving a history. In your package, the function "ftx_get_hourly_markets" solves this problem, I would like to know how you can solve the same problem for other time periods of candles, for example for 15 minute candles?

[andreskull]

Good point! As I do not need other candles at the moment with my trades, it is as it is. I admit it should be implemented at some time, but I cannot give any ETA.

Could you come up with the implementation either implementing a similar function to 'ftx_get_hourly_markets' or, preferably, a general function where the candle id is an input parameter? I'm happy to merge such an improvement.

[mikepumpkineater]

I will try to think over and write the logic for different timeframes. If everything goes well, I will add the code to this branch

[mikepumpkineater]

Hello, The API documentation indicates that the following window length options are valid: (window length in seconds. options: 60, 300, 900, 3600, 14400, 86400). For each window value, I calculated the number of days that fall into such a window, taking into account the limit of 1500 values, and rounded down. These values are constant.

# ftx rest api return only 1500 candles in one require
# 1500 daily candles -> 1500 days (86400)
# 1500 4 hour candles -> 250 days (14400)
# 1500 hourly candles -> 62.5 days (3600)
# 1500 15 min candles -> 15.625 days (900)
# 1500 5 min candles -> 5.2 days (300)
# 1500 1 min candles -> 1.04 days (60)

I rewrote your function 'ftx_get_hourly_markets()' so that it would determine the time periods by which the distance from start_date to end_date should be divided, for the subsequent sequential request to the api.

ftx_get_history_all <- function(markets, 
                                start_date, resolution,
                                end_date) {

  res_values <- c(60, 300, 900, 3600, 14400, 86400)

  if(resolution %in% res_values & start_date < end_date){
    delta <- ifelse(as.integer(resolution) == 60, 1, 
                    ifelse(as.integer(resolution) == 300, 5, 
                           ifelse(as.integer(resolution) == 900, 15, 
                                  ifelse(as.integer(resolution) == 3600, 62, 
                                         ifelse(as.integer(resolution) == 14400, 250, 
                                                ifelse(as.integer(resolution) == 86400, 1500, F))))))

    if(delta != F){
      dates <- seq(from = as.Date(start_date), to = as.Date(end_date), delta)
      if(length(dates) > 1){
        tb <- NULL
        for (i in 2:length(dates)) {
          q <- data.table(from = dates[i-1], to = dates[i])
          tb <- rbind(tb, q)
        }
      } else {
        tb = data.table(from = start_date, to = end_date)
      }

      if(tb$to[nrow(tb)] != end_date){
        last_row <- data.table(from = tb$to[nrow(tb)], to = end_date)
        tb <- rbind(tb, last_row)
      }else tb = tb
    }

    prices <- apply(tb, 1, function(x) {
      ftx_get_multiple_markets(markets, resolution, start_time = x[1], end_time = x[2])
    } )
    prices <- prices %>% 
      dplyr::bind_rows() %>% 
      dplyr::distinct()
    prices <- prices %>% 
      dplyr::select(ticker = .data$ticker, date = .data$start_time, open, high, low, close, volume)
    prices
  }else{
    logerror(msg = 'Unsupported candle resolution value. Supported values are 15, 60, 300, 900, 3600, 14400, 86400 or start date >= end date')
  }
}

To test for missing values, I wrote the next simple code, in which we create a vector of time intervals from start_date to end_date with the interval necessary for checking and compare it with the vector of dates obtained using our function. The test is successful if, when merged, the test vector and data vector are identical and in the same length.

x <- ftx_get_history_all(markets = 'BTC/USD', start_date = Sys.Date()-100, end_date = Sys.Date(), resolution = 300)

test_vec <- seq(ymd_hm('2022-07-26 00:00'),ymd_hm('2022-11-03 00:00'), by = '5 mins')
test_vec <- data.table(date = test_vec)
q <- merge(x = test_vec, y = x, by = 'date', all = T)
nrow(x) == nrow(test_vec)
nrow(q[!na.omit(q)] ) == 0

PS: I might have missed some condition checks in the code, I hope together we can finalize this point.

[andreskull]

At first glance, it looks nice! Could you write a test for it to check that there won't be any missing data points? Another thing is to test the case when start_date = end_date.

[mikepumpkineater]

I've updated my previous post and made the necessary corrections to the code and added a missing value test. Tried the test in different modes

[andreskull]

@mikepumpkineater I merged your code with little modifications. Changed the function name and parameters order. See here https://github.com/andreskull/rFTX/commit/2aa510dc61043ba58fd580806ffb01f53c32668a