How to request historical data multiple times?

I'm a bit confused on this one and can't seem to get a working example.

I create a historical data request for the SPY ticker, 1 day's worth of the latest 5 minute bars. For this simple example, I want to sleep for 5 minutes, then request historical data again. Right now, all it does is just show the bars pulled from the initial request, it doesn't seem to be updating at all.

To recreate this example, I have made the following modifications to the wrapper.go file:

var GlobalHistoricalDataSlice []types.TOHLCV

func GetHistoricalData() []types.TOHLCV {
    return GlobalHistoricalDataSlice
}

func (w Wrapper) HistoricalData(reqID int64, bar *BarData) {
    log.With(zap.Int64("reqID", reqID)).Info("<HistoricalData>",
        zap.Any("bar", bar),
    )
    // Append the bar data to the GlobalHistoricalDataSlice. Bar data is in this format:
    // {"level":"info","ts":1704830870.0692365,"caller":"ibapi2/wrapper.go:353","msg":"<HistoricalData>","reqID":1,"bar":"BarData<Date: 1704800100, Open: 472.910000, High: 472.910000, Low: 472.800000, Close: 472.800000, Volume: 5.000000, Average: 472.856000, BarCount: 4>"}
    // Need to convert it to:
    // type TOHLCV struct {
    //  DateTime time.Time
    //  Open     float64
    //  High     float64
    //  Low      float64
    //  Close    float64
    //  Volume   float64
    // }

    // convert the datetime string "1704818100" format to Int64
    var dateTime int64
    dateTime, _ = strconv.ParseInt(bar.Date, 10, 64)

    // Convert the int dateTime to time.Time for the TOHLCV struct
    var dateTime2 = time.Unix(dateTime, 0)

    GlobalHistoricalDataSlice = append(GlobalHistoricalDataSlice, types.TOHLCV{
        DateTime: dateTime2,
        Open:     bar.Open,
        High:     bar.High,
        Low:      bar.Low,
        Close:    bar.Close,
        Volume:   bar.Volume,
    })
}

As you can see, I created a get method that returns the latest data appended in the newly created GlobalHistoricalDataSlice. This just converts the BarData to a struct of my own creation in the types folder:

package types

import "time"

// Struct used when defining TOHLCV data
type TOHLCV struct {
    DateTime time.Time
    Open     float64
    High     float64
    Low      float64
    Close    float64
    Volume   float64
}

Now the main script looks like this:

package main

import (
    "fmt"
    "time"

    "go_ib/ibapi2"
)

func main() {

    log := ibapi2.GetLogger().Sugar()
    defer log.Sync()

    wrapper := ibapi2.Wrapper{}

    ic := ibapi2.NewIbClient(&wrapper)

    if err := ic.Connect("127.0.0.1", 4002, 0); err != nil {
        log.Panic("Connect failed:", err)
    }

    if err := ic.HandShake(); err != nil {
        log.Panic("HandShake failed:", err)
    }

    // Create a contract for SPY ETF
    spyContract := ibapi2.Contract{
        Symbol:       "SPY",
        SecurityType: "STK",
        Exchange:     "SMART",
        Currency:     "USD",
    }

    // Set the request ID
    reqID := ic.GetReqID()

    // Get the contract details to ensure proper contract is created
    // ic.ReqContractDetails(reqID, &spyContract)

    // Request delayed market data for SPY
    ic.ReqMarketDataType(3) // 3 or 4 for delayed, 1 when you have subscribed to real-time market data
    ic.ReqHistoricalData(reqID, &spyContract, "", "1 D", "5 mins", "TRADES", false, 2, true, []ibapi2.TagValue{}) // time.Now().In(time.FixedZone("EST", -5*60*60)).Format("20060102 15:04:05")
    ic.Run()

    // Example log output:
    /*
        {"level":"info","ts":1704767599.3692198,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/wrapper.go:353","msg":"<HistoricalData>","reqID":1,"bar":"BarData<Date: 1704186000, Open: 476.250000, High: 476.360000, Low: 476.000000, Close: 476.270000, Volume: 323.000000, Average: 476.301000, BarCount: 70>"}
        ...
        {"level":"info","ts":1704767599.4046462,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/wrapper.go:353","msg":"<HistoricalData>","reqID":1,"bar":"BarData<Date: 1704743700, Open: 473.510000, High: 474.000000, Low: 473.500000, Close: 473.760000, Volume: 10218.000000, Average: 473.790000, BarCount: 5146>"}
        {"level":"info","ts":1704767599.4046462,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/wrapper.go:359","msg":"<HistoricalDataEnd>","reqID":1,"startDate":"20240101  20:33:17","endDate":"20240108  20:33:17"}
    */

    // Wait a bit to receive some data
    <-time.After(time.Second * 15)

    // First inspection of the historical data
    fmt.Printf("First inspection of the historical data:\n")
    for _, bar := range ibapi2.GetHistoricalData() {
        fmt.Printf("Date: %v, Open: %v, High: %v, Low: %v, Close: %v, Volume: %v\n",
            bar.DateTime,
            bar.Open,
            bar.High,
            bar.Low,
            bar.Close,
            bar.Volume,
        )
    }

    // Wait 10 minutes to allow for more 5-minute data to be generated by IB
    <-time.After(time.Second * 610)

    // Request more historical data for SPY. Run ic.Run() again?
    //ic.ReqHistoricalData(reqID, &spyContract, "", "1 D", "5 mins", "TRADES", false, 2, true, []ibapi2.TagValue{})
    //ic.Run()

    // Second inspection of the historical data
    fmt.Print("Second inspection of the historical data:\n")
    for _, bar := range ibapi2.GetHistoricalData() {
        fmt.Printf("Date: %v, Open: %v, High: %v, Low: %v, Close: %v, Volume: %v\n",
            bar.DateTime,
            bar.Open,
            bar.High,
            bar.Low,
            bar.Close,
            bar.Volume,
        )
    }

    // Cancel the historical data request
    ic.CancelHistoricalData(reqID)

    ic.Disconnect()
}

You can see that I am employing the wait of about 10 minutes which should pull fresher up to date bar data after the first pull, however when you inspect the outputs, the bar data is the same. When you re-run the entire script, the more recent data IS pulled, however I thought there would be a better way to pull up to date data with the SAME ic instance without having to disconnect/reconnect/make a new client etc. I also have commented out a section that tries to run the ic.Run() method again, but again the output of bars pulled from the server doesn't change.

Any ideas?

Here's a more concise main code, and the example output I'm currently getting. This main code creates a new historical data request every minute for 15 minutes. The desired output should show all of today's bars, where the data should have a new bar added to it every minute. The code runs perfectly for about 3 minutes, and then I start getting the error shown below. I've narrowed it down to this part of the client.go file, however I don't know what the cause of it is. Would like some assistance getting a working code as it appears to be an ibapi issue:

package main

import (
    "fmt"
    "time"

    "github.com/hadrianl/ibapi"
)

type TestWrapper struct {
    ibapi.Wrapper
}

func (t *TestWrapper) HistoricalData(reqID int64, bar *ibapi.BarData) {
    fmt.Printf("HistoricalData. ReqId: %d, BarData: %+v\n", reqID, bar)
}

func (t *TestWrapper) HistoricalDataEnd(reqID int64, startDateStr string, endDateStr string) {
    fmt.Printf("HistoricalDataEnd. ReqId: %d, from %s to %s\n", reqID, startDateStr, endDateStr)
}

func (t *TestWrapper) HistoricalDataUpdate(reqID int64, bar *ibapi.BarData) {
    fmt.Printf("HistoricalDataUpdate. ReqId: %d, BarData: %+v\n", reqID, bar)
}

func main() {

    // Create a new instance of TestWrapper
    wrapper := &TestWrapper{}

    // Create a new instance of IbClient
    client := ibapi.NewIbClient(wrapper)

    // Connect to Interactive Brokers Gateway
    if err := client.Connect("127.0.0.1", 4002, 0); err != nil {
        fmt.Println("Error connecting to IB Gateway:", err)
        return
    }

    // Handshake with the server
    if err := client.HandShake(); err != nil {
        fmt.Println("Handshake failed:", err)
        return
    }

    fmt.Println("Establishing connection...")
    // Wait for the connection to be established
    <-time.After(time.Second * 2)
    fmt.Println("Connection established!")

    // Define SPY contract
    spyContract := ibapi.Contract{
        Symbol:       "SPY",
        SecurityType: "STK",
        Exchange:     "SMART",
        Currency:     "USD",
    }

    // Set the request ID
    reqID := client.GetReqID()

    client.ReqMarketDataType(3) // 3 or 4 for delayed, 1 when you have subscribed to real-time market data

    // Print query for historical data in real-time for 15 minutes, every minute
    startTime := time.Now()
    for time.Since(startTime) < 15*time.Minute {

        // Request latest historical data for today
        client.ReqHistoricalData(reqID, &spyContract, "", "1 D", "1 min", "TRADES", false, 2, true, nil)
        client.Run()

        // Wait for historical data to be received/run again after 60 seconds
        <-time.After(time.Second * 60)

        // Cancel the current historical data request so we can start another one next iteration
        client.CancelHistoricalData(reqID)
    }

    // Disconnect from IB Gateway
    client.Disconnect()

    fmt.Println("Disconnected.")
}

PS I:\nasty\Golang_Projects\Go_Interactive_Brokers> go run .
Establishing connection...
Connection established!
{"level":"info","ts":1708970108.8783593,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3048","msg":"run client"}
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938000, Open: 507.320000, High: 507.320000, Low: 507.220000, Close: 507.220000, Volume: 9.000000, Average: 507.275000, BarCount: 3>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938060, Open: 507.260000, High: 507.260000, Low: 507.260000, Close: 507.260000, Volume: 3.000000, Average: 507.260000, BarCount: 2>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938120, Open: 507.320000, High: 507.320000, Low: 507.310000, Close: 507.310000, Volume: 2.000000, Average: 507.314000, BarCount: 2>
...
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969020, Open: 507.650000, High: 507.670000, Low: 507.580000, Close: 507.590000, Volume: 319.000000, Average: 507.634000, BarCount: 211>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969080, Open: 507.580000, High: 507.620000, Low: 507.470000, Close: 507.490000, Volume: 289.000000, Average: 507.527000, BarCount: 193>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969140, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.470000, Volume: 460.000000, Average: 507.454000, BarCount: 276>
HistoricalDataEnd. ReqId: 1, from 20240225  11:55:09 to 20240226  11:55:09
{"level":"info","ts":1708970168.882816,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3048","msg":"run client"}
{"level":"info","ts":1708970168.9018764,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/wrapper.go:658","msg":"<Error>","reqID":1,"errCode":162,"errString":"Historical Market Data Service error message:API historical data query cancelled: 1"}
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938000, Open: 507.320000, High: 507.320000, Low: 507.220000, Close: 507.220000, Volume: 9.000000, Average: 507.275000, BarCount: 3>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938060, Open: 507.260000, High: 507.260000, Low: 507.260000, Close: 507.260000, Volume: 3.000000, Average: 507.260000, BarCount: 2>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938120, Open: 507.320000, High: 507.320000, Low: 507.310000, Close: 507.310000, Volume: 2.000000, Average: 507.314000, BarCount: 2>
....
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969080, Open: 507.580000, High: 507.620000, Low: 507.470000, Close: 507.490000, Volume: 289.000000, Average: 507.527000, BarCount: 193>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969140, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.470000, Volume: 460.000000, Average: 507.454000, BarCount: 276>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969200, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.450000, Volume: 384.000000, Average: 507.464000, BarCount: 256>
HistoricalDataEnd. ReqId: 1, from 20240225  11:56:09 to 20240226  11:56:09
{"level":"info","ts":1708970228.895973,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3048","msg":"run client"}
{"level":"error","ts":1708970228.895973,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:2942","msg":"write req error","nbytes":11,"reqMsg":"AAAABzI1ADEAMQA=","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goRequest\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:2942"}
{"level":"error","ts":1708970228.8965116,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3028","msg":"got client error in decode loop","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goDecode\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3028"}
{"level":"info","ts":1708970228.9120264,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/wrapper.go:658","msg":"<Error>","reqID":1,"errCode":162,"errString":"Historical Market Data Service error message:API historical data query cancelled: 1"}
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938000, Open: 507.320000, High: 507.320000, Low: 507.220000, Close: 507.220000, Volume: 9.000000, Average: 507.275000, BarCount: 3>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938060, Open: 507.260000, High: 507.260000, Low: 507.260000, Close: 507.260000, Volume: 3.000000, Average: 507.260000, BarCount: 2>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708938120, Open: 507.320000, High: 507.320000, Low: 507.310000, Close: 507.310000, Volume: 2.000000, Average: 507.314000, BarCount: 2>
....
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969140, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.470000, Volume: 460.000000, Average: 507.454000, BarCount: 276>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969200, Open: 507.480000, High: 507.520000, Low: 507.410000, Close: 507.450000, Volume: 384.000000, Average: 507.464000, BarCount: 256>
HistoricalData. ReqId: 1, BarData: BarData<Date: 1708969260, Open: 507.460000, High: 507.480000, Low: 507.420000, Close: 507.450000, Volume: 335.000000, Average: 507.455000, BarCount: 219>
HistoricalDataEnd. ReqId: 1, from 20240225  11:57:09 to 20240226  11:57:09
{"level":"info","ts":1708970288.8977444,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3048","msg":"run client"}
{"level":"error","ts":1708970288.8979797,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:2942","msg":"write req error","nbytes":11,"reqMsg":"AAAABzI1ADEAMQA=","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goRequest\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:2942"}
{"level":"error","ts":1708970288.8985107,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3028","msg":"got client error in decode loop","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goDecode\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3028"}
{"level":"info","ts":1708970348.8977444,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3048","msg":"run client"}
{"level":"error","ts":1708970348.8979797,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:2942","msg":"write req error","nbytes":11,"reqMsg":"AAAABzI1ADEAMQA=","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goRequest\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:2942"}
{"level":"error","ts":1708970348.8985107,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3028","msg":"got client error in decode loop","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goDecode\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3028"}
{"level":"info","ts":170897408.8977444,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3048","msg":"run client"}
{"level":"error","ts":170897408.8979797,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:2942","msg":"write req error","nbytes":11,"reqMsg":"AAAABzI1ADEAMQA=","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goRequest\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:2942"}
{"level":"error","ts":170897408.8985107,"caller":"ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3028","msg":"got client error in decode loop","error":"short write","stacktrace":"github.com/hadrianl/ibapi.(*IbClient).goDecode\n\tC:/Users/chalu/go/pkg/mod/github.com/hadrianl/ibapi@v0.0.0-20230925013244-4f647c0e9c16/client.go:3028"}

As you can see, I get about 3 responses in 3 minutes, then nothing. I get that error traceback. I thought initially it was because of a rate limit issue, however I believe the API allows for up to 60 requests in 10 minutes, so 6 per minute, which I'm clearly under.

hadrianl / ibapi

How to request historical data multiple times? #49