search

antequant / blotter

Microservice to connect to Interactive Brokers and stream market data into Google BigQuery
MIT License
1 stars 4 forks source link

Repeatedly uploading all RealTimeBars results in duplicate data #6

Closed jspahrsummers closed 4 years ago

jspahrsummers commented 4 years ago

We could try uploading a single bar at a time, although BigQuery might get mad at us. Smarter would be some primitive batching strategy.

jspahrsummers commented 4 years ago

https://cloud.google.com/bigquery/quotas

jspahrsummers commented 4 years ago

You are limited to 1,000 operations per table per day whether the operation appends data to a table, overwrites a table, or uses a DML INSERT statement to write data to a table.

This suggests we should batch at least as much necessary to stay under that limit.

jspahrsummers commented 4 years ago

Real-time 5 second bars could result in up to 17,280 bars per day.