Update recent Stock/Options data from Polygon

[jhirschibar]

Pull data from the last year for stocks and options. (not been refreshed since 2023)

Question: what do we do for stocks where there has been a corporate action?

[karlie15]

DoD

• script ready for monthly refreshes
• data is up to date

[karlie15]

UPDATE:

• When pipeline complete #25 , sign up for higher subscription tier
• Pull data set

[jhirschibar]

In order to convert the quote data to useable price data, we require significant compression. The compression algorithm will work as follows:

• download one file (~100 gb)
• parse through each file making a hash with the following fields:

  {options_ticker:
  { close_bid, close_ask, close_midpoint, average_spread_size, average_bid, average_ask, average_midpoint, % above close, % below close, num_quotes}
  }

• then upload to the db.

The challenge will be to make this O(N) time Will do with one file and see how big of a file it creates.

Or it can be an ETL step for the flat file: eg, upload the entire csv to postgres table, compress via a query, upload to a new table, drop the old table. This would likely be better than the python work

[jhirschibar]

Note, when repulling data, only make directories/files for options contracts that have results. Lots of empty files cluttering the file system

[jhirschibar]

Note, when repulling data, only make directories/files for options contracts that have results. Lots of empty files cluttering the file system

This is complete with issue #31 and #40 pr.

Read to delete tables, clear files, and re-pull data.

[jhirschibar]

Need to make sure that Historical prices for options don't make files for contracts that never trade. And the path runners need to know what to crawl. If no file/directory is made for a contract, it needs to know to pass over it

[jhirschibar]

This ticket has morphed to include refactoring the quote data so that we can backfill without timeouts. It involves making an abstraction of the aiomultiprocess pool, worker, and scheduler classes customized to the quote process

[jhirschibar]

kill pill to trigger the TTL if there is a new ticker in the queue for the QuoteWorker second kill pill if there is 16 consecutive queries that return non results. If this occurs, the set running = False, and go and perform the queue_cleanup() to go and remove all remaining tasks that involve that o_ticker. Then let the worker expire.

This will require the o_ticker to be attached to the args in an easily extractable fashion. It will also require the args to be ordererd by oticker so that they work with the QuoteScheduler

Also, update the uploader to only make args for directories/files that exist