P1: Can't do sample plot of term structure with today's date (after close) where date of last day market open works.

[AQuant206]

I got sample_plot_term_structure.py working over weekend and it seems to work with yesterday's date (e.g. 2023-03-24 on Sat 3-25). However it doesn't work with today's date well after 16:00 Central/CBOE time (e.g. 4:50PM Pacific), when Yahoo finance has CBOE values for for 1st and 2nd month forward contracts after today's close.

You should be able to repro this by editing src/examples/sample_plot_term_structure.py - changing the day_of_interest value to today's date (after market close). E.g.

    #plot the term structure for Mon Mar 27, 2023
    day_of_interest = '2023-03-27'
    df_day_of_interest =wide_with_continuous_futures.loc[[day_of_interest]]

This is the output from running the sample with above change. It works if I set day_of_interest to '2023-03-24' and Yahoo finance has CBOE values for market close for 1st and 2nd month forward contracts.

****Running python file: z:\Downloads\Source-Master\vix_utils\examples\aq_sample_plot_today_term_structure.py

WARNING:root: Duplicates detected Trade Date Weekly Tenor_Monthly Tenor_Days Tenor_TradeDays Expiry Open High Low Close Settle Change Total Volume EFP Open Interest Year MonthOfYear Futures
File Expired 37824 2020-02-27 00:00:00-05:00 True 2.0 26 19.0 2020-03-25 00:00:00-04:00 24.15 25.35 22.15 24.25 25.025 3.000 48 0 63 2020 3 H (Mar 2020) 2020-03-25.w.CFE_VX2020.csv True 37825 2020-02-27 00:00:00-05:00 True 2.0 26 19.0 2020-03-25 00:00:00-04:00 24.15 25.35 22.15 24.25 25.025 3.000 48 0 63 2020 3 H (Mar 2020) 2020-03-25.w.CFE_VX2020.csv True 37839 2020-02-28 00:00:00-05:00 True 2.0 25 18.0 2020-03-25 00:00:00-04:00 26.50 27.50 25.65 26.50 26.100 1.075 112 0 139 2020 3 H (Mar 2020) 2020-03-25.w.CFE_VX2020.csv True 37840 2020-02-28 00:00:00-05:00 True 2.0 25 18.0 2020-03-25 00:00:00-04:00 26.50 27.50 25.65 26.50 26.100 1.075 112 0 139 2020 3 H (Mar 2020) 2020-03-25.w.CFE_VX_2020.csv True, cleaning them out Traceback (most recent call last): File "z:\Downloads\Source-Master\vix_utils\examples\aq_sample_plot_today_term_structure.py", line 71, in main() File "z:\Downloads\Source-Master\vix_utils\examples\aq_sample_plot_today_term_structure.py", line 51, in main df_day_of_interest =wide_with_continuous_futures.loc[[day_of_interest]]

  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexing.py", line 1073, in __getitem__
    return self._getitem_axis(maybe_callable, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexing.py", line 1301, in _getitem_axis
    return self._getitem_iterable(key, axis=axis)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexing.py", line 1239, in _getitem_iterable
    keyarr, indexer = self._get_listlike_indexer(key, axis)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexing.py", line 1432, in _get_listlike_indexer
    keyarr, indexer = ax._get_indexer_strict(key, axis_name)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6070, in _get_indexer_strict
  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6070, in _get_indexer_strict
  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6070, in _get_indexer_strict
    self._raise_if_missing(keyarr, indexer, axis_name)
  File "C:\Users\R Dev\anaconda3\envs\VixUtils\Lib\site-packages\pandas\core\indexes\base.py", line 6130, in _raise_if_missing
    raise KeyError(f"None of [{key}] are in the [{axis_name}]")
KeyError: "None of [Index(['2023-03-27'], dtype='object', name='Trade Date')] are in the [index]"

[dougransom]

Ok, if you feel like looking into it please assign to yourself, otherwise I'll look at it eventually.

[dougransom]

Finally looking at this. Could be the files downloaded from https://www.cboe.com/us/futures/market_statistics/historical_data/ are not updated until a certain time of day, or it could be some problem with vix_utils.

[dougransom]

Reproduced this at 1:02PM pacific time July 5, 2023. Problem persists at 3:17PM.

[dougransom]

As it appears to be a problem with the CBOE data source, it is not easy to fix. Same problem with the other Vol indexes (VIX9D, VIX, VIX3M etc.). It is theoretically possible to get the VX futures from https://www.cboe.com/delayed_quotes/vix/future_quotes or https://www.cboe.com/tradable_products/vix/vix_futures/ or https://www.cboe.com/tradable_products/vix/term_structure/

These pages work

https://www.cboe.com/delayed_quotes/vix9d https://www.cboe.com/delayed_quotes/vix https://www.cboe.com/delayed_quotes/shortvol

The data are created with javascript, try scraping with requests-html

[dougransom]

Maybe someone has some ideas about how to scrape the links above or another source of delayed quotes we can use?

[dougransom]

https://www.bing.com/videos/search?q=yahoo+finance+api&docid=603500247230070630&mid=A75A5497AB532B641AF5A75A5497AB532B641AF5&view=detail&FORM=VIRE

We should be able to get real time index values, but not futures: https://pypi.org/project/yfinance/

vectorbt can load data from a variety of sources.

[AQuant206]

I’ve been pulling daily Vix1m, etc from Yahoo finance after market closes as a workaround. If Yahoo has the data 30m after close, doesn’t CBOE?

On Thu, Jul 6, 2023 at 12:10 AM Doug Ransom @.***> wrote:

https://www.bing.com/videos/search?q=yahoo+finance+api&docid=603500247230070630&mid=A75A5497AB532B641AF5A75A5497AB532B641AF5&view=detail&FORM=VIRE

https://pypi.org/project/yfinance/

— Reply to this email directly, view it on GitHub https://github.com/dougransom/vix_utils/issues/18#issuecomment-1622659241, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3454GULNEYYRWWE2NQJAI3XOXX5VANCNFSM6AAAAAAWJZYFLE . You are receiving this because you authored the thread.Message ID: @.***>

[dougransom]

Can you please post some code snippets, maybe in this thread? I don't know how to read the vix futures from yahoo or even access them through the web.

I don't know when CBOE updates their historical data online, not timely obviously.

[AQuant206]

Can do next week (box w/ my code is 6 timezones away).

On Thu, Jul 6, 2023 at 5:16 PM Doug Ransom @.***> wrote:

Can you please post some code snippets, maybe in this thread? I don't know how to read the vix futures from yahoo or even access them through the web.

— Reply to this email directly, view it on GitHub https://github.com/dougransom/vix_utils/issues/18#issuecomment-1623954861, or unsubscribe https://github.com/notifications/unsubscribe-auth/A3454GRT6VHNZ5KYA46E5CDXO3QERANCNFSM6AAAAAAWJZYFLE . You are receiving this because you authored the thread.Message ID: @.***>

[dougransom]

@AQuant206 snippets?

[dougransom]

It is theoretically possible to download from CBOE web site at end of day:

`import json import logging import os import subprocess import sys import time import urllib from logging import getLogger from time import sleep from dotenv import load_dotenv from playwright.async_api import async_playwright import asyncio as aio from icecream import ic from parsel import Selector

setup basic logging for our project which will display the time, log level & log message

logger = getLogger("webscapper.py") logging.basicConfig( stream=sys.stdout, # uncomment this line to redirect output to console format="%(message)s", level=logging.DEBUG, )

capabilities = { "browserName": "MicrosoftEdge", # Browsers allowed: Chrome, MicrosoftEdge, pw-chromium, pw-firefox and pw-webkit "browserVersion": "latest", "LT:Options": { "platform": "Windows 10", "build": "E Commerce Scrape Build", "name": "Scrape Lambda Software Product", "user": os.getenv("LT_USERNAME"), "accessKey": os.getenv("LT_ACCESS_KEY"), "network": False, "video": True, "console": True, "tunnel": False, # Add tunnel configuration if testing locally hosted webpage "tunnelName": "", # Optional "geoLocation": "", # country code can be fetched from https://www.lambdatest.com/capabilities-generator/ }, }

vx_futures_expected_div_text="VX - Cboe Volatility Index (VIX) Futures"

async def main(): async with async_playwright() as pw:

    xpath_to_title=f"//div[text()=\"Futures\"]"
    ic(xpath_to_title)

    browser=await pw.chromium.launch()
    page=await browser.new_page()
    await page.goto("https://www.cboe.com/delayed_quotes/vix/")
    await page.wait_for_selector(xpath_to_title)
    sleep(3)

    page_html=await page.content()

    with open("sample_futures.pw.html","w") as f2:
        f2.write(page_html)

    sel = Selector(page_html)

    xpath_to_table=f"{xpath_to_title}/following-sibling::div/table"
    ic(xpath_to_table)

    div_title=sel.xpath(xpath_to_title)
    ic(div_title)
    table=sel.xpath(xpath_to_table)
    ic(table)

if name == "main": aio.run(main())`

and to parse out

`import asyncio as aio from icecream import ic from parsel import Selector

def toString(a): return f"{a}"

def main(): ic.configureOutput(argToStringFunction=toString) with open("sample_futures.pw.html","r") as i: page_html=i.read()

    sel = Selector(page_html)
    xpath_to_title=f"//div[text()=\"Futures\"]"
    ic(xpath_to_title)

    xpath_to_table=f"{xpath_to_title}/following-sibling::div/table"
    ic(xpath_to_table)

    title=sel.xpath(xpath_to_title)
    ic(title)

    table=sel.xpath(xpath_to_table)
    ic(table)

    cols=table.xpath("//th/text()").getall()
    ic(cols)      
    rows=table.xpath("//tbody/tr")
    ic(rows)
    for row in rows:
        cels=row.xpath("td//text()").getall()
        ic(cels)
        for col,cel in zip(cols,cels):
            #\n\tValue {cel}
            print(f"Column: \t{col}  \n\tText {cel}")

    all_cels=sel.xpath("//td")
    #ic(all_cels,)

if name == "main": main() ` I am not intending to add any such functionality to vix_utils though.

[dougransom]

CBOE has a list of approved data vendors. Probably none of them are free, even for delayed data. There should be no exchange fees though.