evdubs / oic-options-chains

ETL for OIC Options Chains
Mozilla Public License 2.0
26 stars 8 forks source link

where does oic pull its list of symbols from. #1

Closed CatspersCoffee closed 4 years ago

CatspersCoffee commented 4 years ago

Hi evdubs, Just noticed some symbols that are in the nasdaq list on the DB are not having the OIC data pulled from options quotes website. for example if i run in pgAdmin:


select act_symbol, security_name, listing_exchange
from nasdaq.symbol
where act_symbol = 'BYND'; ````

i get back in the correct info for BYND ticker, i.e. its in the nasdaq list. however when i run OIC i do not get under the options-chains/date directory the BYND.html options page. However there is in fact a page and data if i check the website manually

is there any way to probe via racket to get a full list of what ```racket extract.rkt``` sees and should pull data for? i may attempt to write a small program that does this.. unless you have a better idea?

Thankyou. 
evdubs commented 4 years ago

The query to pick symbols is:

select
  component_symbol as symbol
from
  spdr.etf_holding
where
  etf_symbol in ('SPY', 'MDY', 'SLY') and
  date = (select max(date) from spdr.etf_holding) and
  case when $1 != ''
    then component_symbol >= $1
    else true
  end and
  case when $2 != ''
    then component_symbol <= $2
    else true
  end
union
select distinct
  etf_symbol as symbol
from
  spdr.etf_holding
where
  date = (select max(date) from spdr.etf_holding) and
  case when $1 != ''
    then etf_symbol >= $1
    else true
  end and
  case when $2 != ''
    then etf_symbol <= $2
    else true
  end
union
select distinct
  component_symbol as symbol
from
  invesco.etf_holding
where
  date = (select max(date) from invesco.etf_holding) and
  case when $1 != ''
    then component_symbol >= $1
    else true
  end and
  case when $2 != ''
    then component_symbol <= $2
    else true
  end
order by
  symbol;

In effect, this is just picking SPDR ETFs and ETF components (with Invesco, too, but since most of those ADR ETFs are delisted, their impact is minimal).

If you want a bigger list of stocks to pull from OIC, you'll need to replace this query.

evdubs commented 4 years ago

You'll find this query in extract.rkt.

CatspersCoffee commented 4 years ago

Ok so i wrote a C++ program to print the sorted list from nasdaq.symbol (becuase im terrible at racket) the list that was printed does not include the ticker BYND, so i manually added the symbol BYND and checked it under pgAdmin.. its there now in pgAdmin.. however when i go again to print the list its still not there.. i seem to be missing something as to where a query to nasdaq.symbol pulls ticker data OR maybe im not adding the ticker/act_symbol to the correct list?

i have number of tickers = 1517 in total in the list.

heres a sample query and attached screen for AMD and BYND via pgAdmin

select act_symbol, 
security_name, 
listing_exchange,
market_category, 
is_etf, 
round_lot_size,
  is_test_issue,
  financial_status,
  cqs_symbol,
  nasdaq_symbol,
  is_next_shares,
  last_seen
from nasdaq.symbol
where act_symbol = 'BYND' or act_symbol = 'AMD';

q

evdubs commented 4 years ago

Here's a PR with an example of how you could add a symbol to the extract query.

CatspersCoffee commented 4 years ago

Awesome thanks for that, i implemented the change to the code and BYND is now being pulled from the oic page. However im still a little confused about the process. does extract.rkt query the DB for a list of nasdaq symbols or spdr.etf_holding ? it looks like its querys spdr.etf_holding, or does the symbol have to exist in oic.option_chain before extract.rkt will download the page?.. its a little bit of a chicken egg question.

also i have a couple other questions. if you dont mind, i know you mentioned "issues" should be added to their respective project but i feel like htese are more general questions:

in renegade way under options-strategy.rkt what do the options patterns: "BP", "HB", "AT", "IR" and under short-call: "BR", "LB", "DT" "DR" stand for?

lastly, and thanks for reading through all of these, in oic, you take options expiring 2 weeks, 4 weeks, and 8 weeks from the current date. is the a "quick-and-dirty" way to just take the whole lot, i.e., all the strikes and all the expiry's?

thanks for your continued communication :-)

evdubs commented 4 years ago

Awesome thanks for that, i implemented the change to the code and BYND is now being pulled from the oic page. However im still a little confused about the process. does extract.rkt query the DB for a list of nasdaq symbols or spdr.etf_holding ? it looks like its querys spdr.etf_holding, or does the symbol have to exist in oic.option_chain before extract.rkt will download the page?.. its a little bit of a chicken egg question.

It all starts with nasdaq-symbols. That project has no data requirements; it just gets symbols from the NASDAQ FTP and inserts/updates records in the DB. For oic-options-chains, in the Readme, there's this:

These programs currently (2019-11-11) will just extract option chains for S&P 500/400/600 component companies, Invesco ADR component companies, and some SPDR ETFs.

It doesn't currently try to get options for the universe of stocks/ETFs. It just looks for S&P index constitutents and SPDR ETFs (and Invesco, but that can be ignored). So, oic-options-chains relies on spdr-etf-holdings, and spdr-etf-holdings relies on nasdaq-symbols.

There is no reliance on data already existing in oic tables for the oic-options-chains extract process to run.

in renegade way under options-strategy.rkt what do the options patterns: "BP", "HB", "AT", "IR" and under short-call: "BR", "LB", "DT" "DR" stand for?

The renegade-way Readme shows the unabbreviated names:

Patterns. This includes bull pullback, bear rally, high base, low base, ascending triangle, descending triangle, range rally, and range pullback. They are abbreviated in the list.

There's also IR/DR (increasing rank/decreasing rank) and IV/DV (increasing vol/decreasing vol). These are also briefly described in the Readme.

lastly, and thanks for reading through all of these, in oic, you take options expiring 2 weeks, 4 weeks, and 8 weeks from the current date. is the a "quick-and-dirty" way to just take the whole lot, i.e., all the strikes and all the expiry's?

For my use cases, I do not need or want to save all options. I just care about a few expirations and some strikes for those expirations. If you think you need all options, you can change a bit in oic-options-chains/transform-load.rkt to get them. I can help show you how to do that.

CatspersCoffee commented 4 years ago

It doesn't currently try to get options for the universe of stocks/ETFs. It just looks for S&P index constitutents and SPDR ETFs (and Invesco, but that can be ignored). So, oic-options-chains relies on spdr-etf-holdings, and spdr-etf-holdings relies on nasdaq-symbols.

Ok cool. thankyou

Patterns. This includes bull pullback, bear rally, high base, low base, ascending triangle, descending triangle, range rally, and range pullback. They are abbreviated in the list.

There's also IR/DR (increasing rank/decreasing rank) and IV/DV (increasing vol/decreasing vol). These are also briefly described in the Readme.

Awesome thanks for that, i missed that in the readme.

Ok i think i have found the section in transform load in oic that sets up your target strikes and expiry's... ill have a go at modifying it, if i make a huge mess i might ask if you can help.

         [target-strikes (list (* mark-price 70/100) (* mark-price 725/1000) (* mark-price 75/100) (* mark-price 775/1000)
                               (* mark-price 80/100) (* 825/1000) (* mark-price 85/100) (* mark-price 875/1000)
                               (* mark-price 90/100) (* mark-price 92/100) (* mark-price 94/100) (* mark-price 96/100) (* mark-price 98/100)
                               mark-price (* mark-price 102/100) (* mark-price 104/100) (* mark-price 106/100) (* mark-price 108/100)
                               (* mark-price 110/100) (* mark-price 1125/1000) (* mark-price 115/100) (* mark-price 1175/1000)
                               (* mark-price 120/100) (* mark-price 1225/1000) (* mark-price 125/100) (* mark-price 1275/1000)
                               (* mark-price 130/100))]
         [target-expirations (list (+days (folder-date) (* 7 2))
                                   (+days (folder-date) (* 7 4))
                                   (+days (folder-date) (* 7 8)))]

Thanks again for the replies :-)

evdubs commented 4 years ago

How are the modifications going?

CatspersCoffee commented 4 years ago

Hi, thanks for the follow up! so i modified your original transform-load program.. managed to get a host of strikes, but not all of them.. so you could say my modification is quite crude and inelegant, i assume i would have to really modify the mark-price list for allot of different +- percentages, and im not quite sure if this would be the best method to extract all the strikes.. Unfortunately again, my racket is terrible. ill give you an example with BYND again where my modification breaks down:

The below table is the .csv for options contract Calls expiring 2020-07-17 pulled from the DB after the transform-load. The closing price that day for BYND was $144.98, so around the current price i have the data for calls at strikes: 143, 140, 137, 135, 133, 130, 125,120.

Where the BYND.html pulled from oic has: 144, 143, 142, 141, 140, 139, 138. etc

you can see that im missing quite a few strike that lie close the the underlying price on that day.


act_symbol | expiration | strike | call_put | date | bid | ask | vol | delta | gamma | theta | vega | rho |  
-- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | -- | --
BYND | 7/17/2020 | 70 | Call | 7/8/2020 | 71.9 | 74.05 | 1.4646 | 0.9989 | 0.0001 | -0.0065 | 0.0008 | 0.0191 |  
BYND | 7/17/2020 | 75 | Call | 7/8/2020 | 67.1 | 68.5 | 1.4646 | 0.9973 | 0.0002 | -0.0145 | 0.0019 | 0.0204 |  
BYND | 7/17/2020 | 80 | Call | 7/8/2020 | 61.55 | 64.4 | 1.4646 | 0.9942 | 0.0005 | -0.0292 | 0.0039 | 0.0216 |  
BYND | 7/17/2020 | 85 | Call | 7/8/2020 | 56.4 | 59.1 | 1.3565 | 0.9925 | 0.0006 | -0.0336 | 0.0049 | 0.0229 |  
BYND | 7/17/2020 | 90 | Call | 7/8/2020 | 52.4 | 53.6 | 1.2484 | 0.9906 | 0.0009 | -0.0378 | 0.006 | 0.0242 |  
BYND | 7/17/2020 | 95 | Call | 7/8/2020 | 46.15 | 49.15 | 1.1528 | 0.9876 | 0.0012 | -0.0444 | 0.0076 | 0.0255 |  
BYND | 7/17/2020 | 100 | Call | 7/8/2020 | 41.05 | 45.05 | 1.0573 | 0.9837 | 0.0016 | -0.0514 | 0.0096 | 0.0267 |  
BYND | 7/17/2020 | 105 | Call | 7/8/2020 | 35.95 | 40.2 | 0.9614 | 0.9787 | 0.0022 | -0.0588 | 0.0121 | 0.0279 |  
BYND | 7/17/2020 | 110 | Call | 7/8/2020 | 30.9 | 33.75 | 0.886 | 0.9692 | 0.0033 | -0.0736 | 0.0165 | 0.0288 |  
BYND | 7/17/2020 | 115 | Call | 7/8/2020 | 28.1 | 29.15 | 0.8675 | 0.9447 | 0.0054 | -0.1153 | 0.0264 | 0.0292 |  
BYND | 7/17/2020 | 30 | Call | 7/8/2020 | 111.9 | 114.05 | 1.4646 | 1 | 0 | -0.0001 | 0 | 0.0082 |  
BYND | 7/17/2020 | 120 | Call | 7/8/2020 | 22.15 | 24 | 0.7532 | 0.93 | 0.0075 | -0.1203 | 0.0318 | 0.0299 |  
BYND | 7/17/2020 | 125 | Call | 7/8/2020 | 18.5 | 19.3 | 0.6673 | 0.8999 | 0.0111 | -0.1393 | 0.0416 | 0.0301 |  
BYND | 7/17/2020 | 130 | Call | 7/8/2020 | 14.1 | 15.15 | 0.6534 | 0.8272 | 0.0165 | -0.1985 | 0.0605 | 0.0284 |  
BYND | 7/17/2020 | 133 | Call | 7/8/2020 | 12.4 | 12.95 | 0.7011 | 0.7549 | 0.0189 | -0.2617 | 0.0744 | 0.0261 |  
BYND | 7/17/2020 | 135 | Call | 7/8/2020 | 10.95 | 11.45 | 0.6905 | 0.7151 | 0.0208 | -0.2782 | 0.0804 | 0.0249 |  
BYND | 7/17/2020 | 137 | Call | 7/8/2020 | 9.7 | 10.35 | 0.7053 | 0.6674 | 0.0218 | -0.304 | 0.086 | 0.0234 |  
BYND | 7/17/2020 | 140 | Call | 7/8/2020 | 7.75 | 8.25 | 0.6768 | 0.5997 | 0.0241 | -0.3103 | 0.0915 | 0.0213 |  
BYND | 7/17/2020 | 143 | Call | 7/8/2020 | 6.4 | 6.95 | 0.7012 | 0.5259 | 0.024 | -0.3311 | 0.0943 | 0.0188 |  
BYND | 7/17/2020 | 146 | Call | 7/8/2020 | 5.15 | 5.6 | 0.7046 | 0.4551 | 0.0238 | -0.3313 | 0.0939 | 0.0164 |  
BYND | 7/17/2020 | 149 | Call | 7/8/2020 | 4.35 | 5.4 | 0.7735 | 0.401 | 0.0211 | -0.3546 | 0.0915 | 0.0144 |  
BYND | 7/17/2020 | 152.5 | Call | 7/8/2020 | 3.4 | 3.6 | 0.7435 | 0.3246 | 0.0204 | -0.3171 | 0.0852 | 0.0118 |  
BYND | 7/17/2020 | 155 | Call | 7/8/2020 | 2.7 | 3.1 | 0.7497 | 0.2806 | 0.019 | -0.2995 | 0.0798 | 0.0102 |  
BYND | 7/17/2020 | 157.5 | Call | 7/8/2020 | 2.46 | 2.76 | 0.7849 | 0.2506 | 0.0171 | -0.2961 | 0.0753 | 0.0091 |  
BYND | 7/17/2020 | 160 | Call | 7/8/2020 | 2.15 | 2.21 | 0.7929 | 0.2164 | 0.0156 | -0.2757 | 0.0694 | 0.0079 |  
BYND | 7/17/2020 | 165 | Call | 7/8/2020 | 1.5 | 1.88 | 0.8388 | 0.1696 | 0.0127 | -0.2512 | 0.0598 | 0.0062 |  
BYND | 7/17/2020 | 167.5 | Call | 7/8/2020 | 1.25 | 1.43 | 0.832 | 0.1414 | 0.0114 | -0.221 | 0.053 | 0.0052 |  
BYND | 7/17/2020 | 172.5 | Call | 7/8/2020 | 0.86 | 1.12 | 0.8602 | 0.1073 | 0.0091 | -0.1883 | 0.0437 | 0.0039 |  
BYND | 7/17/2020 | 175 | Call | 7/8/2020 | 0.76 | 1.11 | 0.8939 | 0.0992 | 0.0082 | -0.1849 | 0.0413 | 0.0036 |  
BYND | 7/17/2020 | 180 | Call | 7/8/2020 | 0.7 | 1.03 | 0.965 | 0.0875 | 0.007 | -0.1819 | 0.0376 | 0.0032 |  
BYND | 7/17/2020 | 182.5 | Call | 7/8/2020 | 0.46 | 0.74 | 0.9285 | 0.0661 | 0.0058 | -0.1413 | 0.0304 | 0.0024 |  
BYND | 7/17/2020 | 185 | Call | 7/8/2020 | 0.61 | 0.9 | 1.0171 | 0.0747 | 0.0059 | -0.1702 | 0.0334 | 0.0027 |  
BYND | 7/17/2020 | 190 | Call | 7/8/2020 | 0.51 | 0.59 | 1.0261 | 0.0566 | 0.0047 | -0.1383 | 0.0269 | 0.0021 |  
BYND | 7/17/2020 | 195 | Call | 7/8/2020 | 0.44 | 0.7 | 1.1059 | 0.0549 | 0.0042 | -0.1456 | 0.0263 | 0.002 |  
BYND | 7/17/2020 | 200 | Call | 7/8/2020 | 0.33 | 0.5 | 1.1087 | 0.0416 | 0.0034 | -0.1168 | 0.021 | 0.0015 |  
BYND | 7/17/2020 | 205 | Call | 7/8/2020 | 0 | 0.55 | 1.1421 | 0.0353 | 0.0029 | -0.1054 | 0.0184 | 0.0013 |  
BYND | 7/17/2020 | 210 | Call | 7/8/2020 | 0.23 | 0.5 | 1.1754 | 0.0304 | 0.0025 | -0.0959 | 0.0163 | 0.0011 |  
BYND | 7/17/2020 | 215 | Call | 7/8/2020 | 0.05 | 0.45 | 1.2088 | 0.0265 | 0.0021 | -0.0879 | 0.0145 | 0.001 |  
BYND | 7/17/2020 | 220 | Call | 7/8/2020 | 0.2 | 0.28 | 1.2422 | 0.0233 | 0.0019 | -0.0812 | 0.0131 | 0.0008 |  
BYND | 7/17/2020 | 225 | Call | 7/8/2020 | 0.21 | 0.3 | 1.3069 | 0.0236 | 0.0018 | -0.0863 | 0.0132 | 0.0009 |  
BYND | 7/17/2020 | 230 | Call | 7/8/2020 | 0.01 | 0.4 | 1.3268 | 0.0201 | 0.0015 | -0.0765 | 0.0115 | 0.0007 |  
BYND | 7/17/2020 | 235 | Call | 7/8/2020 | 0.14 | 0.35 | 1.3467 | 0.0172 | 0.0013 | -0.0681 | 0.0101 | 0.0006 |  
BYND | 7/17/2020 | 240 | Call | 7/8/2020 | 0.06 | 0.31 | 1.3666 | 0.0149 | 0.0012 | -0.0609 | 0.0089 | 0.0005 |  
BYND | 7/17/2020 | 245 | Call | 7/8/2020 | 0.01 | 0.19 | 1.3865 | 0.0129 | 0.001 | -0.0547 | 0.0079 | 0.0005 |  
BYND | 7/17/2020 | 250 | Call | 7/8/2020 | 0.09 | 0.15 | 1.4064 | 0.0113 | 0.0009 | -0.0493 | 0.007 | 0.0004 |  
BYND | 7/17/2020 | 255 | Call | 7/8/2020 | 0.05 | 0.23 | 1.4064 | 0.009 | 0.0007 | -0.0405 | 0.0057 | 0.0003 |  
BYND | 7/17/2020 | 260 | Call | 7/8/2020 | 0.04 | 0.25 | 1.4064 | 0.0071 | 0.0006 | -0.0331 | 0.0047 | 0.0003 |  
BYND | 7/17/2020 | 265 | Call | 7/8/2020 | 0.02 | 0.12 | 1.4064 | 0.0057 | 0.0005 | -0.027 | 0.0038 | 0.0002 |  
BYND | 7/17/2020 | 270 | Call | 7/8/2020 | 0.07 | 0.14 | 1.4064 | 0.0045 | 0.0004 | -0.022 | 0.0031 | 0.0002 |  
BYND | 7/17/2020 | 72 | Call | 7/8/2020 | 72.05 | 73.5 | 1.0774 | 1 | 0 | -0.0004 | 0 | 0.0177 |  
BYND | 7/17/2020 | 76 | Call | 7/8/2020 | 68 | 69.5 | 1.0774 | 1 | 0 | -0.0007 | 0 | 0.0187 |  
BYND | 7/17/2020 | 83 | Call | 7/8/2020 | 60.85 | 62.65 | 1.0774 | 0.9996 | 0.0001 | -0.0022 | 0.0003 | 0.0204 |  
BYND | 7/17/2020 | 87 | Call | 7/8/2020 | 56.85 | 58.65 | 1.0774 | 0.9991 | 0.0001 | -0.0048 | 0.0007 | 0.0214 |  
BYND | 7/17/2020 | 91 | Call | 7/8/2020 | 52.75 | 54.75 | 1.0774 | 0.9977 | 0.0003 | -0.0101 | 0.0016 | 0.0223 |  
BYND | 7/17/2020 | 94 | Call | 7/8/2020 | 49.55 | 52 | 1.0774 | 0.9959 | 0.0005 | -0.0168 | 0.0027 | 0.023 |  
BYND | 7/17/2020 | 98 | Call | 7/8/2020 | 45.35 | 48.15 | 1.0774 | 0.9918 | 0.0009 | -0.0309 | 0.0051 | 0.0238 |  
BYND | 7/17/2020 | 101 | Call | 7/8/2020 | 42.5 | 45.05 | 1.0595 | 0.988 | 0.0013 | -0.0423 | 0.0071 | 0.0244 |  
BYND | 7/17/2020 | 109 | Call | 7/8/2020 | 34.4 | 37.25 | 0.9165 | 0.9801 | 0.0023 | -0.0565 | 0.011 | 0.0261 |  
BYND | 7/17/2020 | 112 | Call | 7/8/2020 | 31.15 | 34.45 | 0.8762 | 0.9742 | 0.003 | -0.0671 | 0.0137 | 0.0266 |  
BYND | 7/17/2020 | 116 | Call | 7/8/2020 | 27.3 | 30.5 | 0.814 | 0.9649 | 0.0042 | -0.0804 | 0.0176 | 0.0272 |  
BYND | 7/17/2020 | 123 | Call | 7/8/2020 | 20.35 | 24.05 | 0.6052 | 0.9624 | 0.006 | -0.0634 | 0.0186 | 0.0289 |  
BYND | 7/17/2020 | 127 | Call | 7/8/2020 | 18.35 | 19.3 | 0.7033 | 0.8953 | 0.0113 | -0.1621 | 0.0413 | 0.0273 |  
BYND | 7/17/2020 | 136 | Call | 7/8/2020 | 10.1 | 12.4 | 0.6438 | 0.753 | 0.0216 | -0.2578 | 0.0718 | 0.0241 |  
BYND | 7/17/2020 | 139 | Call | 7/8/2020 | 9.45 | 9.8 | 0.6952 | 0.6704 | 0.0229 | -0.3189 | 0.0824 | 0.0216 |  
BYND | 7/17/2020 | 142 | Call | 7/8/2020 | 7.75 | 8 | 0.6982 | 0.5967 | 0.0244 | -0.3425 | 0.0881 | 0.0194 |  
BYND | 7/17/2020 | 145 | Call | 7/8/2020 | 6.1 | 6.5 | 0.6952 | 0.5214 | 0.0252 | -0.3508 | 0.0906 | 0.0171 |  
BYND | 7/17/2020 | 148 | Call | 7/8/2020 | 5.05 | 5.2 | 0.7115 | 0.4489 | 0.0244 | -0.3566 | 0.09 | 0.0148 |  
BYND | 7/17/2020 | 150 | Call | 7/8/2020 | 4.4 | 4.55 | 0.7244 | 0.4043 | 0.0235 | -0.3555 | 0.0881 | 0.0133 |  
BYND | 7/17/2020 | 162.5 | Call | 7/8/2020 | 1.78 | 2.1 | 0.8057 | 0.2008 | 0.0153 | -0.2863 | 0.0639 | 0.0067 |  
BYND | 7/17/2020 | 170 | Call | 7/8/2020 | 1.16 | 1.3 | 0.8584 | 0.1326 | 0.011 | -0.233 | 0.0488 | 0.0044 |  
BYND | 7/17/2020 | 177.5 | Call | 7/8/2020 | 0.82 | 1.2 | 0.9621 | 0.103 | 0.0082 | -0.2184 | 0.0408 | 0.0034 |  

so i end up taking: +/- 0%, 2%, 4%, 6%, 8%, 10%, 12.5%, 15%, 17.5%, 20%, 22.5%, 25%, 27.5%, 30%,
32.5%, 35%, 37.5%, 40%, 42.5%, 45%, 47.5%, 50, 52.5%, 55%, 57.5%, 60%, 62.5, 65, 67.5% 70%
80%, 82.5%, 85%, 87.5%, 90, 92.5%, 95%, 97.5%, 100% strikes

here is my modification:

         [target-strikes (list (* mark-price 50/100) (* mark-price 525/1000) (* mark-price 55/100) (* mark-price 575/1000)
                               (* mark-price 60/100) (* mark-price 625/1000) (* mark-price 65/100) (* mark-price 675/1000)
                               (* mark-price 70/100) (* mark-price 725/1000) (* mark-price 75/100) (* mark-price 775/1000)
                               (* mark-price 80/100) (* 825/1000) (* mark-price 85/100) (* mark-price 875/1000)
                               (* mark-price 90/100) (* mark-price 92/100) (* mark-price 94/100) (* mark-price 96/100) (* mark-price 98/100)
                               mark-price (* mark-price 102/100) (* mark-price 104/100) (* mark-price 106/100) (* mark-price 108/100)
                               (* mark-price 110/100) (* mark-price 1125/1000) (* mark-price 115/100) (* mark-price 1175/1000)
                               (* mark-price 120/100) (* mark-price 1225/1000) (* mark-price 125/100) (* mark-price 1275/1000)
                               (* mark-price 130/100) (* mark-price 1325/1000) (* mark-price 135/100) (* mark-price 1375/1000)
                               (* mark-price 140/100) (* mark-price 1425/1000) (* mark-price 145/100) (* mark-price 1475/1000)
                               (* mark-price 150/100) (* mark-price 1525/1000) (* mark-price 155/100) (* mark-price 1575/1000)
                               (* mark-price 160/100) (* mark-price 1625/1000) (* mark-price 165/100) (* mark-price 1675/1000)
                               (* mark-price 170/100) (* mark-price 1725/1000) (* mark-price 175/100) (* mark-price 1775/1000)
                               (* mark-price 180/100) (* mark-price 1825/1000) (* mark-price 185/100) (* mark-price 1875/1000)
                               (* mark-price 190/100) (* mark-price 1925/1000) (* mark-price 195/100) (* mark-price 1975/1000)
                               (* mark-price 200/100))]
         [target-expirations (list (+days (folder-date) (* 7 1))
                                   (+days (folder-date) (* 7 2))
                                   (+days (folder-date) (* 7 3))
                                   (+days (folder-date) (* 7 4))
                                   (+days (folder-date) (* 7 5))
                                   (+days (folder-date) (* 7 6))
                                   (+days (folder-date) (* 7 8))
                                   (+days (folder-date) (* 7 9))
                                   (+days (folder-date) (* 7 10))
                                   (+days (folder-date) (* 7 11))
                                   (+days (folder-date) (* 7 12)))]
evdubs commented 4 years ago

There's a new "--all-options" (or "-a") parameter that will let you save all options rather than the default set. Please pull to get the changes.

$ racket transform-load.rkt -h
racket transform-load.rkt [ <option> ... ]
 where <option> is one of
  -a, --all-options : Save all options instead of the default select strikes and expirations

The gist is: if you want all options, you shouldn't be adding to the existing filters. get-options contains a let id all-options that already goes through the document building all of the options. The final step of get-options is to filter out the ones that, by default, are not desired. Rather than expand the filter, it can just be ignored by returning all-options. Here's a snippet of code that includes some of the changes:

         [all-options (~> (map (λ (exp-table)
                                 (map (λ (row) (list (extract-option row 0) (extract-option row -1)))
                                      ((sxpath '(td table tr)) exp-table)))
                               ((sxpath '(html body table tr td (table 9) tr)) xexp))
                          (flatten _)
                          (filter (λ (o) (not (empty? (option-underlying o)))) _)
                          (map (λ (o) (flatten-option o)) _))])
    (if (all-options?) all-options
        (flatten (map (λ (te) (let* ([e (option-expiration (closest-expiration te all-options))]
                                     [f (filter (λ (o) (equal? e (option-expiration o))) all-options)])
                                (map (λ (ts) (let ([s (option-strike (closest-strike ts f))])
                                               (filter (λ (o) (equal? s (option-strike o))) f))) target-strikes)))
                      target-expirations)))))

I haven't tested this, so let me know if it works.

evdubs commented 4 years ago

Maybe the diff helps make it clear what the change was.

CatspersCoffee commented 4 years ago

Awesome! thanks so much for that, ill run it now.

With respect to running transform-load directly on the command line on a specific directory (say yesterdays data) if i run:

racket transform-load_3.rkt -p 2020-07-08 --all-options

the program always only attempts to load the current date (host time today).. am i inputting the date incorrectly. or.. for example in a modified .sh , sorry this might be script programming 101.. but i cant seem to work it out:

#!/usr/bin/env bash

today=$(date "2020-07-08")
dir=$(dirname "$0")

racket ${dir}/transform-load.rkt -p "$1" --all-options
evdubs commented 4 years ago

Here's the full command line usage:

$ racket transform-load.rkt -h
racket transform-load.rkt [ <option> ... ]
 where <option> is one of
  -a, --all-options : Save all options instead of the default select strikes and expirations
  -b <folder>, --base-folder <folder> : OIC options chains base folder. Defaults to /var/tmp/oic/options-chains
  -d <date>, --folder-date <date> : OIC options chains folder date. Defaults to today
  -n <name>, --db-name <name> : Database name. Defaults to 'local'
  -p <password>, --db-pass <password> : Database password
  -u <user>, --db-user <user> : Database user name. Defaults to 'user'
  --help, -h : Show this help
  -- : Do not treat any remaining argument as a switch (at this level)
 Multiple single-letter switches can be combined after one `-'; for
  example: `-h-' is the same as `-h --'

In effect, you should be using -d 2020-07-08 to specify the date. -p is used to specify the database password.

evdubs commented 4 years ago

Also, I'll perhaps add this to the readmes, but the idea behind the .sh scripts is for crontab convenience. When running manually, it is recommended to just use racket extract.rkt and racket transform-load.rkt as you have more control over parameters like dates.

CatspersCoffee commented 4 years ago

Oh awesome! seems i was making a rookie mistake with -p instead of -d, thanks for clearing that up for me. Yeah with cron the .sh makes it easy. But, just in the case where i want to reload the daily .html into a new DB or re-load the last few days with all the options strikes. I would need to specify the date/folder.. this -d makes it easy to make a script.

ive run the latest oic with todays data and also the transform-load on yesterdays data and it works great! thanks for that!

With respect to the data set for where renegade takes its daily data for charts, i.e. daily, open, high, low, close prices, which project extracts this data? or just in general is there a project that takes daily, open, high, low, close prices?

thanks for modifying the TL.rkt program for --all-options, its really helped!

evdubs commented 4 years ago

With respect to the data set for where renegade takes its daily data for charts, i.e. daily, open, high, low, close prices, which project extracts this data? or just in general is there a project that takes daily, open, high, low, close prices?

iex-stocks. Note that this requires you to sign up for an account with IEX cloud. It is free, but the data does come with a 12+ hours delay.

Shall this particular issue be closed?

CatspersCoffee commented 4 years ago

Ok great, ill look at iex

Yes we can close this. thanks for your help i really appreciate it :-)