allowMagicalThinking functionality not working?

[SamoPP]

I am aware this is heresy and I am fooling myself and that quantstrat was intentionally very carefully build not to make foolish beginner mistakes like executing on the close of the same bar when the signal is generated. I know, I know, I am fooling myself. I will lose a lot of money if not all etc. I just need to do it in quantstrat.

Below please find reproducible example showing allowMagicalThinking=TRUE is not working as I expect it would (due to strange formatting I also created gist with code below: https://gist.github.com/SamoPP/75ef4e3d3e34c2428001cb36a4ee2523 ):

`# Testing quantstrat with very simple strategy using allowMagicalThinking:

1) go short 1 lot VXX on "2011-02-11" close

2) cover short on open one day after return to cash signal appeared

(go to cash == (signal =0) appeared on "2011-02-10"

#

I was playing with prefer and delay option but do not work the way I

would like them to. Is this by design and my goal can be accomplished

in some other way?

# ###############################################################################

require(PerformanceAnalytics) require(quantstrat)

signals<-function(x, date, label="signal", lag=0) { signal <- xts(rep(0, NROW(x)), order.by=index(x)) signal[date] <- -1 colnames(signal) <- label if (lag != 0) { signal <- lag.xts(signal, k=lag) } signal <- na.locf(signal) signal[is.na(signal), ] <- 0

return(signal)

}

options(width=240) Sys.setenv(TZ="UTC")

VXX.test.data.text <- " date,VXX.Open,VXX.High,VXX.Low,VXX.Close,VXX.Volume,VXX.Adjusted,magical_thinking_price,creative_thinking_price 2011-02-08,114.48,116.76,113.00,113.68,3499400,113.68,113.68,115.76 2011-02-09,114.76,116.44,113.40,114.05,4650800,114.05,114.05,113.68 2011-02-10,116.56,117.00,113.20,114.04,4433500,114.04,114.04,114.05 2011-02-11,115.00,115.20,111.72,112.08,3706500,112.08,112.08,114.04 " VXX.df <- read.csv(textConnection(VXX.test.data.text), stringsAsFactors=FALSE) VXX <- xts(VXX.df[, -1], order.by=as.Date(VXX.df[, 1]))

signal.date <- "2011-02-09"

Test if signals work

VXX.signals <- signals(VXX, signal.date, lag=-1) subset <- "2011-02"

chart_Series(VXX, subset=subset) add_TA(VXX.signals < 0, border=NA, col="lightsalmon", on=-1) add_TA(VXX.signals, type="h")

mkt.data.subset <- "2011-02-08/2011-02-11"

allowMagicalThinking <- TRUE #FALSE preferExitShort2CASH <- "Close" # "creative_thinking_price" #"magical_thinking_price" #"Close" #"Open" preferEnterSHORT <- "Close" # "creative_thinking_price" #"magical_thinking_price" #"Close" #"Open"

.qty <- 1 .th <- 0.0005 .TxnFees=-1.9 initEq <- 100000

initDate <- "2011-01-31" .from <- "2002-10-21" .to <- "2008-07-04"

.strat <- "qs.test" .portf <- "qs.test" .acct <- "qs.test"

############################# CLEAN UP ###################################### try(rm(list=ls(pos=.blotter), pos=.blotter), silent=TRUE) try(rm(list=ls(pos=.strategy), pos=.strategy), silent=TRUE)

currency("USD") stock("VXX", "USD")

initPortf(.portf, symbols="VXX", initDate=initDate) initAcct(.acct, portfolios=.portf, , initEq=initEq, initDate=initDate)

initOrders(.portf, initDate=initDate)

strategy

.strat <- strategy(.strat)

indicators

.strat <- add.indicator(.strat, 'signals', arguments=list( x=quote(VXX), date=signal.date, label="signal.enter", lag=0 ), label='signal.enter')

.strat <- add.indicator(.strat, 'signals', arguments=list( x=quote(VXX), date=signal.date, label="signal.exit", lag=0 ), label='signal.exit')

.strat <- add.signal(.strat, 'sigThreshold', arguments = list(column="signal.exit", relationship="eq", threshold=0, cross=TRUE), label='cash') .strat <- add.signal(.strat, 'sigThreshold', arguments = list(column="signal.enter", relationship="lt", threshold=0, cross=TRUE), label='short')

.strat <- add.rule(.strat, 'ruleSignal', arguments=list(sigcol='cash' , sigval=TRUE, replace=FALSE, TxnFees=.TxnFees, orderside='short', orderqty='all', ordertype='market', prefer=preferExitShort2CASH, delay=0 # We should hopefully get the Open price on the next bar if using value of 1 for this ), type='exit', label='ExitShort2CASH')

.strat <- add.rule(.strat, 'ruleSignal', arguments=list(sigcol='short' , sigval=TRUE, replace=FALSE, TxnFees=.TxnFees, orderside='short', orderqty=-.qty, ordertype='market', prefer=preferEnterSHORT, delay=0 ), type='enter', label='EnterSHORT')

applyStrategy(.strat, .portf, verbose=TRUE, allowMagicalThinking=allowMagicalThinking)

############################################################################### updatePortf(.portf, Symbols="VXX", Dates=paste('/',as.Date(Sys.time()),sep='')) updateAcct(.acct)

############################################################################### stratStats <- tradeStats(.portf, "VXX") View(t(stratStats))

stratReturns <- PortfReturns(.acct) charts.PerformanceSummary(stratReturns, wealth.index=TRUE)

###############################################################################

chart.Posn(.portf, "VXX") VXX.signals <- signals(VXX, signal.date) add_TA(VXX.signals < 0, border=NA, col="lightsalmon", on=-1) add_TA(getPortfolio(.portf)$symbols[["VXX"]]$posPL$Pos.Qty, type="h") add_TA(getPortfolio(.portf)$symbols[["VXX"]]$posPL$Pos.Value, type="h")

print(getOrderBook(.portf))

getTxns(.portf, "VXX")

mktdata

> mktdata["2011-02-08/2011-02-11"]

VXX.Open VXX.High VXX.Low VXX.Close VXX.Volume VXX.Adjusted PreviousC NextO signal.enter signal.exit cash short

2011-02-08 114.48 116.76 113.00 113.68 3499400 113.68 115.08 114.76 0 0 0 0

2011-02-09 114.76 116.44 113.40 114.05 4650800 114.05 113.68 116.56 -1 -1 0 1

2011-02-10 116.56 117.00 113.20 114.04 4433500 114.04 114.04 115.00 0 0 1 0

2011-02-11 115.00 115.20 111.72 112.08 3706500 112.08 114.04 112.00 0 0 0 0

!!! This is what I expected:

> print(getOrderBook(.portf))

$qs.test

$qs.test$VXX

Order.Qty Order.Price Order.Type Order.Side Order.Threshold Order.Status Order.StatusTime Prefer Order.Set Txn.Fees Rule

2011-02-09 "-1" "114.05" "market" "short" NA "closed" "2011-02-09 00:00:00" "Close" NA "-1.9" "EnterSHORT"

2011-02-10 "all" "114.04" "market" "short" NA "closed" "2011-02-10 00:00:00" "Open" NA "-1.9" "ExitShort2CASH"

# #

attr(,"class")

[1] "order_book"

>

> txns <- getTxns(.portf, "VXX")

> txns

Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Net.Txn.Realized.PL

2011-02-01 0 0.00 0.0 0.00 0.00 0.00

2011-02-09 -1 114.05 -1.9 -114.05 114.05 -1.90

2011-02-11 1 114.04 -1.9 114.04 114.04 -2.79

!!!!!!! But I get this:

> print(getOrderBook(.portf))

$qs.test

$qs.test$VXX

Order.Qty Order.Price Order.Type Order.Side Order.Threshold Order.Status Order.StatusTime Prefer Order.Set Txn.Fees Rule Time.In.Force

2011-02-09 "-1" "114.05" "market" "short" NA "closed" "2011-02-10 00:00:00" "Close" NA "-1.9" "EnterSHORT" ""

2011-02-10 "all" "114.04" "market" "short" NA "closed" "2011-02-11 00:00:00" "Close" NA "-1.9" "ExitShort2CASH" ""

# #

attr(,"class")

[1] "order_book"

>

> getTxns(.portf, "VXX")

Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Net.Txn.Realized.PL

2011-01-31 0 0.00 0.0 0.00 0.00 0.00

2011-02-10 -1 114.04 -1.9 -114.04 114.04 -1.90

2011-02-11 1 112.08 -1.9 112.08 112.08 0.06

The issue is clearly seen on the last graph chart.Posn .

The problem is that quantstrat takes Close price from 2011-02-11 instead from 2011-02-10.`

[SamoPP]

After some debugging I realized that the problem is in corruption of passed parameters as dots in ruleOrderProc (allowMagicalThinking becomes equal to ..3 instead of passed TRUE) using match.call:

You can also see that parameter 'Dates' expands to ..1 . I suspect some problem in the .formals list and how it is being passed, which would break any dot-expanded formal arguments, for any rule function.

If list(...) is used instead of match.call the code works properly:

[joshuaulrich]

Hi @SamoPP, is this still an issue, or did you solve (or work around) it?

[SamoPP]

@joshuaulrich Yes, this is still an issue. The only workaround I found is this: In ruleOrderProc.R in line 139 replace if(hasArg(allowMagicalThinking)) allowMagicalThinking=match.call(expand.dots=TRUE)$allowMagicalThinking

with allowMagicalThinking=list(...)$allowMagicalThinking

@braverock has the longer e-mail chain with my attempts to debug this and find the reason why this "match.call(expand.dots=TRUE)$allowMagicalThinking" is not working... I found no reason... It just doesn't work...

[braverock]

perhaps:

if(hasArg(allowMagicalThinking)) {
  allowMagicalThinking=eval(match.call(expand.dots=TRUE)$allowMagicalThinking)
}

[SamoPP]

@braverock Unfortunately, this wrapping of match.call in eval does not do the trick.

[SamoPP]

@braverock Can you please replace

allowMagicalThinking=match.call(expand.dots=TRUE)$allowMagicalThinking

with

allowMagicalThinking=list(...)$allowMagicalThinking

until a better solution is found...

I know you can not stand people making the obvious and disastrous mistake of pretending to be able to trade at the closing price of the bar the signal is being generated and depends upon. Can we just get this fixed please and we will use this functionality at our own peril. Thanks.

[SamoPP]

Hi. Any chance to get the above fix for this issue? Thanks.

[jaymon0703]

Hi @SamoPP using your gist example, were you looking to enter at the Close price on the day of the signal, 2011-02-09 or enter at the Open price on the day after (ie. 2011-02-10) the signal is generated?

[SamoPP]

@jaymon0703 These are two separate problems in quantsrtat. First one is making allowMagicalThinking to work. So one can magically/unrealistically enter at the close of the same bar the signal is generated. Second, I think independent problem, which is, I guess, much harder to solve is that prefer and delay argument is not working for OHLC data as can be easily demonstrated/replicated using the gist code example. Play with changing (commented) values for following variables preferExitShort2CASH <- "Close" # "creative_thinking_price" #"magical_thinking_price" #"Close" #"Open" preferEnterSHORT <- "Close" # "creative_thinking_price" #"magical_thinking_price" #"Close" #"Open" and see for yourself.

[jaymon0703]

Ok thanks for clarifying the 2 issues separately.

When i use prefer='Open' i see the strategy enter the position at the Open price. Please confirm if you see the same, then i will look at the AllowMagicalThinking issue you report?

For the record, my results with prefer='Open': > applyStrategy(.strat, .portf, verbose=TRUE, allowMagicalThinking=allowMagicalThinking) [1] "2011-02-10 00:00:00 VXX -1 @ 116.56" [1] "2011-02-11 00:00:00 VXX 1 @ 112.08"

With prefer='Close': > applyStrategy(.strat, .portf, verbose=TRUE, allowMagicalThinking=allowMagicalThinking) [1] "2011-02-10 00:00:00 VXX -1 @ 114.04" [1] "2011-02-11 00:00:00 VXX 1 @ 112.08"

[SamoPP]

Yes, I get the same as long delay (row 116 of the gist) is 0. As soon as you try and use delay 1 or 2, to me it looks like delay argument is disregarded/ignored. For example, when allowMagicalThinking will be working, I would like to have different prefer and delay for different rules: like enter short at the close of the bar when signal was generated and exit (cover) on open the next bar the cover short signal was generated. If I have allowMagicalThinking=TRUE then I need to use delay. So main use case is that for some rules I want to execute an order/trade on the same bar as the signal and for other rules I want to use next bar open. Typical strategies utilizing this are the ones that want to have a mean reversion long entry on stock market indices instrument like SPY on a daily time frame at the close of the bar mean reversion signal was generated but the exit is done next bar open (next to the bar exit long signal was generated) because the strategy is hoping to catch overnight "anomaly" (which presumably exists or existed). So, try and catch additional favourable overnight move for long position in SPY and exit long next day open... Hope it makes sense. :) I know it is a lousy "strategy", but, I need to replicate it first in order to see whether it has merit with more realistic fill considerations (like you can not enter at the close at the same bar signal was generated).

[jaymon0703]

Thanks @SamoPP. Tested your proposed changed which worked so it is now committed and the package version bumped to 0.16.6.

On using delay, i am copying a comment from @braverock in another chat:

"On delay, I'm not sure delay was ever supposed to work with bar data. delay is aimed at high frequency data. in theory, if we're even trying to process it on the OHLC data path, you would need a delay in seconds larger than your bar length to push potential execution into a later bar."

Hope that helps. If so, let me know if we can close this issue.

Thanks again for the report and the fix.

[SamoPP]

Thanks @jaymon0703 . I appreciate the commit.

delay is not working properly. It looks like that delay=1 when used, influences just the date of next order/trade but it takes the price from the bar before.

> applyStrategy(.strat, .portf, verbose=TRUE, allowMagicalThinking=allowMagicalThinking) [1] "2011-02-09 00:00:00 VXX -1 @ 114.76" [1] "2011-02-11 00:00:00 VXX 1 @ 114.04"

So to me it looks like the price is taken from 2011-02-10 but then transaction is on the 2011-02-11.

How would I be able to "model" the following strategy: Symbols=c("SPY")

1. Long entry: when 2 day RSI < 10 (on the close of the bar when signal is generated - so allowMagicalThinking=TRUE)
2. Long exit: when 2 day RSI > 20 (on open on the bar after the signal was generated)

Thanks for any idea.

[braverock]

As with any magical thinking, your example makes little sense.

If you want to focus on delay, let's open an issue for delay, and build a reproducible example for delay that does not rely on magical thinking.

In any case, delay should delay when an order should be put into the market. So it effectively it shifts forward the time that the order is available for execution. The price of the order is the price at the time the order is initially processed, even though it is not available for execution at this time (and potentially not at this price).

Execution of that order cannot happen until a timestamp after the order is entered (or on the same timestamp if one insists upon magical thinking). So the actual execution price may be different than the order price, depending on where the market is at the time of execution.

I suppose it might make sense to enter an order price of NA, and make the order matching code figure out an execution price, but this is tricky because the strategy may have applied some custom price offset from the observed price, and violating that again violates causality in increasingly unrealistic ways.

[braverock]

see l406-408in rules.R:

    # insert new order
    if(is.timeBased(timestamp)) ordertime<-timestamp+delay
    else ordertime<-as.POSIXct(timestamp)+delay

[jaymon0703]

So delay is perhaps not suitable for what you are trying to achieve @SamoPP. You are essentially looking for allowMagicalThinking=TRUE for entries and allowMagicalThinking=FALSE for exits, with prefer='Open'.

Perhaps you can model your strategy with a pure vectorized non-path-dependent implementation without quantstrat, using blotter if you want to retain the transaction infastructure goodness?

[braverock]

I think that delay should allow Market on Open for the following day, but this is still rather convoluted and unrealistic.

[SamoPP]

@braverock , @jaymon0703 Thanks.

Would it be possible to have allowMagicalThinking parameter per rule (in addRule), not at applyStrategy level?

[braverock]

We could consider a proper pull request and tests for such a split approach @SamoPP , but I'm reluctant to spend much more time on this. As far as I know, you are the only user of this functionality, out of thousands of users that I am aware of, including a number of institutional users.

I think @jaymon0703 's commit fixes allowMagicalThinking as it existed before changes to hasArg and match.call. Could you please validate this so that we can close this issue?

The delay parameter never worked the way you seem to think it did, as I described in my comments above. If you want to change how delay works, we could likewise consider opening a new issue, and supporting it with examples and pull requests that don't conflate the behavior of allowMagicalThinking with delay.

[SamoPP]

I confirm allowMagicalThinking now works. So I am closing this issue.

Will prepare pull request etc.