subset xts by logical matrix throws an error

[cloudcell]

Description

I get an error from the following code:

xtsTS <- xts(x=1:3, order.by=Sys.Date()+1:3)
xtsTS <- cbind(xtsTS,X=c(NA,NA,NA))
xtsTS[is.na(xtsTS)]

Error in [.xts(xtsTS, is.na(xtsTS), ) : 'i' or 'j' out of range

Is it a bug in the xts package?

Expected behavior

the code is supposed to list all the values of the xts object == NA It would also be great if it could work for is.finite() and is.infinite() and is.nan() just as an example, this works for data.frames:

df <- data.frame(bb=c(1,2,NA), cc=c(1,NA,4))
df[is.na(df)]
# this code works, probably, it would be great if xts behaved the same way

Minimal, reproducible example

library(xts)
xtsTS <- xts(x=1:3, order.by=Sys.Date()+1:3)
xtsTS <- cbind(xtsTS,X=c(NA,NA,NA))
xtsTS[is.na(xtsTS),] # either this line
xtsTS[is.na(xtsTS)] # or this line produce almost identical error messages

Session Info

> sessionInfo()
R version 3.6.3 (2020-02-29)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Linux Mint 19.3

Matrix products: default
BLAS:   /usr/lib/x86_64-linux-gnu/blas/libblas.so.3.7.1
LAPACK: /usr/lib/x86_64-linux-gnu/lapack/liblapack.so.3.7.1

locale:
 [1] LC_CTYPE=en_NZ.UTF-8       LC_NUMERIC=C               LC_TIME=en_NZ.UTF-8        LC_COLLATE=en_NZ.UTF-8    
 [5] LC_MONETARY=en_NZ.UTF-8    LC_MESSAGES=en_NZ.UTF-8    LC_PAPER=en_NZ.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_NZ.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
[1] xts_0.12-0 zoo_1.8-7 

loaded via a namespace (and not attached):
[1] compiler_3.6.3  tools_3.6.3     grid_3.6.3      lattice_0.20-40

[joshuaulrich]

Thanks for the report! I would say this is a bug, because it works with zoo objects:

library(xts)
z <- cbind(Z = zoo(1:3, Sys.Date()+1:3), X = c(NA,NA,NA))
z[is.na(z),]  # returns empty zoo object

So this should behave similarly to zoo objects instead of throwing an error.

[joshuaulrich]

Also, this isn't related to is.na(). It happens any time you subset by a logical matrix, so I'm going to update the issue title.

set.seed(21)
x <- .xts(matrix(rnorm(6),3), 1:3)
m <- coredata(x) < 0
x[m]          # error
as.zoo(x)[m]  # empty zoo object

[joshuaulrich]

It would also help to know your use case. I'm not sure zoo does what you expect either, but at least it doesn't error. We may need to create a feature request for the behavior you want.

[cloudcell]

Well, I made a workaround for my case already. I am creating various financial indicators, some of them occasionally produce values that are either infinite or not a number (NaN can be considered as NA). So I had to develop a way to treat those exceptions at the time they are calculated rather than at the time they are already sitting in the xts table. However, the expression z[is.na(z),] feels right. This is how it's supposed to work. Why would xts treat this subsetting any differently from how data frames do that?

[joshuaulrich]

This is related to #63 and #198. It does not have a obvious solution... :-/

[MaxineTheCat]

Also, this isn't related to is.na(). It happens any time you subset by a logical matrix, so I'm going to update the issue title.

set.seed(21)
x <- .xts(matrix(rnorm(6),3), 1:3)
m <- coredata(x) < 0
x[m]          # error
as.zoo(x)[m]  # empty zoo object

This no longer works w/ zoo btw (R 4.0.4 and zoo package release 1.8-8)

[joshuaulrich]

This no longer works w/ zoo btw (R 4.0.4 and zoo package release 1.8-8)

Thanks for pointing that out @MaxineTheCat. @ggrothendieck and @zeileis, was this an intended change in zoo?

[ggrothendieck]

Achim recently verified that this is due to a change in R where the class vector of matrices has been changed (extended) in R. It is not due to any change in zoo. there is more discussion here: https://stackoverflow.com/questions/66468254/error-when-performing-an-na-replacement-in-r-4-0/66488041#66488041

[zeileis]

On Fri, 5 Mar 2021, ggrothendieck wrote:

Acihm recently verified that this is due to a change in R where the class vector of matrices has been changed (extended) in R. It is not due to any change in zoo. there is more discussion here: https://stackoverflow.com/questions/66468254/error-when-performing-an-na-re placement-in-r-4-0/66488041#66488041 Regards.

The problem was that [.zoo and [<-.zoo for some reason used

all(class(i) == "matrix")

to check whether "i" is a matrix. I've replaced this by

inherits(i, "matrix")

The issue is fixed on R-Forge and I've sent it to the Win-Builder prior to making a new CRAN release.

On Fri, Mar 5, 2021 at 11:35 AM Joshua Ulrich notifications@github.com wrote:

This no longer works w/ zoo btw (R 4.0.4 and zoo package release 1.8-8)

Thanks for pointing that out @MaxineTheCat https://github.com/MaxineTheCat. @ggrothendieck https://github.com/ggrothendieck and @zeileis https://github.com/zeileis, was this an intended change in zoo?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/joshuaulrich/xts/issues/331#issuecomment-791534572, or unsubscribe https://github.com/notifications/unsubscribe-auth/AB32F7TR7F6OIHQSGCBGM7LT CEB37ANCNFSM4L7WY3TQ .

-- Statistics & Software Consulting GKX Group, GKX Associates Inc. tel: 1-877-GKX-GROUP email: ggrothendieck at gmail.com

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, orunsubscribe.[AOSLJDXVV3SU24GWQ3YMNKTTCECVPA5CNFSM4L7WY3T2YY3PNVWWK3TUL52HS4 DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOF4W6YXA.gif]

[ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/joshuaulrich/xts/issues/331#issuecomment-791538780", "url": "https://github.com/joshuaulrich/xts/issues/331#issuecomment-791538780", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]