rcyeh
commented
10 years ago On my computer, calc_OI_by_time_buckets by time buckets takes 15-16 seconds for AMZN. Since you scanned a 3-dimensional parameter space, you had 675 runs, which I can see would take 3 hours just for AMZN.
When trying to run the same for SPY, I saw the process memory jumping around 1 GB and it took too long.
I did:
> Rprof(memory.profiling=TRUE, line.profiling=TRUE)
> SOI_buckets_delta_prices <- calc_OI_by_time_buckets(time_bin,trades[which(trades$size>thres1),],bucket_size, F, L, T)
> Rprof(NULL)
> summaryRprof(lines="both")and the top results were:
$by.self
self.time self.pct total.time total.pct
"[.data.frame" 4.00 25.77 5.02 32.35
"[<-.data.frame" 3.72 23.97 3.82 24.61
parser.R#286 1.58 10.18 5.40 34.79
"as.POSIXct.POSIXlt" 0.62 3.99 1.20 7.73
"structure" 0.62 3.99 0.86 5.54
"strptime" 0.48 3.09 1.36 8.76
...
$by.total
total.time total.pct self.time self.pct
"calc_OI_by_time_buckets" 15.52 100.00 0.00 0.00
parser.R#286 5.40 34.79 1.58 10.18
"[" 5.20 33.51 0.00 0.00
"[.data.frame" 5.02 32.35 4.00 25.77
"[<-.data.frame" 3.82 24.61 3.72 23.97
"[<-" 3.82 24.61 0.00 0.00
parser.R#324 3.46 22.29 0.06 0.39
"-" 2.84 18.30 0.16 1.03
"-.POSIXt" 2.68 17.27 0.04 0.26
"difftime" 2.64 17.01 0.12 0.77
"strptime" 1.36 8.76 0.48 3.09
parser.R#245 1.36 8.76 0.00 0.00
parser.R#230 1.34 8.63 0.06 0.39
"as.POSIXct" 1.30 8.38 0.10 0.64
parser.R#247 1.28 8.25 0.04 0.26
parser.R#246 1.26 8.12 0.04 0.26
"as.POSIXct.POSIXlt" 1.20 7.73 0.62 3.99
...I'm surprised at the all the calls related to time, and reading the code, I see you are doing:
time <- strptime(trades[i+q,"time"],"%H:%M:%OS")
volume <- as.numeric(trades[i+q,"size"])
price <- as.numeric(trades[i+q,"price"])in every parsing function. In R, it is more efficient to process vectors than one-element-at-a-time. At the very least, you are doing this 675 times more than you need to. I changed these to a one-time:
a[["time"]] <- strptime(a[["time"]],"%H:%M:%OS")when you first read in the file. Further, size and price are already double-precision floating-point columns --- there's no need to as.numeric them again. I re-run profiling, and discover that there appears to be no improvement:
$by.total
total.time total.pct self.time self.pct
"calc_OI_by_time_buckets" 14.02 100.00 0.02 0.14
...and there continue to be many calls to as.POSIXct and the like. Let's go one step further:
a[["time"]] <- as.integer(as.POSIXct(strptime(a[["time"]],"%H:%M:%OS")))since I don't think you use the microsecond resolution in that column. The result is better, mostly probably due to memory caching:
$by.total
total.time total.pct self.time self.pct
"calc_OI_by_time_buckets" 8.12 100.00 0.00 0.00
"[" 4.00 49.26 0.02 0.25
"[.data.frame" 3.70 45.57 2.80 34.48
parser.R#286 3.06 37.68 1.04 12.81
"[<-.data.frame" 2.04 25.12 1.96 24.14
"[<-" 2.04 25.12 0.00 0.00
parser.R#245 1.08 13.30 0.12 1.48
parser.R#230 0.84 10.34 0.06 0.74
parser.R#246 0.82 10.10 0.08 0.99
parser.R#247 0.80 9.85 0.08 0.99
parser.R#324 0.42 5.17 0.12 1.48
"%in%" 0.38 4.68 0.16 1.97
parser.R#228 0.26 3.20 0.06 0.74
Next, I'm not sure what to do with all of the "[" and data.frame costs --- I think those are due to the element-by-element access done, some of which may be unavoidable due to the VPIN algorithm needing to maintain state. You'd have to do a wholesale rewrite, adding columns to track the running volume to be assigned to the next bucket.
Let's put that aside for now. The next most-expensive area is line 286, which is:
trades[i+q,"size"] <- volume_count + volume - bucket_volume_sizeThat looks innocuous, but I found that by rewriting every one of your trades[i+q,"col"] constructs as trades$col[i+q], I saved another 40% of time. I'm now at:
$by.self
self.time self.pct total.time total.pct
"$<-.data.frame" 1.76 53.33 1.80 54.55
parser.R#286 0.52 15.76 2.32 70.30
parser.R#247 0.18 5.45 0.18 5.45
parser.R#230 0.12 3.64 0.12 3.64
parser.R#245 0.08 2.42 0.08 2.42
...
$by.total
total.time total.pct self.time self.pct
"calc_OI_by_time_buckets" 3.30 100.00 0.00 0.00
parser.R#286 2.32 70.30 0.52 15.76
"$<-.data.frame" 1.80 54.55 1.76 53.33
"$<-" 1.80 54.55 0.00 0.00
"$" 0.22 6.67 0.00 0.00
parser.R#247 0.18 5.45 0.18 5.45
parser.R#230 0.12 3.64 0.12 3.64
parser.R#245 0.08 2.42 0.08 2.42
"var" 0.08 2.42 0.02 0.61
"assign_buy" 0.08 2.42 0.00 0.00which I have committed below at aa4e93cbb2759468706854cc8e5b80a3a3d70f24.
Regarding parser.R lines 40--54: it seems that the sense of the comparison is opposite in lines 40-46 than 48-54. I would guess that lines 41 and 43 are wrong --- would you check?
Regarding parser.R lines 233-239: just to check, you normally don't call this function with just trades, right? Otherwise, the assign_buy function later won't have a quote to use. I think you can get combine the i and the q variables. (By the way, q is also a function allowing you to quit R.)
Regarding parser.R line 300-306: instead of querying the signed variable on every pass through the loop, you can just run abs() on the OI_buckets variable at the end. This only saves space but doesn't save time.
Regarding parser.R line 326: you don't need to check that i == length(trades[,1]) because you discard the information calculated anyway. Removing that check saves about a quarter second.
zc238
jx226
For SOI parser performance enhancement. The code is isolated at RScripts/parserPerfromanceTest.R
It is necessary to source parser.R first for all the necessary functions. Then simply run every line from the file.
Two main functions are called inside the loops to determine the best combinations of bucket size/time bin/thres hold for excluding trades that can achieve the highest R^2.
Side note: For SPY, I had to add the following line (by default, you can assess the memory limit R uses by calling memory.limit() on a windows machine):
invisible(utils::memory.limit(7987))
in R//etc/Rprofile.site in order to obtain enough memory, I am not sure what is the equivalent construct for a Mac.
Appreciate the help!