Plot multi-day minute-level multi-session time series data

[renkun-ken]

I have a dataset that contains a time series of minute-level financial index data for a number of trading days. It's common across the world that the stock markets only run in day time, producing lots of quote records with segmented date/time. The time-stamps in my dataset, for example, are from 2015-03-02 to 2015-06-30 at day-scale, but at minute-scale, it's from 09:30:00 to 11:30:00, and from 13:00:00 to 15:00:00. Each day has several trading sessions.

Using dygraphs to plot such multi-day data with several non-continuous intra-day sessions, the graphics are forced to reflect the data/time in-between non-trading days and non-trading sessions, which is quite distracting and make the plot less useful.

Is there any existing option or method to make the plot continuous and do not reflect non-trading time?

[jjallaire]

You might need to pad the time-series out to reflect all possible intervals and then ensure that the values for inserted periods are missing (NA).

On Sun, Jul 12, 2015 at 4:20 AM, Kun Ren notifications@github.com wrote:

I have a dataset that contains a a time series of minute-level financial index data for a number of trading days. It's common across the world that the stocks markets only run in day time, producing lots of quote records with segmented date/time. The time-stamps in my dataset, for example, are from 2015-03-02 to 2015-06-30 in day-scale, but at minute-scale, it's from 09:30:00 to 11:30:00, and from 13:00:00 to 15:00:00. Each day has several trading sessions.

Using dygraphs to plot such multi-day data with several non-continuous intra-day sessions, the graphics are forced to reflect the data/time in-between non-trading days and non-trading sessions, which is quite distracting and make the plot less useful.

[image: dygraphs-plot1] https://cloud.githubusercontent.com/assets/4662568/8637034/a9cbaf06-28b1-11e5-8e73-eeb0c40e1e2b.png

Is there any existing option or method to make the plot continuous and do not reflect non-trading time?

— Reply to this email directly or view it on GitHub https://github.com/rstudio/dygraphs/issues/70.

[shrektan]

I think what @renkun-ken means is to make the x-axis to hide the non-value point.

[renkun-ken]

@shrektan points out what I want exactly. Here's an example taking @jjallaire's suggestion.

library(dygraphs)
library(data.table)

z <- rbind(
  data.table(t=as.POSIXlt("2015-07-01 09:30:00") + 0:359, y = cumsum(rnorm(360))),
  data.table(t=as.POSIXlt("2015-07-01 11:30:00") + 0:359, y = cumsum(rnorm(360))))
setkey(z, t)
z <- z[data.table(t=seq(min(z$t), max(z$t), by = 1), key = "t")]

dygraph(z)

It is no much more meaningful to view such a padded time-series plot than the original one without joining a full-length time-axis. To make a continuous plot, I have to overwrite the existing data/time values to continuous values.

z[, t := as.Date("2015-07-01") + seq_along(y)]
dygraph(z)

And the x-axis becomes meaningless though.

[algoquant]

I support Kun's request for removing the gaps in dygraphs plots of overnight data. Plotting intraday time series data without displaying overnight data gaps is a high priority for me. The function chartSeries from package quantmod produces very nice plots without plotting overnight data gaps, but it's not interactive. It's possible to wrap chartSeries in shiny, to make it interactive, but it's not as elegant as dygraphs. I do realize that the dygraphs package is an interface to the dygraphs js library, but I don't think that this would require changing the js code, just changing the data that R passes to js.

[edward-wilson]

I support this request! It is also an issue with weekends on longer time frame data sets! Essentially makes DY very difficult to use for most if not all financial time series data sets.Also, having the ability to remove the vertical "connectors" on a step plot would be nice, or if there are large vertical gaps in data sets such as the one pictured above would more accurately reflect financial data.

[londonrockslive]

I support this request. The work around of adding in NA's to "fill the gaps" allows you to get rid of the connecting lines as in renkun-ken's top graph but results in the gaps as in renkun-ken's second graph. I think this is to do with dygraphs interpolating a continuous axis that results in it just plotting empty space when there is no data to fill it. For financial time series this will basically mean you end up with ugly gaps in the evenings/weekends.

[londonrockslive]

It's a really significant issue with an otherwise great package. :(

[jjallaire]

Have you tried dyOptions(connectSeparatedPoints = FALSE)

On Fri, Nov 11, 2016 at 4:00 AM, londonrockslive notifications@github.com wrote:

I support this request. The work around of adding in NA's to "fill the gaps" allows you to get rid of the connecting lines as in renkun-ken's top graph but results in the gaps as in renkun-ken's second graph. I think this is to do with dygraphs interpolating a continuous axis that results in it just plotting empty space when there is no data to fill it. For financial time series this will basically mean you end up with ugly gaps in the evenings/weekends.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

[londonrockslive]

Yes. dyOptions(connectSeparatedPoints = TRUE) causes it to try and draw a line between the points either side of the "gap", dyOptions(connectSeparatedPoints = FALSE) causes it to leave the gap blank like in renkun-ken's examples.

I don't think this has a solution in R, will probably require looking at the javascript, sadly I do not know the language.

[edward-wilson]

Can you create an index column (1:length(x)) and plot the x axis from that. You can then generate the x axis labels from the original timestamp.

Bit of a hack. I ended up implementing this work around in plotly (maybe you can do the same in DY).

[londonrockslive]

I have been able to plot with a new index and that does remove the gaps. When I tried it I didn't see a way to change the x-axis labels though but I can look into it again.

However, a biggest issue would be that if that was to work you would then not have the date/time on the tooltip for your points but the index instead I think.

On 12 Nov 2016 1:25 a.m., "edward-wilson" notifications@github.com wrote:

Can you create an index column (1:length(x)) and plot the x axis from that. You can then generate the x axis labels from the original timestamp.

Bit of a hack. I ended up implementing this work around in plotly (maybe you can do the same in DY).

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/rstudio/dygraphs/issues/70#issuecomment-260091914, or mute the thread https://github.com/notifications/unsubscribe-auth/ASzjEvZG_Wsxp7FfeMsoRu_MqWCBc20zks5q9RWcgaJpZM4FW4uE .

[londonrockslive]

Hi I tried the above and it seems like a dead end. You could maybe do something with synchronized graphs and I am attempting to but I think it will be very messy and likely a dead end.

[londonrockslive]

Synchronized graphs also a dead end. Do you think supporting discontinuous time axis is something that could be supported in future?

[jjallaire]

I'd say the answer then would be write a custom plotter (which we just introduced in the last release). Here's how custom plotters are called from R:

https://rstudio.github.io/dygraphs/gallery-custom-plotters.html

And here is the related JavaScript implementation:

https://github.com/rstudio/dygraphs/tree/master/inst/examples/plotters

If someone comes up with something general we'd be happy to incorporate it into the package proper.

On Sun, Jul 12, 2015 at 4:20 AM, Kun Ren notifications@github.com wrote:

I have a dataset that contains a a time series of minute-level financial index data for a number of trading days. It's common across the world that the stocks markets only run in day time, producing lots of quote records with segmented date/time. The time-stamps in my dataset, for example, are from 2015-03-02 to 2015-06-30 in day-scale, but at minute-scale, it's from 09:30:00 to 11:30:00, and from 13:00:00 to 15:00:00. Each day has several trading sessions.

Using dygraphs to plot such multi-day data with several non-continuous intra-day sessions, the graphics are forced to reflect the data/time in-between non-trading days and non-trading sessions, which is quite distracting and make the plot less useful.

[image: dygraphs-plot1] https://cloud.githubusercontent.com/assets/4662568/8637034/a9cbaf06-28b1-11e5-8e73-eeb0c40e1e2b.png

Is there any existing option or method to make the plot continuous and do not reflect non-trading time?

— Reply to this email directly or view it on GitHub https://github.com/rstudio/dygraphs/issues/70.

[londonrockslive]

I unfortunately do not know javascript so won't be able to do this. It doesn't look too impenetrable though so when I have free time I may give learning it a go and see if I can create a custom plotter that fixes the issue. May take me a while to get round to it though...

[jjallaire]

Okay, if you follow the example custom plotters I don't think it will be too difficult.

On Mon, Nov 14, 2016 at 8:19 AM, londonrockslive notifications@github.com wrote:

I unfortunately do not know javascript so won't be able to do this. It doesn't look too impenetrable though so when I have free time I may give learning it a go and see if I can create a custom plotter that fixes the issue. May take me a while to get round to it though...

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/rstudio/dygraphs/issues/70#issuecomment-260332346, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGXxwy9JnnzLBJJCgfhf_-198TgOismks5q-F_lgaJpZM4FW4uE .

[londonrockslive]

We (members of my team who know javascript) have tried fiddling about with custom plotters a bit with no luck. Don't think this is an "easy" fix unfortunately. Though I hope to be proven wrong!

Is implementing this in the pipeline?

[galiak11]

I support this request. Financial applications are not the only use case for it. We analyze flight data that is polled every second, but have no interest in data that is in between takeoffs. This data may not be 0 or missing. What we need is a way to specify rules or ranges of X values that are not to be plotted (plus, have some visual indication on the graph of missing data in between the valid data (for instance: 3 dots: ...) ).

[dgreenberg3]

Wondering if anyone has made any progress on this problem. We need to implement this for discontinuities in environmental data. Thanks.

[rainervi]

Hi londonrockslive,

have you found a solution? I ran into the same problem.

Thank you Rainer

[richwu]

Same request. I expect this is an opportunity to make dygraph a very outstanding differentiation from other tool/framework. Strongly recommend the team take seriously care of it.

[przmv]

Ok, I'll have a look soon(ish). Could please anybody provide me with a Minimal Working Example (https://www.r-bloggers.com/writing-a-minimal-working-example-mwe-in-r/)?

[acezzz]

Same request as well. This is a must for financial related charting.

[klaukh]

From what I understand of this issue, I think it can be accomplished by first translating the date/time data to a numeric... since the dygraph can take a numeric column for the x axis (has to be the first column in the data.frame)... then use the axisLabelFormatter param in dyAxis to transform the numeric value into string that displays the date and time.

axisLabelFormatter takes a javascript function... create the function as text then wrap as JS(txt)... as the value. This should really simplify the complexity of the JS code required, and requires no changes to the package as it stands.

[sdittmar]

It really is a pity that DyGraph is effectively useless for financial data. It looks like such a promising library for Shiny because it is fast and supports shading! For any serious financial user it needs to plot volume and no time gaps or it is a no-go. I spend one entire day trying if I could use this hack to get around limitations, but I reached my limit. It seems like it can be done. I hope the link can be useful for someone capable in JS. Plotly is struggling with proper financial charts. That means there is no alternative to Highcharter for R-users.

If this request would receive more attention, that would be great.

[helgasoft]

@sdittmar - agree 100%! Many financial data users abandon the library due to this tiny detail. It could/should be fixed permanently with a boolean option in dyOptions().

✅ here is a workaround hack to fix date-only series.

Yet the attempt to include Date+Time series was unsuccessful. It seems the problem is in the JS environment behind the formatter. For instance, new Date("2019-01-25 09:31:00") produces Invalid Date and new Intl.DateTimeFormat() generates ReferenceError: Can't find variable: Intl. Hope someone could dig deeper.