xaxis labels with intraday data

[priikone]

I'm planning on using techanjs in day trading platform and I'm trying to get everything work the way I want. One issue that I've come across is the xaxis labels with intraday data when the day changes. For as long as there are candles visible from the previous day, ticks don't appear at all in the new day (or previous day). If I zoom in to the current day, then suddenly ticks and labels appear.

Here's how it looks before zooming:

And after zooming:

I of course want the ticks and labels to be visible, as the previous day is always visible in day trading, at least at the beginning of the trading day.

How can I get the ticks rendered even if the previous day's candles are visible?

[priikone]

I played around with financetime's ticks() and it certainly is capable of generating more ticks just fine, it's just that the current intraday logic doesn't really work correctly with real intraday trading charts. The desired output would be to have day ticks on day changes and normal intraday ticks (HH:MM) on other ticks. This means not only detecting intraday charts with more than one day's worth of data, but to also detect the day changes to be able to render the day label on those ticks.

[jonathanstrong]

I spent some time working on this today. I am partway there but can't figure out a way to do everything that is needed.

I changed the scale.ticks function in financetime.js to as follows:

    function tickMethod(visibleDomain, indexDomain, count) {
      if(visibleDomain.length == 1) return genericFormat; // If we only have 1 to display, show the generic tick method

      var visibleDomainIncrement = visibleDomain[1] - visibleDomain[0],
        intraday_data = visibleDomainIncrement/dailyStep < 1, // Determine whether we're showing daily or intraday data
        visibleDomainExtent = visibleDomain[visibleDomain.length-1] - visibleDomain[0],
        days_visible = visibleDomainExtent/dailyStep,
        intraday = (intraday_data & days_visible < 3),
        methods = intraday  ? tickMethods.intraday : tickMethods.daily,
        tickSteps = intraday ? intradayTickSteps : dailyTickSteps,
        k = Math.min(Math.round(countK(visibleDomain, indexDomain)*count), count),
        target = visibleDomainExtent/k, // Adjust the target based on proportion of domain that is visible
        i = d3_bisect(tickSteps, target);

      if ( i == methods.length ) { // return the largest tick method
        return methods[i-1]; 
      }
      else {
        if ( i ) {
          //try to search index j (i +/- 1) for
          //tickSteps[j]/target ratio closest to 1
          var diffs = [];
          [i-1, i, i+1].forEach(function(j){
              diffs.push([j, Math.abs(1-tickSteps[j]/target)]);
          });
          diffs.sort(function(a, b){
              return a[1]-b[1];
          });
          return methods[diffs[0][0]]; 
        }
        else {
          return methods[0];
        }
      }
    }

It chooses the intrday ticks/methods if the step between visibleDomain[0] and visibleDomain[1] is less than dailyStep, instead of the old way of checking whether the range of date values was more than one day.

This works insofar as the chart can display intraday ticks for a period of more than one day. I set it to any period under three days in the code, but that could be tweaked.

However, a couple remaining problems:

• I can't figure out how to work tickFormat so that it checks for whether the intraday value is the beginning of a new day (and hence should be processed with dayFormat rather than intradayFormat).

  The formats are defined this way:

  var dayFormat = d3_time.format('%b %e'),
    yearFormat = d3_time.format.multi([
      ['%b %Y', function(d) { return d.getMonth(); }],
      ['%Y', function() { return true; }]
    ]),
    intradayFormat = d3_time.format.multi([
      [":%S", function(d) { return d.getSeconds(); }],
      ["%I:%M", function(d) { return d.getMinutes(); }],
      ["%I %p", function () { return true; }]
    ]),
    genericFormat = [d3_time.second, 1, d3_time.format.multi([
        [":%S", function(d) { return d.getSeconds(); }],
        ["%I:%M", function(d) { return d.getMinutes(); }],
        ["%I %p", function(d) { return d.getHours(); }],
        ['%b %e', function() { return true; }]
      ])
    ];
  

  The d3.time.format.multi method does not allow you to inspect anything other than the date itself, so you can't check like, visibleDomain[i-1] to see if it's another day than the current date being processed.

• Secondly, in testing this out there were times when the intraday ticks were close enough to daily ticks that they were overlapping. I don't know how to fix this.

Any thoughts on how to get this to the finish line?

[priikone]

Good. I've actually worked around this problem by introducing a new option for financetime() which allows me to tell the scale that it's intraday data (no matter how many days worth of data is included). It was just the simplest way to do it.

Then I hacked together some logic to show more ticks. It's definitely not optimal solution, but for now I don't care. I also reverted 497de40a202c8d69b56688cedac5b78b09d02cd4 which messes up the ticks for data with gaps.

I haven't yet included it anywhere (and I doubt it will ever get to official techanjs). Here's the diff:

diff --git a/src/scale/financetime.js b/src/scale/financetime.js
index 4e037b2..a4e51f9 100644
--- a/src/scale/financetime.js
+++ b/src/scale/financetime.js
@@ -6,7 +6,7 @@
  and weekends respectively. When plot, is done so without weekend gaps.
  */
 module.exports = function(d3_scale_linear, d3_time, d3_bisect, techan_util_rebindCallback, scale_widen, zoomable) {  // Injected dependencies
-  function financetime(index, domain, padding, outerPadding, zoomLimit) {
+  function financetime(index, domain, padding, outerPadding, zoomLimit, isIntraDay) {
     var dateIndexMap,
         tickState = { tickFormat: dailyTickMethod[dailyTickMethod.length-1][2] },
         band = 3;
@@ -16,6 +16,7 @@ module.exports = function(d3_scale_linear, d3_time, d3_bisect, techan_util_rebin
     padding = padding === undefined ? 0.2 : padding;
     outerPadding = outerPadding === undefined ? 0.65 : outerPadding;
     zoomLimit = zoomLimit || index.domain();
+    isIntraDay = isIntraDay === undefined ? false : isIntraDay === true;

     /**
      * Scales the value to domain. If the value is not within the domain, will currently brutally round the data:
@@ -112,7 +113,7 @@ module.exports = function(d3_scale_linear, d3_time, d3_bisect, techan_util_rebin
     }

     scale.copy = function() {
-      return financetime(index.copy(), domain, padding, outerPadding, zoomLimit);
+      return financetime(index.copy(), domain, padding, outerPadding, zoomLimit, isIntraDay);
     };

     /**
@@ -165,10 +166,10 @@ module.exports = function(d3_scale_linear, d3_time, d3_bisect, techan_util_rebin

       if(!visibleDomain.length) return []; // Nothing is visible, no ticks to show

-      var method = interval === undefined ? tickMethod(visibleDomain, indexDomain, 10) :
-                    typeof interval === 'number' ? tickMethod(visibleDomain, indexDomain, interval) : null;
+      var method = interval === undefined ? tickMethod(visibleDomain, indexDomain, 10, isIntraDay) :
+                    typeof interval === 'number' ? tickMethod(visibleDomain, indexDomain, interval, isIntraDay) : null;

-      tickState.tickFormat = method ? method[2] : tickMethod(visibleDomain, indexDomain, 10)[2];
+      tickState.tickFormat = method ? method[2] : tickMethod(visibleDomain, indexDomain, 10, isIntraDay)[2];

       if(method) {
         interval = method[0];
@@ -204,21 +205,30 @@ module.exports = function(d3_scale_linear, d3_time, d3_bisect, techan_util_rebin
     return (Math.abs(linear(domain.length-1) - linear(0))/Math.max(1, domain.length-1))*(1-padding);
   }

-  var dayFormat = d3_time.format('%b %e'),
+  var prev_date;
+  var dayFormat = d3_time.format('%m/%e'),
       yearFormat = d3_time.format.multi([
-        ['%b %Y', function(d) { return d.getMonth(); }],
+        ['%b', function(d) { return d.getMonth(); }],
         ['%Y', function() { return true; }]
       ]),
       intraDayFormat = d3_time.format.multi([
-        [":%S", function(d) { return d.getSeconds(); }],
-        ["%I:%M", function(d) { return d.getMinutes(); }],
-        ["%I %p", function (d) { return true; }]
+        ["%m/%e", function(d) {
+           if (prev_date !== undefined && d.getDate() != prev_date.getDate()) {
+               prev_date = d;
+               return true;
+           }
+           prev_date = d;
+           return false;
+        }],
+        ["%H:%M:%S", function(d) { prev_date = d; return d.getSeconds(); }],
+        ["%H:%M", function(d) { prev_date = d; return d.getMinutes(); }],
+        ["%H:%M", function (d) { prev_date = d; return true; }]
       ]),
       genericTickMethod = [d3_time.second, 1, d3_time.format.multi([
-          [":%S", function(d) { return d.getSeconds(); }],
-          ["%I:%M", function(d) { return d.getMinutes(); }],
-          ["%I %p", function(d) { return d.getHours(); }],
-          ['%b %e', function() { return true; }]
+          ["%H:%M", function(d) { return d.getSeconds(); }],
+          ["%H:%M", function(d) { return d.getMinutes(); }],
+          ["%H:%M", function(d) { return d.getHours(); }],
+          ['%m/%e', function() { return true; }]
         ])
       ];

@@ -248,10 +258,12 @@ module.exports = function(d3_scale_linear, d3_time, d3_bisect, techan_util_rebin
     3e5,    // 5-minute
     9e5,    // 15-minute
     18e5,   // 30-minute
-    36e5,   // 1-hour
-    108e5,  // 3-hour
-    216e5,  // 6-hour
-    432e5,  // 12-hour
+    1*36e5, // 1-hour
+    2*36e5, // 2-hour
+    3*36e5, // 3-hour
+    4*36e5, // 4-hour
+    6*36e5, // 6-hour
+    12*36e5,// 12-hour
     864e5   // 1-day
   ];

@@ -265,7 +277,9 @@ module.exports = function(d3_scale_linear, d3_time, d3_bisect, techan_util_rebin
     [d3_time.minute, 15, intraDayFormat],
     [d3_time.minute, 30, intraDayFormat],
     [d3_time.hour, 1, intraDayFormat],
+    [d3_time.hour, 2, intraDayFormat],
     [d3_time.hour, 3, intraDayFormat],
+    [d3_time.hour, 4, intraDayFormat],
     [d3_time.hour, 6, intraDayFormat],
     [d3_time.hour, 12, intraDayFormat],
     [d3_time.day, 1, dayFormat]
@@ -281,17 +295,27 @@ module.exports = function(d3_scale_linear, d3_time, d3_bisect, techan_util_rebin
     return visibleDomain.length/(indexDomain[indexDomain.length-1]-indexDomain[0]);
   }

-  function tickMethod(visibleDomain, indexDomain, count) {
+  function tickMethod(visibleDomain, indexDomain, count, isIntraDay) {
     if(visibleDomain.length == 1) return genericTickMethod; // If we only have 1 to display, show the generic tick method

     var visibleDomainExtent = visibleDomain[visibleDomain.length-1] - visibleDomain[0],
-        intraDay = visibleDomainExtent/dailyStep < 1, // Determine whether we're showing daily or intraday data
+   gap = visibleDomainExtent/dailyStep,
+        intraDay =  isIntraDay === true || gap < 1, // Determine whether we're showing daily or intraday data
         tickMethods = intraDay ? intraDayTickMethod : dailyTickMethod,
         tickSteps = intraDay ? intraDayTickSteps : dailyTickSteps,
         k = Math.min(Math.round(countK(visibleDomain, indexDomain)*count), count),
         target = visibleDomainExtent/k, // Adjust the target based on proportion of domain that is visible
         i = d3_bisect(tickSteps, target);

+    prev_date = undefined;
+
+    if (intraDay && gap > 0.9 && gap < 3.4)
+      i = 8;
+    if (intraDay && gap > 0.85 && gap <= 0.9)
+      i = 7;
+    if (intraDay && gap > 0.67 && gap <= 0.85)
+      i = Math.max(i - 3, 6);
+
     return i == tickMethods.length ? tickMethods[i-1] : // Return the largest tick method
       i ? tickMethods[target/tickSteps[i-1] < tickSteps[i]/target ? i-1 : i] : tickMethods[i]; // Else return close approximation or first tickMethod
   }

Hopefully this gives you some more ideas how to do it actually right, at least it shows you how to work with the tick generation. However, one thing to note is that, please try to avoid doing a solution which has to iterate through the data because it will be too slow in real life use cases.

[jonathanstrong]

Hey - thanks for sending the diff. The intradayFormat multi pattern really helped a lot.

However, I continue to have problems with overlapping axis labels. Stuff like this

It seems like the fundamental mechanism the code is using to determine tick placement does not work for a multi-day intraday precision view. From what I gather it is selecting ticks of equal distance apart and then adding a tick for the beginning of a day on top of those equal distance ticks.

I think the logic needs to be, start with a tick per new day, then select n ticks between each day such that total ticks is close to desired amount. Honestly, though, I haven't gotten into the weeds enough to figure out how the actual tick methods are working.

This is where I am now:

https://github.com/jonathanstrong/techan.js/blob/ee18dad7922410ec7212fe28ff3b8b4b1297ea38/src/scale/financetime.js

[priikone]

Did you revert 497de40? With that commit it I had overlapping ticks, too. Get rid of it, I opened issue https://github.com/andredumas/techan.js/issues/108 about it earlier.

[jonathanstrong]

Yes - I did and forgot to mention that. Still having problems despite reverting that commit.

[priikone]

I don't see that problem with my diff. Might be related to your own changes, me thinks. It does strike me familiar but I can't remember if I did anything to get rid of that other than revert that one commit.

Where does your data come from?

[jonathanstrong]

The raw data I'm using is time & sales tick data from CQG Data Factory that I resampled with Pandas to various levels of precision. The data goes from 9:30am to 4:00pm with no gaps/nan entries in between trading hours.

I looked at your fork and couldn't find the branch/commit with the code from the diff you sent. Is it available? Would help to be able to peruse the source code rather than looking at a diff only. Then I could also fetch it and run a diff against my own code.

[priikone]

As I said earlier my change is not yet in anywhere. But that diff is the only change I've done to financetime scale.

[priikone]

I think, if memory serves, I had the same problem until I added the isIntraDay logic, so that I will only ever deal with the intradayTickMethod and never with the dailyTickFormat despite what zooming level is used. Dunno. I guess you could apply my diff and see how it works with your data, and give isIntraDay as true to financetime(). The diff should apply cleanly after reverting 497de40.

[NickStefan]

@priikone @jonathanstrong do either of you have an example of:

updating chart feed of minute data?

Ive tried both this library and d3cf, and it seems that everything is geared to daily time scales! I just cannot get a clean example to work with minute dates. Basically, on every data update, the candles grow fatter in scale, dont show all of the ticks correctly, or the whole chart awkwardly shifts over to the right.

[NickStefan]

I should clarify that the minute data updating charts work fine when they start out with 100s of data points. Im finding they dont work well when they start with 0 and then attempt to incremently add

[andredumas]

Implementing financetime to be as generic and 'smart' as possible when it came to deriving ticks from irregular data was quite a chore. It's probably going wrong here and here guessing incorrectly and hanging onto intraday. I do see the issue here too http://bl.ocks.org/andredumas/17be8c0973ac92acd6e5

The logic is quite tedious and ultimately there is an API the user can define the ticks. It needs to be reworked and will be at some stage, for me I could live with the ticks being out as long as the plot was ok.