Highlight weekends / interval example

[EmiPhil]

Another take on the weekend/interval plugin.

119 #126

[leeoniya]

looks like this relies on walking the data. what if the data is sparse? what if there are 1M points?

a day highlighting plugin should require nothing more than the u.scales.x.min/max (timestamps), uPlot.tzDate() and u.bbox. any plugin which has to inspect the data will be some mix of very broken and very slow.

i would accept a plugin that satisfies the above requirements, but does not account for DST shifts (which would make it much more complex). the general idea is pretty simple:

• take the x scale min & create a Date object for it using tzDate
• using that info compute the ts of the start of the last occurance of the period(s) you want to highlight. you'll now have a ts smaller than the scale's min.
• work forwards from that to create the highlighted regions up to scale max by just adding some fixed # of seconds to the base ts (since we're not handling DST stuff, this should work okay)

[EmiPhil]

This solution walks the number of days from the left timestamp to the right timestamp regardless of how many points there are. I think sparse data is a non issue from my experience with u.valToPos but I'd need to experiment more with it. Given the input array of days to highlight, it could be a bit more efficient by jumping 86400 * days until next highlight which is definitely a todo. I'd be interested in the use case where someone is looking at 1M days (2.7k years?) and also wants to highlight weekends.

I think what makes highlighting by a particular day of the week an interesting challenge is that the size of the highlights will not all be the same based on DST. If you ignore DST and do a simple width interval, you are doing zebra striping (itself a useful feature and could be a plugin).

Saying the width of the zebra striping is 24 hours worth and started by an offset of x seconds is a third use case (the one you suggest), but I wouldn't associate it with highlighting days because for most users it will be wrong half the time.

Maybe the best way forward would be an interval demo with both zebra striping and day (DST) striping. I can't personally think of a use case that would want the 24hour offset-by-x-days zebra striping that didn't really mean highlight by day, but I can think of examples that might want to offset the zebra stripes by some x amount so that could be in the examples.

[leeoniya]

sorry, i still don't follow.

your demo is trivially easy to break because it relies on datapoints being at hour 0 of every day. shifting the dataset by 5 hours screws it up: for (let hour = 5; hour < 24; hour += 6) {. a valid plugin will work properly over randomly-generated, randomly-spaced data, with a random min and a random max.

the task is to highlight along the x scale. the data there literally does not matter and should not be used. this is no different a problem than uplot's tick generation along x which cannot and does not rely on the data. all that matters is the scale range and what needs to be located along it.

[EmiPhil]

Hmm I was wrongly assuming that u.valToPos would give where the value would be if it were there if the data is sparse. Would have been a convenient shortcut!

There are 2 tasks I think? Highlight uniformly (data doesn't matter) and highlight non-uniformly based on the data (data does matter). I see your point of view that given the first timestamp we could avoid accessing the data after, though, because the properties of time can be known in advance (only applies when x is a timeseries?)

[leeoniya]

hopefully the API docs are clear about what .valToPos() does:

https://github.com/leeoniya/uPlot/blob/e31a2793c5c5942f08f0c6cc47a54f4e06163409/dist/uPlot.d.ts#L92-L95

it gives you a pixel offset (css or canvas) for an arbitrary value along a scale. currently all scales are linear, but when log scales are implemented, it'll have to account for that as well. all scales are continuous, so there's no such concept as sparse, which is only a property of the data.

There are 2 tasks I think? Highlight uniformly (data doesn't matter) and highlight non-uniformly based on the data (data does matter).

both issues you linked to discuss specifically the task of highlighting along a scale.

yes, data highlighting is also useful (like anomaly detection), and obviously that needs to scan the data. these two tasks are completely orthogonal to each other.

i don't think "uniformity" has anything to do with it. yes, you'll probably encounter uniformity on a time scale because the calendar is cyclical. but you may also want to highlight various known events along a time scale like "the great depression", "Vietnam war", which are both non-cyclical, and probably cannot be inferred from the data.

[EmiPhil]

Oh so my assumption was right?

I just put in for (let hour = 5; hour < 24; hour += 6) { into the example code and I'm not sure I get what you originally meant when you said it gets messed up.

When I zoom in after that change the highlights are still from 12am - 12 am Saturday to Sunday? That's exactly the behavior I thought we were trying to achieve :joy: In particular I always zoom into the interval at 03/10 because you can really see the highlighter respecting DST.

I'm totally lost now. I obviously haven't spent enough time on charting stuff because my terminologies must be all messed up. By uniform I mean that the width of each interval is or isn't the same. In the case of highlighting day of the week, the widths of the days won't be the same if there is DST.

Reference screenshot:

Edit: Interestingly, +5 is exactly the change needed to make the other side of the DST land perfectly on the other side! Here it is with +7 for extra sparseness:

[EmiPhil]

I should also mention that if you don't consider u.valToPos to be "accessing the data", then this method actually doesn't access the data except for the max/min. So that's probably another thing I'm confused by.

One thing it isn't doing is finding the period before the start of the timeseries so I guess that's another TODO.

[EmiPhil]

Out of curiosity I changed that section to:

    mos.forEach((month, monthIdx) => {
      for (let day = 1; day < new Date(year, monthIdx + 1, 0).getDate(); day++) {
        for (let hour = 0; hour < 24; hour += 1) {
          for (let m = 0; m < 60; m += 1) {
            for (let s = 0; s < 60; s += 30) {
              let dateStr = day + " " + month + " " + year + " " + hour + ":" + m + ":" + s + " UTC";
              ts.push(Date.parse(dateStr) / 1000);
              dataSetA.push(vals[mos.length - monthIdx - 1]);
              dataSetB.push(vals[monthIdx]);
            }
          }
        }
      }
    });

which I think generates 2 60 24 * 365 = 1,051,200 (>1M data points) and I didn't notice a difference in render time. So both sparse and 1M data points works with this method.

[EmiPhil]

Hmmmm regarding my comment that I'm not finding the period before the first timestamp I'm not sure it's necessary. The plugin is for sure checking if the first timestamp is in an interval, and it uses the left edge of the day as the output, so I don't think it's required. I'd be open to your opinion on where that assumption might normally fail though.

As for the efficiency gains of skipping days that we know we aren't highlighting, is that worth the extra code? That kinda leads in to a bigger discussion about plugins generally that I've been meaning to start. (Eg. are the plugins on the demos "official production ready" or just "here is how to use feature x?" ; will there be official production ready plugins made available via more repos (maybe uplot could follow the pattern that Express and the like have followed?)) I did a < 10s search for a discussion about that kind of thing but couldn't find any in the issues. Maybe it's worth starting up?

It'd be really nice to be able to go npm i @uplot/plugin-x and know that you are getting official plugins but I'd also understand if that's out of the scope of what you are wanting to do with uplot

[leeoniya]

i did a test with random data (below) and it does seem to highlight consistently, so i apologize for my rant. i'll need to look more carefully into how it works. the quasi-code i had in my head seems like it should be simpler and perhaps more generic, but i should try to actually write it before declaring this as fact.

as for whether the demos are meant for general consumption, kind-of. people certainly copy-paste the plugins from the demos and use them as-is, so i would like to keep the plugins generic enough for that purpose. i'm not sure if i'd like to make official/versioned plugins, because i'm kinda worried that the plugins will now have to have official APIs and feature requests for customizing various aspects via options. their current state is nebulous enough where i dont have to commit to supporting any specific API.

let's get #216 polished and landed so i can cut a new release and then i'll circle back to reviewing this and maybe trying my own version, too.

thanks for your help!

    function randInt(min, max) {
        min = Math.ceil(min);
        max = Math.floor(max);
        return Math.floor(Math.random() * (max - min + 1)) + min;
    }

    let dayRef = ["Sunday", "Monday", "Tuesday", "Wednesday", "Thursday", "Friday", "Saturday"];
    //                       S, M, T, W, T, F, S
    let highlightWeekends = [1, 0, 0, 0, 0, 0, 1];
    let highlightThursday = [0, 0, 0, 0, 1, 0, 0];
    let highlightMonThurs = [0, 1, 0, 0, 1, 0, 0];
    let highlightDaysThatEndWithDay = [1, 1, 1, 1, 1, 1, 1];

    let HIGHLIGHTS = highlightWeekends;
    let TIMEZONE = "America/Chicago";
    //let TIMEZONE = "Africa/Juba";

    let base = 1546300800;    // Jan 1 2019
    let max  = 1577836800;    // Jan 1 2020

    // random amount of data
    let numPts = randInt(0, 1000);

    let ts = [];

    while (numPts--)
        ts.push(randInt(base, max));

    ts.sort((a, b) => a - b);

//    ts.unshift(base);
//    ts.push(max);

    let d = ts.map(v => 0);

    function highlightDaysPlugin(days = [1, 0, 0, 0, 0, 0, 1], color = "#bdffdb") {
        let all = days.reduce((acc, cur) => acc + cur, 0) === 7;

        function tstz(ts) {
            // We can"t use the date functions directly, so we have to calculate the offset
            // In production, you should probably use a more robust method.
            let uDate = uPlot.tzDate(new Date(ts * 1e3), TIMEZONE);
            let tsDate = uPlot.tzDate(new Date(ts * 1e3), "UTC");

            let offset = tsDate.valueOf() > uDate.valueOf()
                ? -1 * (24 - uDate.getHours())
                : uDate.getHours();
            offset *= 60 * 60;

            return ts - offset;
        }

        function shouldHighlight(ts) {
            // https://stackoverflow.com/questions/36389130/how-to-calculate-the-day-of-the-week-based-on-unix-time
            let leftTs = Math.floor(ts / 86400);
            return [!!days[(leftTs + 4) % 7], leftTs * 86400];
        }

        function highlightDays(u) {
            let { ctx } = u;
            let { height, top } = u.bbox;
            u.ctx.save();
            u.ctx.fillStyle = color;

            function highlightAt([from, to]) {
                u.ctx.fillRect(from, top, to - from, height);
            }

            if (all) {
                // Shortcut!
                highlightAt([u.bbox.left, u.bbox.left + u.bbox.width]);
                u.ctx.restore();
                return;
            }

            function valToPos(val) {
                return u.valToPos(val, "x", true);
            }

            let [s, end] = [u.data[0][0], u.data[0][u.data[0].length - 1]]
            while (s < end) {
                // left = [shouldHighlight, startOfDay (unix)]
                let left = shouldHighlight(s);

                if (left[0]) {
                    let right = left;
                    while (right[0])
                        right = shouldHighlight(right[1] + 86400);
                    s = right[1];
                    highlightAt([left[1], right[1]].map(tstz).map(valToPos));
                }

                s += 86400;
            }

            u.ctx.restore();
        }

        return {
            hooks: {
                drawClear: highlightDays
            }
        };
    }

    const opts = {
        width: 1920,
        height: 600,
        title: "Highlight Weekends",
        tzDate: (ts) => uPlot.tzDate(new Date(ts * 1e3), TIMEZONE),
        plugins: [
            highlightDaysPlugin(HIGHLIGHTS)
        ],
        series: [
            {
                label: "Day of Week",
                value: (u, v) => dayRef[uPlot.tzDate(new Date(v * 1e3), TIMEZONE).getDay()]
            },
            {
                stroke: "green",
            },
        ],
    };

    let u = new uPlot(opts, [ts, d], document.body);

[EmiPhil]

What probably threw you off is my silly use of:

let [s, end] = [u.data[0][0], u.data[0][u.data[0].length - 1]]

which really should have been

let [s, end] = [u.scales.x.min, u.scales.x.max]

or something to that effect.

I'm happy to help, thank you for creating the lib!

[ryantxu]

This seems similar to the kinda awkward grafana feature:

[leeoniya]

it's only awkward when you're looking at random-walk data :).

real data frequently has clear patterns that align with days of the week, times of the day, solar or tidal cycles, etc. delineating these regions is really helpful in spotting anomalies that fall outside of expected trends, normal maintenance or deployment windows, etc.

[zac-yang]

Is this viewed as the correct / most efficient way to highlight weekends?

Combining this with zoom + pan, I found that this was rendering on the canvas areas outside of the over div. Made some tweaks that addressed this:

  ...
  let ctxLeftBound, ctxRightBound;
  ...
    function highlightAt([from, to]) {
      if (to < ctxLeftBound || from > ctxRightBound) return;
      if (from < ctxLeftBound) from = ctxLeftBound;
      if (to > ctxRightBound) to = ctxRightBound;
      ctx.fillRect(from, top, to - from, height);
    }
  ...
  return {
    hooks: {
      setSize: (u) => {
        const overDiv = u.root.querySelector(".u-over");
        const devicePixelRatio = window.devicePixelRatio;
        ctxLeftBound = parseInt(overDiv.style.left) * devicePixelRatio;
        ctxRightBound =
          ctxLeftBound + parseInt(overDiv.style.width) * devicePixelRatio;
      },
      drawClear: highlightDays,
    },
  };

Before (see right edge):

After:

[lticadler]

Stumbled upon the same problem as @zac-yang :

Combining this with zoom + pan, I found that this was rendering on the canvas areas outside of the over div. Made some tweaks that addressed this:

But as I understood there is no need to add on other setSize hook. The u.bbox has already this information. Therefore I used this little function to check the x values.

const checkXAgainstUBox = (xValue, u) => {
    const { left, /* top, */ width /* , height */ } = u.bbox;
    if (xValue < left)
      return left;
    if (xValue > left + width)
      return left + width;
    return xValue;
  };