Wrong UTC / Local time interpretation

[iliatimofeev]

{

  "layer": [
    { 
      "mark": {"type": "point", "color": "#3366cc"},
      "encoding": {
        "x": {"type": "ordinal", "field": "date", "timeUnit": "date"},
        "y": {"type": "ordinal", "field": "date", "timeUnit": "hours"}
      }
    },
    {
      "mark": {"type": "point", "color": "#dc3912", "shape": "square"},
      "encoding": {
        "x": {"type": "ordinal", "field": "date", "timeUnit": "date"},
        "y": {"type": "ordinal", "field": "date", "timeUnit": "utchours"}
      }
    }
  ],
  "data": {
    "values": [
      {"date": "2018-02-01"},
      {"date": "2018-02-02 00:00"},
      {"date": "2018-02-03 00:00 UTC"},
      {"date": "2018-02-04 00:00Z"}
    ]
  },
  "$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json"
}

I have UTC+3 so expected:

as

{

  "layer": [
    { 
      "mark": {"type": "point", "color": "#3366cc"},
      "encoding": {
        "x": {"type": "ordinal", "field": "date", "timeUnit": "date"},
        "y": {"type": "ordinal", "field": "local"}
      }
    },
    {
      "mark": {"type": "point", "color": "#dc3912", "shape": "square"},
      "encoding": {
        "x": {"type": "ordinal", "field": "date", "timeUnit": "date"},
        "y": {"type": "ordinal", "field": "UTC"}
      }
    }
  ],
  "data": {
    "values": [
      {"date": "2018-02-01" ,"local":0,"UTC":21},
      {"date": "2018-02-02 00:00","local":0,"UTC":21},
      {"date": "2018-02-03 00:00 UTC", "local":3,"UTC":0},
      {"date": "2018-02-04 00:00Z","local":3,"UTC":0}
    ]
  },
  "$schema": "https://vega.github.io/schema/vega-lite/v2.6.0.json"
}

[domoritz]

If you're using UTC, you should use the UTC time units: https://vega.github.io/vega-lite/docs/timeunit.html#utc

[iliatimofeev]

Well I read it twice, I can't find any mention why I can't show UTC or local depending on what I put in data. only warning about double conversion on scale but I don't use it.

Do not use UTC time unit and the UTC scale type at the same time since that will cause Vega-Lite to shift the output time twice.

The only issue I can see that "2018-02-01" is interpreted as UTC, but i believe that it should be local and if it is ISO than it has to be local if it is not ISO it has to be local accoding to documentation

[domoritz]

Except that JavaScript interprets strings in a particular way: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse

Thanks for the example. We will look at these and see what we can do from the Vega-Lite and Vega side.

[ijlyttle]

UPDATE - never mind - by adding a scale to the tooltip, it works OK

The parser was doing the right thing, parsing to UTC - I needed to tell the tooltip to display in UTC.

Sorry for the interruption.

Bad way, in action: https://bl.ocks.org/ijlyttle/84ce12ce2c1093070f645a2bae3b8fa0

Better way, in action: https://bl.ocks.org/ijlyttle/bccff7673c98e61b887684dbe597cbd3

[jakevdp]

@iliatimofeev

Copying a comment from slack to get it here. I also initially expected the same as you, but it turns out the current behavior (shown in the left-most plot) is consistent with the documented behavior of javascript date parsing (see https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date/parse)

The date time string may be in a simplified ISO 8601 format. For example, "2011-10-10" (just date) or "2011-10-10T14:48:00" (date and time) can be passed and parsed. Where the string is ISO 8601 date only, the UTC time zone is used to interpret arguments. If the string is date and time in ISO 8601 format, it will be treated as local.

When you use an ISO fragment, it it treated as UTC, while a full ISO string is treated as local time.

So the question I suppose is whether this is a big enough problem to tackle adding a custom date parser in Vega. Whether or not that happens, I'm planning to make Altair serialize all dates in full ISO so that they will be treated as local time.

[iliatimofeev]

vega

{
  "$schema": "https://vega.github.io/schema/vega/v3.json",
  "autosize": "pad",
  "padding": 5,
  "data": [
    {
      "name": "table",
      "values": [
        {"date": "2018-02-01", "local": 0, "UTC": 21},
        {"date": "2018-02-02 00:00", "local": 0, "UTC": 21},
        {"date": "2018-02-03 00:00 UTC", "local": 3, "UTC": 0},
        {"date": "2018-02-04 00:00Z", "local": 3, "UTC": 0}
      ],
      "transform": [
        {"type": "formula", "expr": "toDate(datum['date'])", "as": "timestamp"},
      {"type": "formula", "expr": "hours(datum['timestamp'])", "as": "hrs"},
      {"type": "formula", "expr": "utchours(datum['timestamp'])", "as": "utchrs"},
      {"type": "formula", "expr": "date(datum['timestamp'])", "as": "day"}
      ]
    }
  ],

  "marks": [
    {
      "name": "layer_0_marks",
      "type": "symbol",
      "style": ["point"],
      "from": {"data": "table"},
      "encode": {
        "update": {
          "opacity": {"value": 0.7},
          "fill": {"value": "transparent"},
          "stroke": {"value": "#3366cc"},
          "x": {"scale": "x", "field": "day"},
          "y": {"scale": "y", "field": "hrs"}
        }
      }
    },
    {
      "name": "layer_1_marks",
      "type": "symbol",
      "style": ["point"],
      "from": {"data": "table"},
      "encode": {
        "update": {
          "opacity": {"value": 0.7},
          "shape": {"value": "square"},
          "fill": {"value": "transparent"},
          "stroke": {"value": "#dc3912"},
          "x": {"scale": "x", "field": "day"},
          "y": {"scale": "y", "field": "utchrs"}
        }
      }
    }
  ],
  "scales": [
    {
      "name": "x",
      "type": "point",
      "domain": {
        "fields": [
          {"data": "table", "field": "day"}
        ],
        "sort": true
      },
      "range": {"step": 22},
      "padding": 0.5
    },
    {
      "name": "y",
      "type": "point",
      "domain": {
        "fields": [
          {"data": "table", "field": "hrs"},
          {"data": "table", "field": "utchrs"}
        ],
        "sort": true
      },
      "range": {"step": 22},
      "padding": 0.5
    }
  ],
  "axes": [
    {
      "scale": "x",
      "orient": "top",
      "grid": false,
      "title": "date",
      "labelOverlap": true,
      "encode": {
        "labels": {
          "update": {
            "text": {"signal": "format(datum.value, ',')"},
            "angle": {"value": 270},
            "align": {"value": "left"},
            "baseline": {"value": "middle"}
          }
        }
      },
      "zindex": 1
    },
    {
      "scale": "y",
      "orient": "left",
      "grid": false,
      "title": "local, UTC",
      "labelOverlap": true,
      "zindex": 1
    }
  ]
}

[jakevdp]

@ijlyttle

My issue is with how the tooltip is not parsing to UTC as I would expect

The issue is that "2018-01-01" is parsed as UTC, and then adjusted to local time before being displayed. If you want it to be parsed as local time, then you can use "2018-01-01T00:00:00".

[ijlyttle]

@jakevdp In this case, I want to keep everything as UTC so that the rendering is insensitive to the browser's time-zone. I found a fix, the tooltip encoding can accept a UTC scale.

[jakevdp]

My preferred solution for Altair is to try to keep everything local, because by default vega/vega-lite renders in local time. I'd rather have the default just work than require users to manually flag all outputs as UTC every time they use temporal data.

[iliatimofeev]

In fact it is the same issue https://github.com/vega/vega-lite/issues/4044 I just found it, sorry for doubling issue.

[domoritz]

Also https://github.com/vega/vega-lite/pull/4094

[kanitw]

Since it's the same as https://github.com/vega/vega-lite/issues/4044 and https://github.com/vega/vega-lite/issues/4044 is a wontfix, I'll close this too.

Please feel free to reopen if you have a suggestion for better solution.