Allow data providers to specify timestamps for coordinates

[sgillies]

See #76 for a discussion of drift and WGS 84.

This issue is not about representing trajectories of features or paths in time. It's about letting providers specify that coordinates in a collection were measured at a particular time so that future users can account for WGS 84 revisions and drift between, say, Ordnance Survey National Grid and WGS.

I suggest that we consider one single timestamp for a collection and not mix features measured at different times into one collection.

As a part of this work we can also specify interpretation of measurement time when no timestamp is provided, using filesystem or transport protocol timestamps.

[dret]

are you suggesting we explicitly exclude per-coordinate timestamps? i am asking because the lack of timestamp support was one of the motivations of the https://github.com/dret/GeoJSON-X work, where the goal is to allow coordinates (not collections) to have timestamps associated with them. GeoJSON-X is generally about a more well-defined extension model, but timestamps are certainly one of the main motivating examples.

[breynolds]

We need both timestamps at the collection level and timestamps (and other things) at the coordinate level. The timestamp at the collection level indicates a snapshot of data. That is great for describing a static polygon area or a static path from one place to another. This would be very useful for describing the polygon of a state's boundaries or describing the coordinates for a path of directions that a driver will follow. The timestamp at the coordinate level indicates that the data is geotagged data. It indicates dynamic data over time as something traversed a path. Useful applications of this would be analyzing traffic or gas usage from data about a car that drove somewhere, or measuring heart rate of a runner or cadence of a cyclist. The GeoJson standard seems to support static data, but would be infinitely more useful if it could be used with any other arbitrary sensor data. At our company we will use GeoJson describing static data so that we can use it with geospatial searches in Mongo, but we're also going to have to build redundant data using something like GeoJson-x just to house all of our other data. It would be VERY useful for us to have support for dynamic geo data built into GeoJson so that it can be worked into other tools. Taking a look at the GeoJson-x site.... it looks pretty good. We're go ahead and start using it for data storage and build tools around it.

[dret]

On 2015-09-05 20:38, breynolds wrote:

We need both timestamps at the collection level and timestamps (and other things) at the coordinate level. The timestamp at the collection level indicates a snapshot of data. That is great for describing a static polygon area or a static path from one place to another. This would be very useful for describing the polygon of a state's boundaries or describing the coordinates for a path of directions that a driver will follow. The timestamp at the coordinate level indicates that the data is geotagged data. It indicates dynamic data over time as something traversed a path. Useful applications of this would be analyzing traffic or gas usage from data about a car that drove somewhere, or measuring heart rate of a runner or cadence of a cyclist.

that's exactly one of the examples given for GeoJSON-X https://github.com/dret/GeoJSON-X, but please keep in mind that according to the current GeoJSON spec, that example is not valid GeoJSON because the timestamp is not a number.

The GeoJson standard seems to support static data, but would be infinitely more useful if it could be used with any other arbitrary sensor data. At our company we will use GeoJson describing static data so that we can use it with geospa!tial searches in Mongo, but we're also going to have to build redundant data using something like GeoJson-x just to house all of our other data.

to better separate concerns, here's my current view of the world:

• issue 1 is whether the RFC might improve GeoJSON's extension model. both the general model https://github.com/geojson/draft-geojson/issues/59, and the coordinate part of it https://github.com/geojson/draft-geojson/issues/57, could be improved. to me, the current model seems a bit random, with coordinate extensions only being possible when elevation data is present, and extension values being restricted to numbers.
• issue 2 is whether there might be some discovery/metadata model for extensions, and https://github.com/geojson/draft-geojson/issues/56 addresses this issue (though only for coordinates). this is where https://github.com/dret/GeoJSON-X tries to enter the picture. but keep in mind that it would also be possible to keep this entirely out of GeoJSON, and issue something like GeoJSON-X as a separate, layered spec, that would use GeoJSON's extension model to make extensions more self-describing. https://github.com/dret/GeoJSON-X/issues/1 is what i just created to reflect this question for GeoJSON-X.

It would be VERY useful for us to have support for dynamic geo data built into GeoJson so that it can be worked into other tools. Taking a look at the GeoJson-x site.... it looks pretty good. We're go ahead and start using it for data storage and build tools around it.

please provide feedback and please also be aware that the current GeoJSON spec does not allow non-number values. so if you do add values that are not numbers, you are not in line with the current GeoJSON spec.

[breynolds]

I realize that the GeoJSON spec does not support non-numeric data in the coordinates array and has no way to describe the new fields that extend the coordinates. That essentially makes the standard unusable for us for the bulk of our data processing. The standard as it stands is useful for small bits of geo meta data that we could store for use in geo queries. For the stream based processing that we need to do, the geo data needs to be situated close to the other sensor data. We'll go ahead and start using GeoJSON-x because it's much closer to what we need.

Dynamic sensor data is not an edge case; it should be the target case. It reflects the changeover of geo analysis from static data done on stationary computers in the past to the large amounts of sensor, mobile, and IoT data being created. Lat, Long, Alt, and Time are fundamental info but it's the other sensor data that are yielding the new applications.

This is the innovators dilemma. Stay with the assumptions of the past, or realize the direction of the future and jump on board.

Whomever defines the JSON based standard for storing streams of geotagged sensor data will be extremely relevant to modern computing. That more general standard will quickly supersede anything that is based solely on Lat, Long, and Alt alone. To be honest, I'm shocked that there is nothing good already out there. OpenSensorData.net is closer to what we need as a format, but the tools are GPL'ed so they are commercially unusable for us and the privacy intrusion of their server policies is comical.

[dret]

On 2015-09-06 07:43, breynolds wrote:

I realize that the GeoJSON spec does not support non-numeric data in the coordinates array and has no way to describe the new fields that extend the coordinates. That essentially makes the standard unusable for us for the bulk of our data processing. The standard as it stands is useful for small bits of geo meta data that we could store for use in geo queries. For the stream based processing that we need to do, the geo data needs to be situated close to the other sensor data. We'll go ahead and start using GeoJSON-x because it's much closer to what we need.

please stay in touch and give feedback via the GeoJSON-X repo. i want to make sure that whatever it is doing is as much in line with GeoJSON as possible, and also with the ongoing effort to create an GeoJSON RFC.

maybe it's time to write up something for GeoJSON-X so that it is a bit more formal and well-defined. do you think it would be helpful to create an I-D for GeoJSON-X and publish it, so that there's something to point to?

according to https://github.com/dret/GeoJSON-X/issues/1 this might end up getting integrated into GeoJSON, or becoming a separate spec, but just trying to write things down usually helps to move things forward.

Dynamic sensor data is not an edge case; it should be the target case. It reflects the changeover of geo analysis from static data done on stationary computers in the past to the large amounts of sensor, mobile, and IoT data being created. Lat, Long, Alt, and Time are fundamental info but it's the other sensor data that are yielding the new applications.

yes, but one counter-argument is that not all sensor-data is necessarily geotagged, and thus shoving it into a model that's geo-centric in nature is using the wrong foundation.

personally, i agree that it would be great to see something evolving, and that maybe for geo-centric use cases, GeoJSON could be the one.

https://github.com/geojson/draft-geojson/issues/64 is making the point that it also might be interesting to look beyond just JSON and instead explore how JSON text sequences may help for streaming.

Whomever defines the JSON based standard for storing streams of geotagged sensor data will be extremely relevant to modern computing.

that's probably correct, and i hope that GeoJSON at least will improve its extension model, and maybe even start supporting something like GeoJSON's self-description.

[dret]

97 is something i just added because it appeared to me that independent of how/if extensions will be made more flexible and/or self-describing, it would be very useful to be able to signal that some GeoJSON follows certain conventions.

[sgillies]

@breynolds @dret sensor data is super interesting, indeed. In a previous life I was a meteorology student and have a pretty good background in theory and operation of weather stations, balloon sondes, etc. I'm pretty convinced that the GeoJSON format has nothing to offer that kind of data and very concerned about the incidental complexity that will result from adding time series data to GeoJSON geometry coordinates.

At any rate, this thread is way off course now. This issue is not about time series data at all. It's about metadata that will allow future use of WGS84 coordinates collected today.

[dr-shorthair]

FWIW - We prepared an OGC-style spec for serialization of Observations and Measurements using JSON ("OM-JSON" to complement OM-XML). We adopted GeoJSON-style geometry encoding within that. We have attempted to formalize it using JSON Schema, though that is not without its challenges. (52North have done something similar.) It is posted on OGC site, for consideration at an upcoming OGC meeting, though (unfortunately for this group) the OGC portal is member only, but I could make it available to interested parties.

Early feedback has already identified the need for higher dimension data, so we are considering how to hedge the requirement, leveraging GeoJSON geometry where it is enough, but allowing other encodings where necessary.

Perhaps we should compare notes Erik?

[breynolds]

@dr-shorthair I hadn't seen anything about your standard. I'll keep an eye out for more public information. Thanks for sharing that. Coordination with GeoJSON seems sensible.

@dret & @sgillies Non geographic sensor data like what you'd get from a weather station doesn't really fit here. I'd leave that to some other standard (like OM-JSON perhaps). However Geo tagged sensor data could stay in the overall realm of GeoJSON because the two go hand in hand. The tools that work on the geo tagged sensor data would often need to do line/polygon intersection between the sensor stream and some shape. Having their data be compatible with each other would be very helpful.

What you could do is rather than extend the core types, leave them as they are to describe pure geometry. Extend laterally to have other types of data that are location based, but keep the same flavor of JSON so that tools could easily work with both. @dret I took an example you had given and modified it slightly. I added non-numeric data and modified the names of fields. A requirement for a GeoSensorStream would be that it must contain "lat"and "lng", however "alt" would be optional.

{
  "type" : "GeoData",
  "geoData" : 
  [
    { 
      "type" : "GeometryCollection",
      "geometries" : [
        { 
          "type" : "LineString",
          "coordinates" : [
            [
              -122.45671039447188,
              37.786870915442705,
              0.4000000059604645
            ]
          ]
        }
      ]
    },
    {
      "type" : "GeoDataCollection",
      "geodata":[
        { 
          "type" : "GeoSensorStream",
          "fields" : [ "lat", "long", "alt", "time", "atemp", "hr", "cad", "engineStatus" ],
          "datapoints" : [
            [
              -122.45671039447188,
              37.786870915442705,
              0.4000000059604645,
              "2014-11-06T19:16:06.000Z", 
              31.0, 
              99, 
              0,
              "ok"
            ]
          ]
        }
      ]
    }
  ]
}

Perhaps you could work with @dr-shorthair so that the registry for data types and possibly the data structure is compatible with the possibly non-geo sensors in OM-JSON.

I look forward to seeing what you all come up with. I'll stop hijacking this thread and let you get back to the standards creation.

[dret]

On 2015-09-06 14:22, Simon Cox wrote:

Early feedback has already identified the need for higher dimension data, so we are considering how to hedge the requirement, leveraging GeoJSON geometry where it is enough, but allowing other encodings where necessary. Perhaps we should compare notes Erik?

sounds like a good idea, but not here. but please feel free to raise issues for GeoJSON-X, for example. thanks!

[dret]

On 2015-09-06 15:59, breynolds wrote:

What you could do is rather than extend the core types, leave them as they are to describe pure geometry. Extend laterally to have other types of data that are location based, but keep the same flavor of JSON so that tools could easily work with both. @dret https://github.com/dret I took an example you had given and modified it slightly. I added non-numeric data and modified the names of fields. A requirement for a GeoSensorStream would be that it must contain "lat"and "lng", however "alt" would be optional.

from my perspective, that is unnecessarily complicated. GeoJSON already does allow extensions, only that the model could be improved. once it is, we can take it from there and see what's possible and what not.

[dr-shorthair]

Perhaps this conversation would be better assigned to #59 ?

[dret]

@dr-shorthair, it certainly is relates to #59 (and #57 and #56), but with a more specific focus.

[sgillies]

Discussion brought to the list: https://mailarchive.ietf.org/arch/msg/geojson/aY-Y1aC27D-HE-vjJ2COpeD7RNU.

Proposed change to the draft: #121.

[mxfh]

Not sure if this right place to extend on this, but in my opinion GeoJSON should stay largely agnostic about the CRS' used.

Additional position elements MAY be included but MUST follow the three specified above and MAY be ignored by software.

I would even drop the limitation to positional attributes from the third position onward. But limit the data type to numbers (Unlike GeoJSON-X). The handling of those elements might be preferably described in the collections CRS field as a WKT CRS structured string or alternatively as a referencing URI to such a description.

In general WKT CRS seems versatile enough to fit most use cases described above.

Especially temporal CRS/TIMECRS allows for creating custom Compound CRS like:

   COMPOUNDCRS[“GPS position and time”,
     GEODCRS[“WGS 84”,
       DATUM[“World Geodetic System 1984”,
         ELLIPSOID[“WGS 84”,6378137,298.257223563]],
       CS[ellipsoidal,2],
         AXIS["(lon)",east,ORDER[1]],
         AXIS["(lat)",north,ORDER[2]],
         ANGLEUNIT[“degree”,0.0174532925199433]],
     TIMECRS[“GPS Time”,
       TIMEDATUM[“Time origin”,TIMEORIGIN[1980-01-01]],
       CS[temporal,1],
         AXIS["time (T)",future],
         TIMEUNIT[“day”,86400]]]

The Compound CRS specification even allows for multiple temporal CRS as described in GeoAPI reference

More than one Temporal CRS may be added if these axes represent different time quantities. For example, the oil industry sometimes uses "4D seismic", by which is meant seismic data with the vertical axis expressed in milliseconds (signal travel time). A second time axis indicates how it changes with time (years), e.g. as a reservoir is gradually exhausted of its recoverable oil or gas).

For sensor data there is Parametric CRS:

     PARAMETRICCRS[“WMO standard atmosphere layer 0”,
               PDATUM["Mean Sea Level“,ANCHOR[”1013.25 hPa at 15°C"]],
               CS[parametric,1],
               AXIS["pressure (hPa)“,up],PARAMETRICUNIT[”HectoPascal",100.0]]

So point data arrays like [lon, lat, ts, hPa] seem perfectly reasonable.

As it looks like, there are no JavaScript WKT-CRS-parsers yet; the closest thing is ESRI's ogc-crs-wkt-parser implemented in C++. Yet for me, the approach of keeping CRS issues out of the GeoJSON scope and let the implementations handle the CRS on their own seems more versatile and keeps the GeoJSON spec short and precise.

[sgillies]

From the list: we don't have any consensus on recommending a timestamp.