Support moving-point CSV files

[rsignell-usgs]

What would be the best way to get some bird migration data into TerriaJS?

Basically, this is basically a bunch of CSV files (one for each bird type), and each CSV file looks like this:

Flycatcher.csv:
"yearday","lon","lat"
1,-77.1685735208084,5.36328777248622
2,-77.1511228203405,5.38111171240858
3,-77.1315657168625,5.39856209286252

I looked at the CSV specification here https://github.com/NICTA/nationalmap/wiki/csv-geo-au but it seemed not designed for this type of data (even if I converted yearday to dates for a specific year).

Would CZML be the best bet?

[kring]

@rsignell-usgs I guess it depends on what sort of visualization you're trying to create. I changed the CSV file above to:

date,lon,lat
2016-01-01,-77.1685735208084,5.36328777248622
2016-01-02,-77.1511228203405,5.38111171240858
2016-01-03,-77.1315657168625,5.39856209286252

And dropped it onto nationalmap.gov.au. It shows each point in the list until the time slider gets to the next time, at which time the point is shown at that location.

If you'd instead like to draw a line representing the path, and then smoothly animate a point along the line over time, CZML is a good way to do that. Here's what that would look like:

[{
    "id": "document",
    "name": "flycatcher",
    "version": "1.0",
    "clock": {
        "interval": "2016-01-01/2016-01-03",
        "currentTime": "2016-01-01",
        "multiplier": 21600.0,
        "range": "LOOP_STOP",
        "step": "SYSTEM_CLOCK_MULTIPLIER"
    }
}, {
    "id": "flycatcher",
    "availability":"2016-01-01/2016-01-03",
    "position": {
        "cartographicDegrees": [
            "2016-01-01", -77.1685735208084, 5.36328777248622, 0.0,
            "2016-01-02", -77.1511228203405, 5.38111171240858, 0.0,
            "2016-01-03", -77.1315657168625, 5.39856209286252, 0.0
        ]
    },
    "point": {
        "color": {
            "rgba": [
                0, 0, 255, 255
            ]
        },
        "outlineWidth": 0,
        "pixelSize": 10,
        "show": true
    },
    "path": {
        "show": true,
        "width": 1,
        "material": {
            "solidColor": {
                "color": {
                    "rgba": [
                        0, 0, 255, 255
                    ]
                }
            }
        }
    }
}]

[rsignell-usgs]

@kring , awesome! Thanks! I'll see if I can report back with a non-toy extension of what you've given me to start with!

[stevage]

It's conceivable that we csv-geo-au should cover this use case. Probably there would need to be some kind of id column, so that:

id,date,lat,lon
1,2015-03-01,...
1,2015-03-02,...
2,2015-03-01,...
2,2015-03-02,...

...is interpreted as two things from point A to point B over two days, as opposed to four things, each of which exists for one day.

[rsignell-usgs]

That would be fantastic, as there are actually 118 different bird types in this use case!

Basically I was thinking how cool it would be if we could recreate something like this animation https://www.allaboutbirds.org/mesmerizing-migration-watch-118-bird-species-migrate-across-a-map-of-the-western-hemisphere/ in TerriaJS.

Instead of just watching a cool animation, users could then find out which species are which and mash up the bird data with other geospatial content!

[stevage]

Yeah, pretty cool use case. Not exactly trivial to implement though!

[kring]

It's pretty easy with CZML, right? You can even do that trail effect easily with the path element (though it would be more like a line trailing behind rather than a shadow of the previous dots).

[rsignell-usgs]

@kring , I just modified your CZML example (https://github.com/rsignell-usgs/ehzrap/blob/master/testing/bird_data.czml) to add the rest of the data for a year (365 days instead of 3) but when I drag-and-drop https://github.com/rsignell-usgs/ehzrap/blob/master/testing/bird_data_1year.czml onto the TerriaJS instance at nationalmap.gov.au, , the fan on my laptop comes on, I see memory growing to 1GB and TerriaJS becomes sluggish and unresponsive.

This is just a tiny amount of data (365 geospatial locations with time stamps), so I must be doing something very wrong in the CZML, right?

[kring]

I finally looked into this, and the reason it's so slow is because Cesium, by default, subsamples positions so that successive samples are no more than 60 seconds apart. This makes the polyline have 525600 vertices instead of 365. This Cesium behavior seems a bit... wrong... but at least it's easy to override. Change the path part of the CZML to look like this:

        "path": {
            "width": 1,
            "material": {"solidColor": {"color": {"rgba": [
                0,
                0,
                255,
                255
            ]}}},
            "show": true,
            "resolution": 86400.0
        },

Note the added resolution property at the end. After I did that the performance was pretty good. Another approach may still be necessary when you bring in the full dataset, though.

[rsignell-usgs]

@kring, awesome! With resolution set to one day, I tried all the birds, and it performs great! The CZML file is here: https://drive.google.com/open?id=0BzAHlPEEP_ujRVhCNFMyZE5Ga3M and here is a snapshot:

The next thing I'd like to do is have users be able to click on a dot and get the time, location and bird type. That must be possible with CZML, right? I tried to figure this out from the CZML docs, but failed. Would this be a billboard?

[kring]

Yes, you just need to add a description property to each entity. That property can be HTML or (in TerriaJS at least) Markdown, and it will be displayed in the feature info popup when the entity is clicked. For example:

[{
    "id": "document",
   ...
}, {
    "id": "flycatcher",
    "availability":"2016-01-01/2016-01-03",
    "description": "This HTML will be shown in the <strong>feature info</strong> panel.",
   ...
}]

[kring]

Actually, come to think of it, that doesn't really help you with the time and location. We can probably easily extend TerriaJS to allow the feature info to show the current time and the current position of the point. If you showed the paths of the birds, and wanted to allow the user to click anywhere on the path and show the time and position of the clicked point, that would take a bit more effort.

[rsignell-usgs]

Just to report back that eventually I did figure out how to use the python czml package to write the description tag, with a bit of help from @ocefpaf. The notebook is here, in case folks are interested: https://github.com/rsignell-usgs/CZML/blob/master/birds/bird_csv_to_czml.ipynb

And if you want to see the bird migration (with monthly air-temperature maps underneath), do this:

1. Go to: http://comt.sura.org/proxy_3001/#https://raw.githubusercontent.com/rsignell-usgs/ehzrap/master/testing/bird_migration.json
2. Click on the time-slider to start animating
3. Click on a dot and get the bird type

[stevage]

I think it's ok to leave this as open issue to support moving-point CSV files at some point in the future.

[RacingTadpole]

@rsignell-usgs Would you be willing to share your csv files so I try it in the latest terriajs? If you have one csv file in the format

id,date,lat,lon

it should work now, though we'll need to specify the id column via json. Maybe I should have made a column named id work by default...

[rsignell-usgs]

@RacingTadpole, here is a CSV file for a single bird species: https://github.com/rsignell-usgs/ehzrap/blob/master/testing/flycatcher.csv

I could also make a single CSV for all bird species, identifying them via a new column of text, if that would work, like:

id, date, lon, lat
"flycatcher", 2015-01-01, -71.5, 43.12
"oriole", 2015-01-01, -72.3, 41.5
...

Is this what you would rather have?

[RacingTadpole]

thanks - yes, that would be ideal, but if it's too much work, don't worry. It would just be for testing/showing off. :)

[RacingTadpole]

It would be pretty cool to show off though. :-) Especially if you there are any other numeric columns relevant to the birds, eg. speed or height... as the feature info panel now shows a plot of any numeric values over time as well. Eg.

[RacingTadpole]

FYI, I've written some python code to extract the csv I need from the czml, with a (quickly constructed, hopefully roughly correct) speed column too:

import json, math
with open('bird_migration.czml', 'r') as fp:
    birds = json.load(fp)
def speed(pos1, pos2):
    # Assume all date differences equal
    dlon = pos2[1] - pos1[1]
    dlat = pos2[2] - pos1[2]
    dx = 10000./90 * dlon * math.cos(pos1[2])
    dy = 10000./90 * dlat
    return math.sqrt(dx * dx + dy * dy)

def is_key_date(date_string):
    # just to reduce the file size a bit
    return (date_string.split('T')[0].split('-')[-1] in ['01', '08', '15', '21'])

out = []
for bird in birds[1:]:
    position_flatlist = bird['position']['cartographicDegrees']
    positions = list(zip(*(iter(position_flatlist),) * 4))
    out += [list(position[0:3]) + [speed(positions[i-1], position), bird['id']]
            for i,position in enumerate(positions) if is_key_date(position[0])]

with open('bird_migration.csv', 'w') as fp:
    fp.write('date,lon,lat,speed,id\r')
    for row in out:
        fp.write(','.join(str(x) for x in row) + '\r')

And a sample plot of bird speeds from NationalMap:

[rsignell-usgs]

@RacingTadpole , looks correct to me! Very cool!

[RacingTadpole]

@rsignell-usgs - would you mind if I put the link to the animated bird migration csv file (with the extra speed column) I made here? It makes quite a nice demo!

[rsignell-usgs]

@RacingTadpole , that would be great!

[RacingTadpole]

Cool - here it is: https://gist.githubusercontent.com/RacingTadpole/f308c8d01630b3d0b15d946e259b0cff/raw/266b62cab3716d76fe4c24f9788489ebfd8c7efa/bird_migration.csv

For anyone arriving at this issue - you can see the bird migrations animated on the map without downloading any data, by going to http://nationalmap.gov.au/, clicking “Add data” -> “My Data” -> “Add web data” and pasting the above URL into the text box, then clicking “Add”. Hit the escape key or click “Done” to remove the popup window, and you'll see them in action :-)