Let's talk maps

[shermp]

As you are looking into implementing maps, I thought it would be worth opening an issue on it here for us to discuss ideas, problems and solutions.

For example, tool tips. Do you want tool tips that display when the mouse hovers over a point? I notice the pop-up works fine already when clicking on a point.

When I had a quick look at it earlier today, it seems the starting zoom scale for the map might be too close for OpenStreetMaps. I had to zoom out for the map to show, and leaflet wouldn't let me zoom back in.

[SteveCossy]

That's where the Javascript comes in! The code for tool tips is very similar to the popups, but I don't know how to implement it. :-( Come to think of it - perhaps the answer is to search for both functions, and find where they are implemented together! Another thing I want to do is create a switch on the Cayenne dashboard that can be used to trigger code that will archive the current results csv file, and start a new one. (Starting a new one is already happening - the current code checks before writing each line, and creates a new file if necessary.)

[SteveCossy]

The answer is probably in this page somewhere (I really like the format): https://codepen.io/mmsmdali/pen/KqjNPN

[shermp]

I don't know how you generated your RSSILatLong.geojson, but I highly recommend using RFC 3339 style date/timestamp. Most software can decode this format into a date/time object, and it is still human readable.

For example, the following

"Time": "26/11/19 13:50:31"
"Time_Stamp": "20191126135031"

Could be consolidated to

2019-11-26T13:50:31Z

(For UTC)

Or

2019-11-26T13:50:31+13:00

(For NZDT)

[shermp]

Ah, I think I see the reason why you were having issues with tooltips. You were using a very old version of leaflet (0.6.4), the current version is 1.6.0. In fact, they consider 0.7.7 legacy, and that was released in 2013!

I shall update to the latest version, and see what that changes.

EDIT: Yep, that did the trick. Also updated jquery while I was at it.

[SteveCossy]

Cool thanks - will look at that in a while. Right now am writing some documentation in a Project Wiki. That will help with clarifying the question about date formats. I definitely agree with your summary.

[SteveCossy]

I have created a very rough Wiki page that details how the geoJson is created. Plan is to change the timestamp in the csv file, so there is a Time_Stamp field in the geoJson that will be RFC 3339 compliant. How easy would it be to reformat this to a simpler format for use on the map? Then I could remove the 'Time' field from the geoJson. Less it better! Any other suggestions? I'll be getting on to this about 2:30ish. PS - have you used the GitHub Wiki. Not sure I'm keen. Keeps stuff together, but not easy to use.

[shermp]

I've never used Github Wiki feature before, so can't offer much insight on that at the moment.

Yeah, there are libraries available to parse the RFC3339 format into a date object, which can then be used to perform calculations, manipulations and output formatting. Apparently the native date string parser in JavaScript can be a bit iffy, especially on older browsers, but a library like MomentJS could do the trick.

[SteveCossy]

I wonder what the overhead is for having the friendly date in the geoJson as opposed to us js to generate it for each point. I'll challenge my fear of js, and modify the html! Will be home in 30 minutes or so. 😊

On Wed, 27 Nov 2019, 1:18 PM Sherman Perry, notifications@github.com wrote:

I've never used Github Wiki feature before, so can't offer much insight on that at the moment.

Yeah, there are libraries available to parse the RFC3339 format into a date object, which can then be used to perform calculations, manipulations and output formatting. Apparently the native date string parser in JavaScript can be a bit iffy, especially on older browsers, but a library like MomentJS https://momentjs.com/ could do the trick.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/SteveCossy/IOT/issues/2?email_source=notifications&email_token=AENGJJFC32CK6O2W47HQ2I3QVW4G5A5CNFSM4JRWPEB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFH3P3Y#issuecomment-558872559, or unsubscribe https://github.com/notifications/unsubscribe-auth/AENGJJCKVJOAVXOQYCGZGMLQVW4G5ANCNFSM4JRWPEBQ .

[shermp]

I'd advise having a look at #4 and maybe merging it before you start working further on it. It will make everyone's life easier because it avoids merge conflicts.

The overheads of converting the dates to a friendly format should be fairly minimal. Should only do it once per feature on initial load. The upside is it provides future flexibility if you decide you want to format it differently, or do other stuff with the timestamp later on.

[SteveCossy]

Oops! Shouldn't have closed the issue - I am still working on it. Did a test - the updated file doesn't work on IE (our main student management app at work requires it, because it uses old, dangerous, Flash :-( ). Have saved a copy of the old index.html. Will call it IEindex.html, and work out how to call it if one is using IE. I wonder if Andrew's Firefox will read the new one? I don't have an old Firefox to test with.

[shermp]

Yeah, IE won't work (I mentioned that in the PR). If IE is a hard requirement, then changes will need to be made to the popup_html variable to avoid using the literal template syntax. If that's the case, I could create a function to generate the table instead.

Dunno about leaflet and IE.

If you have it on your work PC's, Microsoft Edge should work fine.

[shermp]

Ok, leaflet appears to work with IE11, at least at a basic level.

I'll fix it so it works on IE11

[SteveCossy]

Remember - I'm working on the geoJson! It will be coming in with an ref 3339 TimeStamp value soon. (Changed from Time_Stamp.) I was not too worried about IE, but if you want to make it work, that's fine.

[shermp]

Got it working with IE11.

Bonus, if we need to create other tables, there's a function for it :)

Will wait for you to make your geojson changes before commiting.

[shermp]

And the next PR will also improve zoom levels. Can now zoom in way closer, and no more grey tiles. (It upscales the closest available tiles)

[SteveCossy]

Ok - have uploaded a sample of the new geoJson https://github.com/SteveCossy/IOT/blob/master/CayenneMQTT/RSSILatLong.geojson :-)

[shermp]

Just a note, the timestamps there are not valid RFC3339, they are missing the timezone suffix (either Z for UTC or +/-hh:mm for localtime)

And I just remembered that generating proper RFC3339 timestamps in Python can be a bit of a pain :(

[SteveCossy]

Opps! I was close! Yeah - I read that getting the timezone could be a pain. All my machines are set to local time, so I ignored it. Happy to add it though.

[SteveCossy]

Oh! My kingdom for a tzinfo subclass! Definitely not simple!

[shermp]

You're running python 2.7 I take it?

You might need to use the pytz library.

Python 3.3+ includes a timezone class in the standard libs

[shermp]

A note on timezones:

My motto is do everything in UTC where possible, and only display local time to the user. That saves a heap of grief in the long run, especially if you are saving data.

[SteveCossy]

For the first 20 years of teaching, my motto was to leave development to developers, then I discovered Software Defined Networks (SDN) and decided that I was going to have to learn some development skills. I've been catching up ever since. SDN and IoT both use lots of Python. I've steadfastly avoided Javascript (other than Google Sheets - another story). How is the new html file coming along?

[shermp]

New file is mostly there.

Instead of getting the JS libraries via CDN, I'm thinking of downloading them and adding them to a scripts subdirectory. Mainly to assist in development/debugging, and I personally don't like the idea of relying on third party services to host essential libraries.

Are you OK with this?

[SteveCossy]

No worries. How does all that work? Javascript is client-side, so the client downloads libraries from our server? On a similar note, one of the refs I found suggested that OSM would rather we used a different source, rather than downloading the tiles from them. It was a old page and no one else has mentioned this. I will study your code with great interest! Some other cosmetic thoughts I had:

• only have the top two lines on the tooltip. I thought perhaps set up two variables TimeRSSI_html and LatLong_html. Put them both into bindPopup and just TimeRSSI in to bindTooltip
• make the pins a little smaller
• something else I've forgotten ...

Longer-term I wondered about writing some Python to put all my backup geoJson files into a folder on the server, then write some js to list the names of the backup files. When someone clicks on one, they get a map showing the points that are contained in the selected file. Sort of like starting to build my own little content management system! (At that point it starts to sound a little scary.)

[shermp]

No worries. How does all that work? Javascript is client-side, so the client downloads libraries from our server?

Yeah, basically that.

On a similar note, one of the refs I found suggested that OSM would rather we used a different source, rather than downloading the tiles from them. It was a old page and no one else has mentioned this.

The official policy on that can be found here. TLDR: it depends. If our usage is low volume enough, then it should be alright. At some point it might be worth investigating our own tile server. Some options are switch2osm or OpenMapTiles. Certainly a lot more involved than pointing leaflet to the OSM tile server!

only have the top two lines on the tooltip. I thought perhaps set up two variables TimeRSSI_html and LatLong_html. Put them both into bindPopup and just TimeRSSI in to bindTooltip make the pins a little smaller

Good ideas.

Longer-term I wondered about writing some Python to put all my backup geoJson files into a folder on the server, then write some js to list the names of the backup files. When someone clicks on one, they get a map showing the points that are contained in the selected file. Sort of like starting to build my own little content management system! (At that point it starts to sound a little scary.)

At that point you may as well investigate storing the geographic data in a GIS database like PostgreSQL with PostGIS extension. Gives you all the sorting, filtering and integrity that a database provides. Heck, PostGIS can also output GeoJSON from a query if that's what you want ST_AsGeoJSON

[SteveCossy]

As it happens, I have very little interest in GIS data. Andrew wanted to be able to remember where he left his rubbish bin (long story), and I wanted to learn more about processing MQTT data. Probably the biggest single step in this project was when you told me to use a different ClientID for logging in. That was why I was not able to read from Cayenne in previous efforts. Short term goal is to get the interest of some kids at the Tahi Rua Tora Tech! final next week. I want to research how much changing weather affects transmission of 433 mHz data. (Answer is expected to be 'not much', but I want something more definite.)

[shermp]

Fair enough.

I just figured storing it all in a database might be easier than dealing with lots of geojson files, and would allow for further analysis if desired. (Distance of sensor to receiver? LOS? Add weather conditions at time of sensor reading?).

[SteveCossy]

True, that does have merit. I started a student project at work with the intension of getting to that point, but it was going too far from my networking roots, and I couldn't get any Application lectures to take it up. In theory we can interigate data on Cayenne's cloud, but I am not sure about that as a long-term solution.

On Wed, 27 Nov 2019, 10:01 PM Sherman Perry, notifications@github.com wrote:

Fair enough.

I just figured storing it all in a database might be easier than dealing with lots of geojson files, and would allow for further analysis if desired. (Distance of sensor to receiver? LOS? Add weather conditions at time of sensor reading?).

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHub https://github.com/SteveCossy/IOT/issues/2?email_source=notifications&email_token=AENGJJGSGCDYXHLZ2LJCRWDQVYZNHA5CNFSM4JRWPEB2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEFIZHHA#issuecomment-558994332, or unsubscribe https://github.com/notifications/unsubscribe-auth/AENGJJDYI7BGAFMPNKTZKT3QVYZNHANCNFSM4JRWPEBQ .

[shermp]

There are other DB options than a full database server as well. Something like SpatiaLite is an option. It's basically a SQLite database with GIS specific extensions.

Such an option gives one the power of running SQL queries on the data, with the convenience of a single portable file.

You can also directly connect to these database options from a Python script as well.

[shermp]

If you want to stick with the idea of allowing the user to select from multiple geojson files, here's an idea that might work.

Create an "index" or "toc" json file that includes something like the following:

{
    "currentGeo": {
        "title": "Current GeoJSON",
        "path": "geojson/current.geojson"
    },
    "archiveGeo": [
        {
            "title": "Prevous GeoJSON 1",
            "path": "geojson/date-prev1.geojson"
        },
        {
            "title": "Prevous GeoJSON 2",
            "path": "geojson/date-prev2.geojson"
        }
    ]  
}

(Note, the above is just an example)

index.html could then load this json first, render the "currentGeo", and display a list of the "archiveGeo" entries.

Your python script could create and maintain such a file.

[SteveCossy]

Been making quite a few maps over the past six weeks or so. I guess it is finished as far as it needs to be! http://students.pcsupport.ac.nz/OSM/ http://students.pcsupport.ac.nz/OSM/form.py/