Errors in East/north to lat/long convertions

[GilesGibson]

live site.

The datasets that have been converted from east/north seem to have all their locations roughly the same amount of distance out. They all seem to need to move West and a little North. Is this an issue with the conversion script or bad original data?

[GilesGibson]

Now being reported by other users.

[dataunity]

Can you confirm the names of the datasets which are having issues - want to confirm it's not the DU conversion lat/long process (don't think any of the DU datasets are still in use though).

[GilesGibson]

Childrens centres, community gardens, GP practices.... those have been highlighted so far.

[dataunity]

Ok, thanks. To confirm, these aren't ones that have been processed via DU.

By the way, I think you can find out lat/long of place in Google maps in case you wanted to check the lat/longs in the datasets:

https://support.google.com/maps/answer/18539?hl=en

[pmackay]

Its not DU, I did these conversions using a Ruby library. If they're wrong, they're wrong. I guess the Ruby lib is not that great. Obviously need to investigate but it would be useful to know what the goal is. Do we need to find a way to do accurate conversions at this stage? If so, for what purpose?

On Tue, Aug 19, 2014 at 1:12 PM, Data Unity notifications@github.com wrote:

Ok, thanks. To confirm, these aren't ones that have been processed via DU.

By the way, I think you can find out lat/long of place in Google maps in case you wanted to check the lat/longs in the datasets:

https://support.google.com/maps/answer/18539?hl=en

— Reply to this email directly or view it on GitHub https://github.com/folklabs/communitydata/issues/111#issuecomment-52624320 .

[GilesGibson]

yup, thats what one of the users did too, checked it via google maps.

Is suppose credibility is important regarding the accuracy of the pins. If people see things that they are familiar with being shown in the wrong place then they start to question other stuff. I think Apple now realise the need to put things on a map where they actually are....

[pmackay]

Of course, but that is not important if this is promoted as a prototype, right? The quality of the data shouldnt matter so much.

[GilesGibson]

in theory I would agree. However if the original data was good but our processing of it was not as good as users expected then it may cause people to not want to get too involved and reduce any ideas they had of going through the process of creating maps, adding datasets etc. Maybe people think that by now the computer map apps are good enough to have accurate pins - they maybe don't understand the issues of the conversions. It may also make lambeth publish in Lat/Long as well as east/North.

[pmackay]

@GilesGibson The script I used is based on http://www.movable-type.co.uk/scripts/latlong-gridref.html, which mentions a few things:

The National Grid system starts with a top level grid denoted by 2 letter grid squares, e.g the Lambeth data is probably grid TQ. That could correspond to the 5 and 1 digits of each E/N reference, e.g. 529318 175614 for Grafton Square (Dr Ala) 8B Grafton Square Brixton London SW4 0DE It would be good to confirm this.

Also a 6 digit reference (the GP practice data and most others is 6 digit) only has an accuracy of 100m, so that is bound to result in a shift of pins when converted. 8 or 10 digit references would give 10m or 1m accuracy respectively.

Can you get some more information about the original datasets from Tom or someone else, regarding these points? Until we understand more clearly the input data I'm not sure I work out a strategy.

[GilesGibson]

Malcolm, when he saw the raw data of the converted dataset commented that it had not been done to enough decimal points so would be out.

Are we (you) suggesting that the original data only had a few digits so would never be that accurate? I can chase if this is the case. The datasets on the Lambeth site are directly from Tom as an export of his system (GIS software etc). If he is exporting only approximate numbers (few digits after decimal point) then we need to take it back there. I note that the GP surgery data only has 6 digits in the E/N columns, may be our problem.

[pmackay]

@GilesGibson yes it would be good to get clarity from Tom on what accuracy he thinks is possible from the data. How do the E/N figures work in the Lambeth datasets? What does he think they should support?

[GilesGibson]

Feedback from Malcolm is that the numbers are in metres so the error is in the conversion.

[pmackay]

Can he give more detail than that, because they are 6-figure numbers which suggests only an accuracy of 100m.

[GilesGibson]

Have asked for more detail. Maybe the conversion algorithm always places the pin in one corner of the 100m square box rather than in the centre?

[pmackay]

Right but pinning in one corner of 100m square still cant provide accuracy of smaller than 100m.

[GilesGibson]

Malcolm and Tom are adamant that six figures gives an accuracy to 1m.

[pmackay]

@GilesGibson would be great if you could review http://5.101.100.119/dataset/lambeth-gp-practices/resource/7bdb1221-4b1b-4724-a3a5-e222eff80c2a when you're back.

[GilesGibson]

I cannot add this to COD. No idea why, just the error 500 message.

[pmackay]

@GilesGibson are the coordinates ok though, even on the CKAN site? That's what this issue is about.

[GilesGibson]

How would I review the coordinates without seeing them on a map? Apologies if being stupid.

[pmackay]

@GilesGibson the link is above to the dataset on CKAN.

[GilesGibson]

I can only tell on a map, not a csv file. has anything changed on the geomapping feature so that if I created a new vis the locations would be accurate? happy to continue testing if they have.

[pmackay]

Yes but CKAN has the ability to view the data as a map, so can you try that and test them?

[GilesGibson]

OK, I checked the gp practices lat/long fix csv file. Looks much better. What was changed?

[pmackay]

Closing as done.