a plan for rnaturalearth

[andysouth]

@karthik @hadley @hrbrmstr

I have put a brief plan for rnaturalearth into the readme here : https://github.com/AndySouth/rnaturalearth/blob/master/README.md

Comments/input welcome both in the plan and implementation.

Thanks.

[hadley]

I like it! Comments:

• I'd recommend picking function names to match USAboundaries, e.g. world_countries(), world_states() (would be nice to have a better word for admin level 1, 2 etc)
• I think it might be better not to have any joining functions in this package - it would be better focus as much as possible on the data. Joins are complicated!
• I'm not sure I'd worry about picking compression levels - I think bzip2 should be fine 90% of the time (and if it's not, you can just manually override)
• For missing countries etc, would it be better to file issues with naturalearth and have them fix it upstream?

[andysouth]

Thanks, that's useful.

Re joins, it is because users have difficulty with them that I thought they should be in there ;-) Happy to start without. Do you have a vision for how users will join their data to the maps ?

[hadley]

Joins are hard because I think there are multiple ways to do them - via sp, base, dplyr, data.table, ...

[hadley]

Is there anything I can do to help kick this into motion? I'd really like to be able to cite it in the book, which I need to have done in 3 weeks (rnaturalearth doesn't need to be on CRAN, but it at least needs to be heading in that direction). I think it's reasonable to just focus on the data (rather than the tools) for the first version.

[andysouth]

What you've just done may be enough to kick me into motion! I'm busy with other stuff, but the chance to get it cited is a big incentive. Getting selected maps in is an easy first step. Tools to allow users to import other maps from natural earth may be less straightforward e.g. decision about whether to import rgdal functions. I'll get started today.

On 20 July 2015 at 13:08, Hadley Wickham notifications@github.com wrote:

Is there anything I can do to help kick this into motion? I'd really like to be able to cite it in the book, which I need to have done in 3 weeks (rnaturalearth doesn't need to be on CRAN, but it at least needs to be heading in that direction). I think it's reasonable to just focus on the data (rather than the tools) for the first version.

— Reply to this email directly or view it on GitHub https://github.com/AndySouth/rnaturalearth/issues/2#issuecomment-122864934 .

[andysouth]

@hadley I've made a start, take a look. Question : do you think ne_download() which requires rgdal should be in the same or a different package ? I'm now away until Thursday next week can get back to it then.

[hrbrmstr]

So, with the R-pkg-devel announcement yesterday-ish - https://stat.ethz.ch/pipermail/r-package-devel/2015q3/000332.html - I found that the new caretaker has: https://github.com/adeckmyn/mapdataNE

Perhaps a ping to Alex shld be done before this effort continues?

On Fri, Jul 24, 2015 at 11:59 AM, Andy South notifications@github.com wrote:

@hadley https://github.com/hadley I've made a start, take a look. Question : do you think ne_download() which requires rgdal should be in the same or a different package ? I'm now away until Thursday next week can get back to it then.

— Reply to this email directly or view it on GitHub https://github.com/AndySouth/rnaturalearth/issues/2#issuecomment-124566337 .

[andysouth]

Thanks @hrbrmstr for the heads up.

Hello Alex, @adeckmyn, well done for taking on the maps packages and starting to get updated boundaries in there. rnaturalearth is a new package that aims to make current Natural Earth data more accessible to R users.

My initial thought is that we have slightly different objectives, rnaturalearth provides sp objects rather than the format used by maps and mapdata.

I had Natural Earth data in rworldmap and did a lot of formatting, in rnaturalearth I aim (following the suggestion from @hadley) to keep the data as close as possible to Natural Earth thus minimising maintenance and improving reproducibility.

Can you see a way we can share efforts ?

[hrbrmstr]

Also, @adeckmyn, if you want/need any help with the maps pkgs you just took over, @ironholds & I wld be willing to pitch in.

On Fri, Jul 31, 2015 at 8:17 AM, Andy South notifications@github.com wrote:

Thanks @hrbrmstr https://github.com/hrbrmstr for the heads up.

Hello Alex, @adeckmyn https://github.com/adeckmyn, well done for taking on the maps packages and starting to get updated boundaries in there. rnaturalearth is a new package that aims to make current Natural Earth data more accessible to R users.

My initial thought is that we have slightly different objectives, rnaturalearth provides sp objects rather than the format used by maps and mapdata.

I had Natural Earth data in rworldmap and did a lot of formatting, in rnaturalearth I aim (following the suggestion from @hadley https://github.com/hadley) to keep the data as close as possible to Natural Earth thus minimising maintenance and improving reproducibility.

Can you see a way we can share efforts ?

— Reply to this email directly or view it on GitHub https://github.com/AndySouth/rnaturalearth/issues/2#issuecomment-126673385 .

[adeckmyn]

Hi, News travels fast, even my github has been tracked:) Thanks for the offers of help & collabration. I will be posting to R-sig-Geo pretty soon, but maybe this is a good moment to discuss a bit, because there is indeed some overlap in our efforts.

The mapdataNE package on github will probably never be released on CRAN in its current form. After discussions with Ray Brownrigg, we decided that it was probably more useful to update the maps package itself (and possibly mapdata). The work got delayed some months (I now have a baby son who insists he's more important than world maps :-) but I am more or less ready with a beta version of maps which includes the Natural Earth 1:50m world map as the new "world" database. 1:10m as a replacement for worldHires is a different matter, because the resolution is not as high. The 1/110m version, by the way, turned out to be impossible to convert. The polygons don't always align exactly, so converting to polylines would be hard work.

The format is quite different, of course. I plan to upload my toolbox (mapEdit) when I find time this weekend. Not to be used without parental guidance, though. It's a completely undocumented load of scripts, including some to import Natural Earth and turn it into a "maps" database. Still a lot of manual work is involved, though, so I hope NE does not update too often.

[adeckmyn]

I should add that "maps" will only import the world country database, not much else from Natural Earth (possibly lakes and rivers). I think it should stay a simple & fast tool for basic mapping.

[andysouth]

Thanks @adeckmyn, lets keep in touch about what we're up to. Sounds like we have slightly different aims, but lets keep an eye out for where we can share effort. Hoping that your new son will come to love maps one day ;-)

[hadley]

I wonder if it might be possible to automatically translate from the sp format the rnaturalearth wants to the format used by maps(). It would be really nice if you could use an sp dataset with the maps package.

[adeckmyn]

It's not straightforward, I think. maps() uses polylines in a rather specific format. I have gone some way in transforming automatically (Natural Earth world map comes only as polygons, as far as I remember), but there is quite a way to go. It took me quite a bit of manual work. Another complication is that maps() actually reads its data from disc, not from memory. So without modifying the source code drastically (which I am currently not inclined to do in a hurry), you would have to write the data to disc first. Not a very fast procedure.

On the other hand, transforming a clean set of polylines (e.g. rivers) to a file that maps can use, should be relatively straightforward. My mapEdit package (just uploaded), which is in no way meant as a CRAN package, contains routines that already do some parts of this. In a completely unorganised way, alas.

[adeckmyn]

I may have answered too quickly yesterday. The world maps are not the simplest format (polylines and polygon lists), but map() can also directly use data that is already in polygons (as.polygon=TRUE). Such map-class objects are simple lists with $x, $y and $names. There are more elements, and I'm not sure which ones are vital. The output of map(...as.polygon=TRUE) is such an object and can be used as a map database. So using that format, it could in fact be quite easy to translate sp class objects to be used in map().

[adeckmyn]

Some months ago, @hadley proposed the possibility to convert the sp format to something useful for the 'maps' package. It turned out to be quite simple. I have had several requests for admin-1 borders for particular countries, and now have a simple tool to extract them directly from Natural Earth shape files (well, by first importing to a SpatialPolygonDataFrame). But it would be even simpler to use if it were linked to a rnaturalearth. Most simple use is to create a list with elements $x, $y and $names which contains all the polygons. That can already be used in 'maps', but without some of the advanced possibilities of 'maps'. I'm working on a new package that will convert it to polylines format and save in the same format as used by 'maps' and 'mapdata'. But simply extracting the polygons is often enough and already works fine. Probably best to have it all in a separate package from 'rnaturalearth' itself (though I'd be willing to contribute it), but the possibility to create map datasets on the fly without having to download yourself, seems appealing.