j-norwood-young / SA-Maps

South African Shapefiles, GIS data and other useful mapping stuff
34 stars 30 forks source link

Parsing tools #1

Open stephenfrank opened 10 years ago

stephenfrank commented 10 years ago

Hey Jason,

I'm working on a project requiring a city/suburb data and this data looks pretty accurate (or about as accurate as SA can be). This is after I bought the www.geopostcodes.com data and was pretty disappointed with it.

Firstly, is there any one-stop (hopefully free) source for this data?

If not, do you have any hot tips for parsing these .shx, .shp and *.dbf files into something manageable so that I can build this database?

I've opened up the .dbf files but they lack the lng/lat data. I've used this parsing library (https://github.com/fillerwriter/PHP5-Shape-File-Parser) which is able to read the .shp file but doesn't seem to combine that with the .dbf data. Argh!

-Stephen

adieyal commented 10 years ago

Hi Stephen

What data are you looking for? The census contains information on subplaces and mainplaces (which are more or less analogous to suburbs and towns).

with regards to parsing shape files - I generally use pyshp ([1]) but I'm not sure what the equivalent php tool is.

Happy to help more if I can

[1] http://code.google.com/p/pyshp/

j-norwood-young commented 10 years ago

Hey Stephen

I've been using Topojson to generate stuff I can use straight on the unterwebs. It runs on Node.js, but should work through Homebrew. (I just popped up a Vagrant instance for it, based on https://github.com/semmypurewal/node-dev-bootstrap, but I had that lying around.)

Also check out Tilemill, you can import the shape files straight in there and start making cool maps right away. http://www.mapbox.com/tilemill/

What OS you on? Adi knows all the cool Linux tools, and I'm starting to get my head around the Mac tools.

J

On 26 Sep 2013, at 6:24 PM, stephenfrank notifications@github.com wrote:

Hey Jason,

I'm working on a project requiring a city/suburb data and this data looks pretty accurate (or about as accurate as SA can be). This is after I bought the www.geopostcodes.com data and was pretty disappointed with it.

Firstly, is there any one-stop (hopefully free) source for this data?

If not, do you have any hot tips for parsing these .shx, .shp and *.dbf files into something manageable so that I can build this database?

I've opened up the .dbf files but they lack the lng/lat data. I've used this parsing library (https://github.com/fillerwriter/PHP5-Shape-File-Parser) which is able to read the .shp file but doesn't seem to combine that with the .dbf data. Argh!

-Stephen

— Reply to this email directly or view it on GitHub.

stephenfrank commented 10 years ago

@adieyal I was looking to use the "main places" and "sub places" as something analogous to towns/suburbs though I can see how they might not be exactly the same thing. I'm willing to see how it turns out.

@j-norwood-young I'm on OSX/Linux and will check out those tools.

That all being said, I've just noticed that the .dbf file opens in OpenOffice and has a column "OBJECTID,N,9,0" that correlates to the exactly to the number of polygons in the .shp file. I'm going to take a crack at munging them together.

adieyal commented 10 years ago

Yup - dbf/shp are an archaic format. The only index between them is line number. If you just need to extract the data, then openoffice is fine. If you need to do anything more than that, I really suggest you find an appropriate library. You're walking down a well travelled path.

Regarding main places vs sub places - know that in a metro area such as Cape Town, large suburbs are considered to be main places. For instance Khayelitsha is a main place

Adi

Adi Eyal Data Specialist phone: +27 78 014 2469 skype: adieyalcas linkedin: http://za.linkedin.com/pub/dir/Adi/Eyal

On 27 September 2013 10:25, stephenfrank notifications@github.com wrote:

@adieyal https://github.com/adieyal I was looking to use the "main places" and "sub places" as something analogous to towns/suburbs though I can see how they might not be exactly the same thing. I'm willing to see how it turns out.

@j-norwood-young https://github.com/j-norwood-young I'm on OSX/Linux and will check out those tools.

That all being said, I've just noticed that the .dbf file opens in OpenOffice and has a column "OBJECTID,N,9,0" that correlates to the exactly to the number of polygons in the .shp file. I'm going to take a crack at munging them together.

— Reply to this email directly or view it on GitHubhttps://github.com/j-norwood-young/SA-Maps/issues/1#issuecomment-25230016 .

WillemBadenhorst commented 1 year ago

Dear Jason. Seems like there is an issue with the zipped voting districts files. Can you please reload or is it me that doesn't know how to download?

Kind regards Willem

MotsumiDarts commented 1 year ago

You can get updated Cadastral data directly from the state. Really simple fill in a form send it in they will provide a shared drive and download it. About 13G of data.

Would be awesome to get a way to get continously updated data. Seems like all data available still is a few months old??

Regards

D

On Fri, 21 Jul 2023, 11:11 WillemBadenhorst, @.***> wrote:

Dear Jason. Seems like there is an issue with the zipped voting districts files. Can you please reload or is it me that doesn't know how to download?

Kind regards Willem

— Reply to this email directly, view it on GitHub https://github.com/j-norwood-young/SA-Maps/issues/1#issuecomment-1645261797, or unsubscribe https://github.com/notifications/unsubscribe-auth/BAZ4IRRSYKWAHNEINMY7PC3XRJBSLANCNFSM4AIJ7AKA . You are receiving this because you are subscribed to this thread.Message ID: @.***>

WillemBadenhorst commented 1 year ago

Thanks very much!!!

www.mandalagis.com

Willem Badenhorst

PLATO Professional GISc Practitioner, PGP1411

South African Planning Institute, 12578

Geo-Information Society of South Africa, YSXB­7391

Postnet Suite 402; Private Bag x025; Lynnwood Ridge; Pretoria; 0040; Gauteng

Cell: 082 467 5110 - LinkedIn: http://goog_644993275 http://goog_644993275www.linkedin.com/in/wjbadenhorst

www.mandalagis.com Click to see map http://maps.google.co.za/maps/place?cid=17036365196488124616&q=83+Alcade+Road,+Pretoria,+Gauteng&hl=en&authuser=0&t=h&cd=1&cad=src:pplink&ei=DZOzT7yMDJDj8QbHo5igCA Click to see StreetView http://cbk1.google.com/cbk?output=thumbnail&cb_client=maps_sv&thumb=2&thumbfov=60&ll=-25.767849,28.280866&cbll=-25.767619,28.280986&thumbpegman=1&w=300&h=118

Disclaimer

This e-mail is intended solely for the person to whom it is addressed and may contain confidential or privileged information. If you have received it in error please notify us immediately by return e-mail or by telephone +27 (0)82 467 5110 and delete the e-mail. You may not copy, distribute, disclose or take any action in reliance on it. While we take reasonable precautions to minimise the risk of an attachment containing computer viruses this possibility still exists. We cannot accept liability for any damage which you may sustain as a result of a software virus. We recommend that you carry out your own virus checks before opening an attachment. We do not accept responsibility for any amendments made to any attached documents without our approval. We do not accept responsibility for private or personal messages e-mailed from our domain.

On Fri, Jul 21, 2023 at 11:41 AM MotsumiDarts @.***> wrote:

You can get updated Cadastral data directly from the state. Really simple fill in a form send it in they will provide a shared drive and download it. About 13G of data.

Would be awesome to get a way to get continously updated data. Seems like all data available still is a few months old??

Regards

D

On Fri, 21 Jul 2023, 11:11 WillemBadenhorst, @.***> wrote:

Dear Jason. Seems like there is an issue with the zipped voting districts files. Can you please reload or is it me that doesn't know how to download?

Kind regards Willem

— Reply to this email directly, view it on GitHub < https://github.com/j-norwood-young/SA-Maps/issues/1#issuecomment-1645261797>,

or unsubscribe < https://github.com/notifications/unsubscribe-auth/BAZ4IRRSYKWAHNEINMY7PC3XRJBSLANCNFSM4AIJ7AKA>

. You are receiving this because you are subscribed to this thread.Message ID: @.***>

— Reply to this email directly, view it on GitHub https://github.com/j-norwood-young/SA-Maps/issues/1#issuecomment-1645305486, or unsubscribe https://github.com/notifications/unsubscribe-auth/BBNJAVUHTXBXLXGQ7XPKQPTXRJFELANCNFSM4AIJ7AKA . You are receiving this because you commented.Message ID: @.***>