Having issues with loader.py

[mtravis]

Really keen to give this a go and been testing it out by trying to load addressbase plus gml into a postrgesql database.

I've downloaded all the necessary and changed the loader.config so its as such:

---

The directory containing your source files.

All supported files in the specified directory and

it's decendents will be loaded.

src_dir=C:\GML_Loader\data\import

The directory used to store the translated data

if writing to a file based format such as ESRI

Shape, MapInfo TAB etc.

out_dir=C:\loader_data\export

The directory used to store temporary working files

during loading.

tmp_dir=C:\loader_data\temp

The ogr2ogr command that will be used to load the data.

Here you can specify the destination format and any

associated settings (for example database connection

details if you are writing to PostGIS).

ogr_cmd=ogr_cmd=ogr2ogr -overwrite -f PostgreSQL PG:dbname=test active_schema=public host=localhost port=5432 user=postgres password=****\ C:\GML_Loader\data\import\AddressBase_Plus_SX9090.gml

Advanced configuration

The command used to prepare the source

data so it is suitable for loading with OGR

prep_cmd=python prepgml4ogr.pyC:\GML_Loader\data\import\AddressBase_Plus_SX9090.gml prep_osgml.prep_addressbase

Optional OGR .gfs file used to define the

feature attributes and geometry type of

the feautes read from the GML.

gfs_file=../gfs/osmm_topo_postgres.gfs

Whether to output debug messages and keep

temporary files (True or False)

debug=False

---

Once I've done this I then run the C:\python27\python loader.py loader.config command.

I then get this error message:

---

C:\GML_Loader\python>C:\python27\python loader.py loader.config Traceback (most recent call last): File "loader.py", line 218, in main() File "loader.py", line 198, in main config.append([parts[0], os.path.expandvars(parts[1])]) IndexError: list index out of range

---

I suppose what I firstly need to establish is if the config file has been edited properly (which would be a miracle) and if that has why I'm getting the error message above.

Thanks in ad

[mtravis]

I have since modified the config to include a \ at the end of the temp path and used \ in the gps path.

Matt

[walkermatt]

@mtravis it looks as though your config file includes the path to your GML on a line by itself which is what's tripping it up. Try using this config based on what you've provided above: https://gist.github.com/2868386. I've turned debugging on which will mean that the temporary files that Loader creates during a run will not get cleaned up so I'd try running it first on a small sample of files.

Thanks,

Matt.

[mtravis]

Thanks for getting back to me on that @walkermatt. I updated the config and ran it again.

src_dir=C:\GML_Loader\Data\import

out_dir=C:\GML_Loader\Data\export

tmp_dir=C:\GML_Loader\Data\temp\

ogr_cmd=ogr_cmd=ogr2ogr -overwrite -f PostgreSQL PG:dbname=test active_schema=public host=localhost port=5432 user=postgres password=Dartm00r $file_path

prep_cmd=python prepgml4ogr.py $file_path prep_osgml.prep_addressbase

gfs_file=C:\GML_Loader\gfs\addressbase_standard_postgres.gfs

debug=True

This time I got the following error message in cmd window:

C:\GML_Loader\python>C:\python27\python loader.py loader.config Config: {'tmp_dir': 'C:\GML_Loader\Data\temp\', 'out_dir': 'C:\GML_Loader\ Data\export', 'ogr_cmd': 'ogr_cmd=ogr2ogr -overwrite -f PostgreSQL PG:dbname=te st active_schema=public host=localhost port=5432 user=postgres password=Dartm00r $file_path', 'gfs_file': 'C:\GML_Loader\gfs\addressbase_standard_postg res.gfs', 'debug': 'True', 'src_dir': 'C:\GML_Loader\Data\import', 'prep_cmd' : 'python prepgml4ogr.py $file_path prep_osgml.prep_addressbase'} Processing: C:\GML_Loader\Data\import\AddressBase_Plus_SX9090.gml Prepared file: C:\GML_Loader\Data\temp\loader_20120606_164410\AddressBase_Plus_S X9090 Prep command: python prepgml4ogr.py C:\GML_Loader\Data\import\AddressBase_Plus_S X9090.gml prep_osgml.prep_addressbase Loading: C:\GML_Loader\Data\import\AddressBase_Plus_SX9090.gml OGR command: ogr_cmd=ogr2ogr -overwrite -f PostgreSQL PG:dbname=test active_sche ma=public host=localhost port=5432 user=postgres password=Dartm00r C:\GML_Loader \Data\temp\loader_20120606_164410\AddressBase_Plus_SX9090 Traceback (most recent call last): File "loader.py", line 218, in main() File "loader.py", line 207, in main loader.run(config) File "loader.py", line 78, in run self.load() File "loader.py", line 133, in load self.load_file(root, name) File "loader.py", line 169, in load_file subprocess.call(ogr_args, stderr=sys.stderr) File "C:\python27\lib\subprocess.py", line 486, in call return Popen(_popenargs, *_kwargs).wait() File "C:\python27\lib\subprocess.py", line 672, in init errread, errwrite) File "C:\python27\lib\subprocess.py", line 882, in _execute_child startupinfo) WindowsError: [Error 2] The system cannot find the file specified

I checked and there is definitely a subprocess.py in my python27 folder. Is this what the folder is referring to?

Thanks

@mtravis

[walkermatt]

@mtravis it looks as though you've not got Python in your PATH environment variable.

You can either set it system wide (something like: right click My Computer > Properties > Advanced > Environment Variables then add ;C:\Python27 to the end of the Path entry in "System variables") or just for your command line session by executing: SET PATH=%PATH%;C:\Python27\.

The alternative is to add the full path to the Python interpretor to the prep_cmd in your config as described here: https://github.com/AstunTechnology/Loader/wiki/Installation

Hope that helps,

Matt.

[mtravis]

Hi Matt

I have just checked and the system PATH and C:Python27 is included plus I can access Python from normal cmd window but I'm still getting the same error message as above.

Also, how can I access the PGdump driver you mention in this thread.

http://osgeo-org.1560.n6.nabble.com/OpenData-and-QGIS-td3741400.html

Cheers

Matt T

[walkermatt]

@mtravis looking again at the output it looks like the utility ogr2ogr can't be found. The easiest approach is to install just GDAL (which includes the OGR utilities) from the OSGeo4W installer and then run Loader from the OSGeo4W shell as described in the install notes on the wiki.

Cheers,

Matt.

[mtravis]

Hi Matt

Still having problems here. We have defintely got the GDAL installed and I sue the ogr2ogr utility in FWtools so I don't think it is that.

To make sure I went to OSGeo4W advanced installer and ensured all the GDAL command utilities. I don't have to select the source box for these do I?

Thanks for your help with this. I really want to give this a go!

Matt T

[walkermatt]

@mtravis have you tried running Loader from the FWTools shell?

[mtravis]

@walker_matt

I did mean to but I forgot. I will try it and let you know how I get on.

Cheers

Matt T

On Fri, Jul 20, 2012 at 4:01 PM, Matt Walker < reply@reply.github.com

wrote:

@mtravis have you tried running Loader from the FWTools shell?

---

Reply to this email directly or view it on GitHub: https://github.com/AstunTechnology/Loader/issues/5#issuecomment-7132377

[mtravis]

Hi @walkermatt

I've tried running this with FWtools but alas to no avail. So I had a good look at the code and realised that the prep cmd was written incorrectly.

I ran the loader again and it seemed to be doing something very briefly but then stopped again. There was more code this time. Could you take a look when you have a minute it for me please.

Cheers

@mtravis

C:\GML_Loader\python>C:\python27\python loader.py loader.config debug=True Config: {'tmp_dir': 'C:\GML_Loader\Data\temp\', 'out_dir': 'C:\GML_Loader\ Data\export', 'ogr_cmd': 'ogr_cmd=ogr2ogr -overwrite -f PostgreSQL PG:dbname=te st active_schema=public host=localhost port=5432 user=postgres password=Dartm00r $file_path', 'gfs_file': 'C:\GML_Loader\gfs\addressbase_standard_postg res.gfs', 'debug': 'True', 'src_dir': 'C:\GML_Loader\Data\import', 'prep_cmd' : 'C:\Python27\Python.exe prepgml4ogr.py $file_path prep_osgml.prep_osmm_t opo'} Processing: C:\GML_Loader\Data\import\M00001177-SX6085-5c6309.gml Prepared file: C:\GML_Loader\Data\temp\loader_20120724_102619\M00001177-SX6085-5 c6309 Prep command: C:\Python27\Python.exe prepgml4ogr.py C:\GML_Loader\Data\import\M0 0001177-SX6085-5c6309.gml prep_osgml.prep_osmm_topo Traceback (most recent call last): File "prepgml4ogr.py", line 39, in from xml.sax import saxutils File "C:\OSGeo4W\apps\Python27\lib\xml\sax\saxutils.py", line 6, in import os, urlparse, urllib, types File "C:\OSGeo4W\apps\Python27\lib\urllib.py", line 26, in import socket File "C:\OSGeo4W\apps\Python27\lib\socket.py", line 47, in import _socket ImportError: DLL load failed: %1 is not a valid Win32 application. Loading: C:\GML_Loader\Data\import\M00001177-SX6085-5c6309.gml OGR command: ogr_cmd=ogr2ogr -overwrite -f PostgreSQL PG:dbname=test active_sche ma=public host=localhost port=5432 user=postgres password=Dartm00r C:\GML_Loader \Data\temp\loader_20120724_102619\M00001177-SX6085-5c6309 Traceback (most recent call last): File "loader.py", line 218, in main() File "loader.py", line 207, in main loader.run(config) File "loader.py", line 78, in run self.load() File "loader.py", line 133, in load self.load_file(root, name) File "loader.py", line 169, in load_file subprocess.call(ogr_args, stderr=sys.stderr) File "C:\OSGeo4W\apps\Python27\lib\subprocess.py", line 493, in call return Popen(_popenargs, *_kwargs).wait() File "C:\OSGeo4W\apps\Python27\lib\subprocess.py", line 679, in init errread, errwrite) File "C:\OSGeo4W\apps\Python27\lib\subprocess.py", line 893, in _execute_child

startupinfo)

WindowsError: [Error 2] The system cannot find the file specified

[mtravis]

Just realised that the line ImportError: DLL load failed: %1 is not a valid Win32 application. might have something to do with 32/64 bit.

I have also just realised that the OSgeo4W command shell I'm using states it is using GDAL 1.7

GDAL 1.7.0b2, FWTools 2.4.7, released 2010/01/19

I have used the advanced installer and that is showing 1.9 in the command utilities. I must be missing something somewhere but just can't figure it out.

Do you do call outs?

[mtravis]

Hello again Matt

I've done a bit of tinkering and seem to be getting somewhere. Here's the new code:

C:\GML_Loader\python>C:\python27\python loader.py loader.config debug=True Config: {'tmp_dir': 'C:\GML_Loader\Data\temp\', 'out_dir': 'C:\GML_Loader\ Data\export', 'ogr_cmd': 'ogr2ogr -append -skipfailures -f PostgreSQL PG:dbname =test active_schema=public host=localhost port=5432 user=postgres password=Dartm 00r $file_path', 'gfs_file': '../gfs/osmm_topo_shape_qgis.gfs', 'debug': 'True', 'src_dir': 'C:\GML_Loader\Data\import', 'prep_cmd': 'python prepgml4ogr.py $ file_path prep_osgml.prep_osmm_topo'} No valid gfs file found, output schema and geometry types will be determed dynam ically by OGR Processing: C:\GML_Loader\Data\import\M00001177-SX6085-5c6309.gml Prepared file: C:\GML_Loader\Data\temp\loader_20120724_162428\M00001177-SX6085-5 c6309 Prep command: python prepgml4ogr.py C:\GML_Loader\Data\import\M00001177-SX6085-5 c6309.gml prep_osgml.prep_osmm_topo Loading: C:\GML_Loader\Data\import\M00001177-SX6085-5c6309.gml OGR command: ogr2ogr -append -skipfailures -f PostgreSQL PG:dbname=test active_s chema=public host=localhost port=5432 user=postgres password=Dartm00r C:\GML_Loa der\Data\temp\loader_20120724_162428\M00001177-SX6085-5c6309 FAILURE: Unable to open datasource `active_schema=public' with the following drivers. -> ESRI Shapefile -> MapInfo File -> UK .NTF -> SDTS -> TIGER -> S57 -> DGN -> VRT -> REC -> Memory -> BNA -> CSV -> NAS -> GML -> GPX -> KML -> GeoJSON -> Interlis 1 -> Interlis 2 -> GMT -> SQLite -> ODBC -> PGeo -> OGDI -> PostgreSQL -> MySQL -> XPlane -> AVCBin -> AVCE00 -> DXF -> Geoconcept -> GeoRSS -> GPSTrackMaker -> VFK Loaded 2 files

using this config:

The directory containing your source files.

All supported files in the specified directory and

it's decendents will be loaded.

src_dir=C:\GML_Loader\Data\import

The directory used to store the translated data

if writing to a file based format such as ESRI

Shape, MapInfo TAB etc.

out_dir=C:\GML_Loader\Data\export

The directory used to store temporary working files

during loading.

tmp_dir=C:\GML_Loader\Data\temp\

The ogr2ogr command that will be used to load the data.

Here you can specify the destination format and any

associated settings (for example database connection

details if you are writing to PostGIS).

ogr_cmd=ogr2ogr -append -skipfailures -f PostgreSQL PG:'dbname=OSMM_test active_schema=public host=localhost port=5432 user=postgres password=Dartm00r $file_path

Advanced configuration

The command used to prepare the source

data so it is suitable for loading with OGR

prep_cmd=python prepgml4ogr.py $file_path prep_osgml.prep_osmm_topo

Optional OGR .gfs file used to define the

feature attributes and geometry type of

the feautes read from the GML.

gfs_file=../gfs/osmm_topo_shape_qgis.gfs

Whether to output debug messages and keep

temporary files (True or False)

debug=True

[mtravis]

Hi Matt

Think I've actually managed to get it working at last! It looks like I was missing an apostrophe after the password in the ogr cmd.

Unbelievable eh! The data is in the postgres and I've just viewed it in QGIS as a PostGIS layer. Fantastic stuff! Got to try and work out how to style it now.

Thanks for all your help. Might still need to get you down at some point.

Cheers Matt

[walkermatt]

@mtravis Really glad you've managed to get up and running :-)