Standard PostgreSQL support?

TimJMartin commented 9 years ago

I have noticed that the standard PostgreSQL OGR format is not standard and it has now become a requirement for the tool I am trying to build.

I wasn't sure how to do this so attempted to manually edit this https://github.com/naturalatlas/node-gdal/blob/master/deps/libgdal/libgdal_formats.gypi with an entry for 'pg'. I then created a new ogr_pg.gyp file in the gyp-formats folder, and in it i reference the files in https://github.com/naturalatlas/node-gdal/tree/master/deps/libgdal/gdal/ogr/ogrsf_frmts/pg

I then tried

npm install node-gdal --build-from-source

And got a horrible number of errors and realised that I should use:

npm install node-gdal --build-from-source --shared_gdal

But again received errors during the build.

I have now just read all the closed and open issues about 'builds' and found this comment: https://github.com/naturalatlas/node-gdal/issues/88#issuecomment-74834030

Which makes me think the routine is to build node-gdal using

npm install node-gdal --build-from-source

And then have the deplibs folder containing the folder structure as outlined in that comment and then running

npm install --build-from-source --msvs_version=2013 --shared_gdal=c:/projects/node-gdal/deplibs/

but I do not know where the contents for deplibs comes from. It doesn't seem to be created during a standard build routine.

Really sorry for asking such a NOOB question. Am sure it is super simple and am just overlooking a solution on Windows.

BTW using Windows 7 x64 and have MSVS2013

wilhelmberg commented 9 years ago

I took a look at this but it's not that simple. Since node-gdal aims to be self contained libpq would have to be pulled in to make this work.

Maybe @brandonreavis or @brianreavis could add their $0.02 if they want to include support for PG.

I took a first stab at https://github.com/naturalatlas/node-gdal/compare/issue-132?diff=split&name=issue-132

It's a start, but

everything hardcoded to my local paths
build is still failing, because of some missing references

Feel free to delete the branch, if you decide against PG support (@brandonreavis @brianreavis).

brianreavis commented 9 years ago

@TimJMartin Will you always have postgres on the machine you're trying to use node-gdal on, or are you looking for a self-contained build that can work anywhere? For the latter, like @BergWerkGIS said, we're going to have to pull in libpq and all its dependencies and build them from source to static link (as far as I know). Refs https://github.com/naturalatlas/node-gdal/issues/75 (which I'd love to have, but I'm not sure is practical right now)

For the first option, we can at least make npm install node-gdal --build-from-source include postgres support if it finds it one the system. What errors are you getting when running that? Really unfamiliar with Windows building... definitely @BergWerkGIS's turf.

TimJMartin commented 9 years ago

@brianreavis @BergWerkGIS

Yes people would have to have PostgreSQL and PostGIS installed. So hopefully that makes life a lot easier to include it as a format option.

I am aiming to use the node module 'pg' to setup the database tables then load in the data using

--config PG_USE_COPY YES

As our data is pretty big. The reason I couldn't just use PGDump is because I would then like to enable export back out of PostGIS.

It is more than likely it is my routine that is wrong.

I have a package.json with the dependencies pointing at

"dependencies": { "gdal": "naturalatlas/node-gdal#win-node-4.2.1" }

Then I do

npm install gdal --build-from-source --msvs_version=2013

This builds successfully, i then edit the

libgdal_formats.gypi deps/libgdal/gyp-formats/ogr_pg.gyp

And then run

npm install gdal --build-from-source --msvs_version=2013

But this seems to overwrite my edits to the files above by downloading all the source files again.

So as I said it is probably my routine that is the issue.

wilhelmberg commented 9 years ago

@TimJMartin

The branch you would have to work against currently is issue-132:

"dependencies": {
  "gdal": "naturalatlas/node-gdal#issue-132"
}

I made it compile with the library and headers that come with a default Postgres install.

how I did it:

Paths in the gyp files are still hardcoded to my Postgres install: C:/Program Files/PostgreSQL/9.4/lib.

I used VS2015 and I'm not sure about the consequences when Postgres libs are built with another compiler. Might lead to unpredictable behavior.

SET PATH=C:\Python27;%PATH%
SET PATH=set path=C:\Program Files\PostgreSQL\9.4\bin;%PATH% <- for pg_config
git clone https://github.com/naturalatlas/node-gdal.git
cd node-gdal
git checkout issue-132
npm install --build-from-source --msvs_version=2015

Directly within the clone directory I used this:

var gdal = require('./');
//console.log(gdal.drivers.getNames());
try{
    var ds = gdal.open("PG:dbname='XXXX' host='localhost' port='5432' user='XXXX' password='XXXX'");
    ds.layers.forEach(function (lyr){
        console.log(lyr.name);
    });
}
catch(err){
    console.log(err);
    console.log(gdal.lastError);
}

brianreavis commented 9 years ago

(just referencing this here... forgot it existed: https://github.com/naturalatlas/node-gdal/issues/20)

TimJMartin commented 9 years ago

@BergWerkGIS

Just tried your routine and it seemed to have worked for the new pg but got the following build error:

  libgdal_ogr_wasp_frmt.vcxproj -> C:\Temp\issue132\node-gdal\build\Release\\libgdal_ogr_wasp_frmt.lib
gyp ERR! build error
gyp ERR! stack Error: `C:\Program Files (x86)\MSBuild\12.0\bin\msbuild.exe` failed with exit code: 1
gyp ERR! stack     at ChildProcess.onExit (C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\lib\build.js:2
70:23)
gyp ERR! stack     at emitTwo (events.js:87:13)
gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:200:12)
gyp ERR! System Windows_NT 6.1.7601
gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\node_modules\\node
-gyp\\bin\\node-gyp.js" "build" "--fallback-to-build" "--module=C:\\Temp\\issue132\\node-gdal\\lib\\binding\\gdal.node"
"--module_name=gdal" "--module_path=C:\\Temp\\issue132\\node-gdal\\lib\\binding"
gyp ERR! cwd C:\Temp\issue132\node-gdal
gyp ERR! node -v v4.2.1
gyp ERR! node-gyp -v v3.0.3
gyp ERR! not ok
node-pre-gyp ERR! build error
node-pre-gyp ERR! stack Error: Failed to execute 'node-gyp.cmd build --fallback-to-build --module=C:\Temp\issue132\node-
gdal\lib\binding\gdal.node --module_name=gdal --module_path=C:\Temp\issue132\node-gdal\lib\binding' (1)
node-pre-gyp ERR! stack     at ChildProcess.<anonymous> (C:\Temp\issue132\node-gdal\node_modules\node-pre-gyp\lib\util\c
ompile.js:83:29)
node-pre-gyp ERR! stack     at emitTwo (events.js:87:13)
node-pre-gyp ERR! stack     at ChildProcess.emit (events.js:172:7)
node-pre-gyp ERR! stack     at maybeClose (internal/child_process.js:818:16)
node-pre-gyp ERR! stack     at Process.ChildProcess._handle.onexit (internal/child_process.js:211:5)
node-pre-gyp ERR! System Windows_NT 6.1.7601
node-pre-gyp ERR! command "C:\\Program Files\\nodejs\\node.exe" "C:\\Temp\\issue132\\node-gdal\\node_modules\\node-pre-g
yp\\bin\\node-pre-gyp" "install" "--fallback-to-build"
node-pre-gyp ERR! cwd C:\Temp\issue132\node-gdal
node-pre-gyp ERR! node -v v4.2.1
node-pre-gyp ERR! node-pre-gyp -v v0.6.12
node-pre-gyp ERR! not ok
Failed to execute 'node-gyp.cmd build --fallback-to-build --module=C:\Temp\issue132\node-gdal\lib\binding\gdal.node --mo
dule_name=gdal --module_path=C:\Temp\issue132\node-gdal\lib\binding' (1)

npm ERR! Windows_NT 6.1.7601
npm ERR! argv "C:\\Program Files\\nodejs\\node.exe" "C:\\Program Files\\nodejs\\node_modules\\npm\\bin\\npm-cli.js" "ins
tall" "--build-from-source" "--msvs_version=2013"
npm ERR! node v4.2.1
npm ERR! npm  v2.14.7
npm ERR! code ELIFECYCLE
npm ERR! gdal@0.7.2 install: `node-pre-gyp install --fallback-to-build`
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the gdal@0.7.2 install script 'node-pre-gyp install --fallback-to-build'.
npm ERR! This is most likely a problem with the gdal package,
npm ERR! not with npm itself.

Does this look like a problem with using MSVC2013 rather than 2015?

I will try and launch an AWS EC2 windows box and try it on a clean machine of PostgreSQL 9.4 and MSVC2015

wilhelmberg commented 9 years ago

I see you have npm ERR! node v4.2.1. I tried it with 0.10.40 and 0.12.7. I branched off master and not the win4 branch.

TimJMartin commented 9 years ago

@BergWerkGIS ok just fired up an EC2 so will get 0.12.7 installed and MSVS 2015 Community Edition

TimJMartin commented 9 years ago

@BergWerkGIS so got it working using your routine

I used MSVS 2013 Express as described in the node gyp requirements for Windows 7.

Installed PostgreSQL 9.4 64bit and PostGIS 2.2 64bit

Installed Node 0.12.7

Followed your instructions and the build successfully worked as did your script to connect and see layers in the database.

That is excellent!

So would the next steps be to include that along with the 4.2.1 branch and do a release at some point?

wilhelmberg commented 9 years ago

So would the next steps be to include that along with the 4.2.1 branch and do a release at some point?

Yes, suppose so, but I think it might take a while, as it would require some additional effort. @brianreavis I'm all ears on your opinion about this.

I'm tight on available free resources right now, so might only be able to help out again for small Windows specific tasks.

disable Postgres support by default. How big is the demand? Looking through the issues this seems to be the first request for Postgres support?
make it available when building from source, e.g. npm install --build-from-source --pg_enable=1?
incorporate this change in the gyp files
check if pg_config is available. If not continue without PG support or bail out?
parse the output of pg_config
create new tests for Postgres
- setup up test database during test
- change CI scripts (travis, AppVeyor) to setup test db and run Postgres tests
anything else?

TimJMartin commented 9 years ago

@BergWerkGIS sorry for slow reply

totally understand. You guys have been super helpful and I want to say a big thank you for all your hard work to get vsigzip sorted and the electron version done.

If and when you get time to look at this let me know as I can test a version

naturalatlas / node-gdal

Standard PostgreSQL support? #132

how I did it: