Closed RamiNaim closed 4 years ago
bwitham
commented
4 years ago So you want to create your own instance of a hootapidb separate from the one hoot installs by default and conflate data from it? Can you read and write successfully from the default installed hootapidb? You stated that you created the 'DcGisRoads' table in your database beforehand. You don't need to do that. When you run convert it will create that layer table for you automatically.
RamiNaim
commented
4 years ago @bwitham Thanks for a quick reply.
I have created DcGisRoads table beforehand because while running the same command on an empty database I encountered following error:
Attempting to open _db hootapidb://10.1.10.128:5432/hoot/DcGISRoads but found zero tables. Does the DB exist? Has it been populated?
I thought that I need to create table DcGisRoads myself. But I have just tested it with a placeholder table in the public schema and it did not yielded this error. Thanks for pointing it out.
Considering other part of your comment: I would like to perform conflation and other operations with hoot using remote database as one of the sources. I can access database from Docker container via psql tool without any problems under hoot user. What do you mean by "read and write successfully from the default installed hootapidb"? You advise to make a default hoot's "database pipeline" and try these commands on it?
UPD: I have created database inside the Docker container using default values in hoot's configs and scripts. By running the same command I get the same error:
Error running convert:
ERROR: syntax error at end of input
LINE 1: EXECUTE Seeing that error message asking if the database had been populated up front indicates to me that the database creation script you ran beforehand wasn't successful in creating the database somehow.
I was thinking to simplify the problem up front to make sure we can verify each step along the way. I only ever use the vagrant instance, and there's also an RPM install....those are the only two configuration we test on a daily basis so they're the only ones I know should definitely work. If you bring up the vagrant instance, you should be able to run your convert command against the vagrant db instance to load your data with no additional up front configuration. Maybe try that first. If you're successful, we can rule out your command or your input data as the source of the problem. Then we'd need to look at the db installation script that's creating the custom database as the source of the problem next, I think.
RamiNaim
commented
4 years ago I have not seen any evidence that created database is not valid. Thank you for your reply, I now will try to use vagrant to launch hoot. As I understood, f I will install vagrant instance of hootenanny, it should not be a problem to use remote database as one of the sources for conflation and other operations?
bwitham
commented
4 years ago If I could see which database query was the source of your error in the original log, that would definitely help in figuring this out. Usually hoot logs each executed sql string before query execution, so I'm surprised the sql is not in the log.
When you say "remote" database you mean having the hoot command line application on one server and the hootapidb database on another server? That should be no problem. I've personally never run the two distributed, but it should work.
RamiNaim
commented
4 years ago Hi.
Following your advice, I have created a vagrant machine with centos inside and installed hoot using guide from here (hoot rpm). Unfortunately, the same exact error persists.
Is there any chance it can be fixed by launching hoot using this instruction (vagrant installation)?
Also, I understood that I do not fully get the hoot's API Hoot DB API. Could you please clarify, what does it mean:
name_of_dataset - what exactly does it mean? Schema, table name? Does it mean the same in the context of different operations (convert, stats, usage of database as a source or target)?
bwitham
commented
4 years ago Yes, sorry I meant for you to use the vagrant installation to test this. Both installs get tested in our CI pipeline, but I'm only familar with the details of the Vagrant one. If you bring up that Vagrant instance and your convert command does not work, then maybe we have a data issue. If the command works from there, then we can start trying to figure out what's different about the database in that instance vs your failing installation.
Hootenanny uses a version of the API database (hootapidb://) internally with a schema that's slightly different that the public OSM API database (osmapidb://) with some optimizations and additional features. The most significant difference is that hootapidb allows multiple data layers (maps) per database, one table for each map. The public OSM API can only have one map (the entire world). The two dbs are not directly compatible and require using the convert command to move data from one to the other.
RamiNaim
commented
4 years ago Hello! I have installed hoot inside vagrant, but encountered the same error. During build process there was no error and nothing seemed suspicious.
But after, inside vagrant, I have run tests, and with no apparent reason, it worked! During test some maps were added to my remote database, and after that I was able to use hootenanny with remote database, even from docker container I have build in the beginning. I haven't done a lot of testing, but simple convert, stats commands seems to work correctly.
This gave me an idea that maybe I indeed forgot something during database creation. Do you have any idea what could it be? Also, during vagrant installation, I haven't changed Database config, and changed it only after build. Maybe it is also somehow affected the situation?
Thank you in advance!
bwitham
commented
4 years ago Is the error text the same as before? ...the generic sql exec error with no other information?
I'm not really sure offhand what's going on. You may be the first person to have tried doing this. Running the working database inside the vagrant instance and calling to it from another location doesn't fit your use case?
Maybe we could try a simpler query against your created hootapidb database... try doing a select against the users table or something like that and see if it works.
RamiNaim
commented
4 years ago @bwitham the problem was indeed in the initialization of my database.
After build complete, I changed credentials in DatabaseConfigDefault, and ran make run-liquibase inside hoot-services. It launched liquibase, and my remote database was initialized correctly.
By the way, it also works inside the docker container on centos07.
Thanks a lot for your help!
bwitham
commented
4 years ago Excellent. I did forget about liquibase being in the mix. Let me know if you run into any other issues.
Describe the bug I am trying to use hootenanny inside docker image. I would like to use the hootapidb to perform conflation between data inside database and osm file. I am trying to load into a postgres DataBase created with this script: (https://github.com/ngageoint/hootenanny/blob/master/scripts/database/SetupHootDb.sh) with changed DB credentials. The database located on my host machine, after it has been created it is accessible and seems as a valid DB. I can access this database from my docker container and from my host machine. Then I create table named DcGISRoads inside the hoot database and try to run this command:
hoot convert -D hootapi.db.writer.create.user=true -D api.db.email='my_email' test-files/DcGisRoads.osm hootapidb://hoot:hoottest@10.1.10.128:5432/hoot/DcGISRoadsUsing trace logging level I receive this:Log
```bash 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 666) append: 0 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 667) remove: 0 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 668) prepend: 0 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 676) optionName: hootapi.db.writer.create.user 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 678) optionVal: true 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 686) values: [1]{true} 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 666) append: 0 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 667) remove: 0 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 668) prepend: 0 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 676) optionName: api.db.email 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 678) optionVal: alnaim.rami@starline.ru 11:27:14.098 TRACE ...pp/hoot/core/util/Settings.cpp( 686) values: [1]{alnaim.rami@starline.ru} 11:27:14.098 TRACE src/main/cpp/hoot/core/Hoot.cpp( 154) Hoot instance reinit... 11:27:14.098 TRACE .../hoot/core/cmd/BaseCommand.cpp( 105) args: [2]{test-files/DcGisRoads.pbf, hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2} 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 62) args.size(): 2 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 63) args: [2]{test-files/DcGisRoads.pbf, hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2} 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 82) arg: test-files/DcGisRoads.pbf 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 82) arg: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 98) inputs.size(): 2 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 99) argIndex: 2 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 102) inputs.size(): 1 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 103) inputs: [1]{test-files/DcGisRoads.pbf} 11:27:14.098 TRACE ...p/hoot/core/cmd/ConvertCmd.cpp( 104) output: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.098 DEBUG ...hoot/core/io/DataConverter.cpp( 210) _convertOps: [0]{} 11:27:14.098 TRACE ...hoot/core/io/DataConverter.cpp( 275) inputs.size(): 1 11:27:14.098 TRACE ...hoot/core/io/DataConverter.cpp( 276) inputs: [1]{test-files/DcGisRoads.pbf} 11:27:14.098 TRACE ...hoot/core/io/DataConverter.cpp( 277) output: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.098 TRACE ...hoot/core/io/DataConverter.cpp( 278) _translation: 11:27:14.098 TRACE ...hoot/core/io/DataConverter.cpp( 279) _shapeFileColumns: [0]{} 11:27:14.098 TRACE ...hoot/core/io/DataConverter.cpp( 280) _ogrFeatureReadLimit: 0 11:27:14.098 TRACE ...hoot/core/io/DataConverter.cpp( 283) IoUtils::isSupportedOsmFormat(inputs.at(0)): 0 11:27:14.098 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 64) input: test-files/DcGisRoads.pbf 11:27:14.098 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 65) allowDir: 1 11:27:14.098 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 72) QFileInfo(input).isDir(): 0 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 91) OgrUtilities::getInstance().getSupportedFormats(false): [.dxf, .gdb, PG:, .gmt, .gpkg, .kmz, .mif, .s57, .tab, .csv, WFS:, gltp:, .sqlite, .db, .shp, GFT:, .bna, .dbf, CouchDB:, .pix, MSSQL:, MySQL:, OCI:, .gpx, .kml, .gtm, ODBC:, .gml, .vrt, .sql, SDE:] 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 92) QFileInfo(input).suffix(): pbf 11:27:14.100 TRACE ...hoot/core/io/DataConverter.cpp( 284) IoUtils::isSupportedOgrFormat(inputs.at(0), true): 0 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 129) input: test-files/DcGisRoads.pbf 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 131) file: test-files/DcGisRoads.pbf 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 64) input: test-files/DcGisRoads.pbf 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 65) allowDir: 1 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 72) QFileInfo(input).isDir(): 0 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 91) OgrUtilities::getInstance().getSupportedFormats(false): [.dxf, .gdb, PG:, .gmt, .gpkg, .kmz, .mif, .s57, .tab, .csv, WFS:, gltp:, .sqlite, .db, .shp, GFT:, .bna, .dbf, CouchDB:, .pix, MSSQL:, MySQL:, OCI:, .gpx, .kml, .gtm, ODBC:, .gml, .vrt, .sql, SDE:] 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 92) QFileInfo(input).suffix(): pbf 11:27:14.100 TRACE ...hoot/core/io/DataConverter.cpp( 286) IoUtils::areSupportedOgrFormats(inputs, true): 0 11:27:14.100 TRACE ...hoot/core/io/DataConverter.cpp( 287) IoUtils::isSupportedOsmFormat(output): 1 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 64) input: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 65) allowDir: 0 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 72) QFileInfo(input).isDir(): 0 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 91) OgrUtilities::getInstance().getSupportedFormats(false): [.dxf, .gdb, PG:, .gmt, .gpkg, .kmz, .mif, .s57, .tab, .csv, WFS:, gltp:, .sqlite, .db, .shp, GFT:, .bna, .dbf, CouchDB:, .pix, MSSQL:, MySQL:, OCI:, .gpx, .kml, .gtm, ODBC:, .gml, .vrt, .sql, SDE:] 11:27:14.100 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 92) QFileInfo(input).suffix(): 11:27:14.100 TRACE ...hoot/core/io/DataConverter.cpp( 288) IoUtils::isSupportedOgrFormat(output): 0 11:27:14.101 DEBUG ...hoot/core/io/DataConverter.cpp( 325) outputInfo.dir().absolutePath(): /var/lib/hootenanny/hootapidb:/hoot:hoottest@localhost:5432/hoot 11:27:14.101 STATUS ...pp/hoot/core/util/Progress.cpp( 85) Convert (0%): Converting test-files/DcGisRoads.pbf to ocalhost:5432/hoot/DcGISRoads2... 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 64) input: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 65) allowDir: 1 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 72) QFileInfo(input).isDir(): 0 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 91) OgrUtilities::getInstance().getSupportedFormats(false): [.dxf, .gdb, PG:, .gmt, .gpkg, .kmz, .mif, .s57, .tab, .csv, WFS:, gltp:, .sqlite, .db, .shp, GFT:, .bna, .dbf, CouchDB:, .pix, MSSQL:, MySQL:, OCI:, .gpx, .kml, .gtm, ODBC:, .gml, .vrt, .sql, SDE:] 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 92) QFileInfo(input).suffix(): 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 129) input: test-files/DcGisRoads.pbf 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 131) file: test-files/DcGisRoads.pbf 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 64) input: test-files/DcGisRoads.pbf 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 65) allowDir: 1 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 72) QFileInfo(input).isDir(): 0 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 91) OgrUtilities::getInstance().getSupportedFormats(false): [.dxf, .gdb, PG:, .gmt, .gpkg, .kmz, .mif, .s57, .tab, .csv, WFS:, gltp:, .sqlite, .db, .shp, GFT:, .bna, .dbf, CouchDB:, .pix, MSSQL:, MySQL:, OCI:, .gpx, .kml, .gtm, ODBC:, .gml, .vrt, .sql, SDE:] 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 92) QFileInfo(input).suffix(): pbf 11:27:14.101 DEBUG ...hoot/core/io/DataConverter.cpp( 796) general convert 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 108) input: test-files/DcGisRoads.pbf 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 110) file: test-files/DcGisRoads.pbf 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 64) input: test-files/DcGisRoads.pbf 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 65) allowDir: 1 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 72) QFileInfo(input).isDir(): 0 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 91) OgrUtilities::getInstance().getSupportedFormats(false): [.dxf, .gdb, PG:, .gmt, .gpkg, .kmz, .mif, .s57, .tab, .csv, WFS:, gltp:, .sqlite, .db, .shp, GFT:, .bna, .dbf, CouchDB:, .pix, MSSQL:, MySQL:, OCI:, .gpx, .kml, .gtm, ODBC:, .gml, .vrt, .sql, SDE:] 11:27:14.101 TRACE ...n/cpp/hoot/core/io/IoUtils.cpp( 92) QFileInfo(input).suffix(): pbf 11:27:14.101 DEBUG ...ot/core/io/ElementStreamer.cpp( 92) ops: [0]{} 11:27:14.101 DEBUG ...ore/io/OsmMapWriterFactory.cpp( 89) url: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.101 TRACE ...cpp/hoot/core/util/Factory.cpp( 79) baseName: hoot::OsmMapWriter 11:27:14.101 DEBUG ...ore/io/OsmMapWriterFactory.cpp( 92) names.size(): 13 11:27:14.101 TRACE ...ore/io/OsmMapWriterFactory.cpp( 97) name: hoot::HootApiDbBulkInserter 11:27:14.101 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.101 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.101 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.101 TRACE ...e/io/HootApiDbBulkInserter.cpp( 72) urlStr: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.101 TRACE ...ore/io/OsmMapWriterFactory.cpp( 97) name: hoot::HootApiDbWriter 11:27:14.101 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 211) Closing writer... 11:27:14.101 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.101 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.101 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.101 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.101 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.101 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.101 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.101 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 211) Closing writer... 11:27:14.101 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_nodes_-1 11:27:14.101 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_ways_-1 11:27:14.101 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_way_nodes_-1 11:27:14.102 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_relations_-1 11:27:14.102 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_relation_members_-1 11:27:14.102 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.102 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.102 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.102 TRACE ...ot/core/io/HootApiDbWriter.cpp( 87) Closing database writer... 11:27:14.102 TRACE ...ot/core/io/HootApiDbWriter.cpp( 105) Finalizing write operation... 11:27:14.102 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.102 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.102 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.102 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.102 DEBUG ...ot/core/io/ElementStreamer.cpp( 58) writerName: hoot::HootApiDbWriter 11:27:14.102 TRACE ...ore/io/OsmMapReaderFactory.cpp( 117) url: test-files/DcGisRoads.pbf 11:27:14.102 TRACE ...ore/io/OsmMapReaderFactory.cpp( 118) useDataSourceIds: 1 11:27:14.102 TRACE ...ore/io/OsmMapReaderFactory.cpp( 119) defaultStatus: Unknown1 11:27:14.102 TRACE ...cpp/hoot/core/util/Factory.cpp( 79) baseName: hoot::OsmMapReader 11:27:14.102 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::GeoNamesReader 11:27:14.102 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::HootApiDbReader 11:27:14.102 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.102 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.102 TRACE ...p/hoot/core/io/ApiDbReader.cpp( 877) Closing database reader... 11:27:14.102 TRACE ...p/hoot/core/io/ApiDbReader.cpp( 841) Finalizing read operation... 11:27:14.102 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.102 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.102 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.102 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.102 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OgrReader 11:27:14.102 DEBUG ...hoot/core/util/IdGenerator.cpp( 42) ConfigOptions().getIdGenerator(): hoot::DefaultIdGenerator 11:27:14.102 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmApiDbReader 11:27:14.102 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.102 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.102 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.102 TRACE ...p/hoot/core/io/ApiDbReader.cpp( 877) Closing database reader... 11:27:14.103 TRACE ...p/hoot/core/io/ApiDbReader.cpp( 841) Finalizing read operation... 11:27:14.103 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.103 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.103 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmApiReader 11:27:14.103 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.103 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.103 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmGeoJsonReader 11:27:14.103 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.103 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmJsonReader 11:27:14.103 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.103 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmPbfReader 11:27:14.103 DEBUG ...ore/io/OsmMapReaderFactory.cpp( 95) Using input reader: hoot::OsmPbfReader 11:27:14.103 DEBUG ...ot/core/io/ElementStreamer.cpp( 59) OsmMapReaderFactory::hasElementInputStream(input): 1 11:27:14.103 TRACE ...ore/io/OsmMapWriterFactory.cpp( 52) url: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.103 TRACE ...ore/io/OsmMapWriterFactory.cpp( 55) writerOverride: 11:27:14.103 TRACE ...cpp/hoot/core/util/Factory.cpp( 79) baseName: hoot::OsmMapWriter 11:27:14.103 TRACE ...ore/io/OsmMapWriterFactory.cpp( 67) names[i]: hoot::HootApiDbBulkInserter 11:27:14.103 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.103 TRACE ...e/io/HootApiDbBulkInserter.cpp( 72) urlStr: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 211) Closing writer... 11:27:14.103 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.103 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.103 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.103 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.103 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.103 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 211) Closing writer... 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_nodes_-1 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_ways_-1 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_way_nodes_-1 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_relations_-1 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_relation_members_-1 11:27:14.103 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.103 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.103 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.103 TRACE ...ore/io/OsmMapWriterFactory.cpp( 67) names[i]: hoot::HootApiDbWriter 11:27:14.103 DEBUG ...ore/io/OsmMapWriterFactory.cpp( 71) Using output writer: hoot::HootApiDbWriter 11:27:14.103 TRACE ...ot/core/io/HootApiDbWriter.cpp( 87) Closing database writer... 11:27:14.103 TRACE ...ot/core/io/HootApiDbWriter.cpp( 105) Finalizing write operation... 11:27:14.103 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.103 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.103 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.103 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.103 DEBUG ...ot/core/io/ElementStreamer.cpp( 60) OsmMapWriterFactory::hasElementOutputStream(output): 1 11:27:14.103 DEBUG ...ot/core/io/ElementStreamer.cpp( 61) ConfigUtils::boundsOptionEnabled(): 0 11:27:14.103 DEBUG ...ot/core/io/ElementStreamer.cpp( 62) ConfigOptions().getWriterXmlSortById(): 1 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 117) url: test-files/DcGisRoads.pbf 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 118) useDataSourceIds: 1 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 119) defaultStatus: Unknown1 11:27:14.104 TRACE ...cpp/hoot/core/util/Factory.cpp( 79) baseName: hoot::OsmMapReader 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::GeoNamesReader 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::HootApiDbReader 11:27:14.104 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.104 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.104 TRACE ...p/hoot/core/io/ApiDbReader.cpp( 877) Closing database reader... 11:27:14.104 TRACE ...p/hoot/core/io/ApiDbReader.cpp( 841) Finalizing read operation... 11:27:14.104 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.104 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.104 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.104 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OgrReader 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmApiDbReader 11:27:14.104 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.104 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.104 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.104 TRACE ...p/hoot/core/io/ApiDbReader.cpp( 877) Closing database reader... 11:27:14.104 TRACE ...p/hoot/core/io/ApiDbReader.cpp( 841) Finalizing read operation... 11:27:14.104 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.104 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmApiReader 11:27:14.104 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.104 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmGeoJsonReader 11:27:14.104 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmJsonReader 11:27:14.104 TRACE ...ot/core/util/GeometryUtils.cpp( 306) boundsStr: 11:27:14.104 TRACE ...ore/io/OsmMapReaderFactory.cpp( 91) Checking input test-files/DcGisRoads.pbf with reader hoot::OsmPbfReader 11:27:14.104 DEBUG ...ore/io/OsmMapReaderFactory.cpp( 95) Using input reader: hoot::OsmPbfReader 11:27:14.104 TRACE ...ore/io/OsmMapWriterFactory.cpp( 52) url: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.104 TRACE ...ore/io/OsmMapWriterFactory.cpp( 55) writerOverride: 11:27:14.104 TRACE ...cpp/hoot/core/util/Factory.cpp( 79) baseName: hoot::OsmMapWriter 11:27:14.104 TRACE ...ore/io/OsmMapWriterFactory.cpp( 67) names[i]: hoot::HootApiDbBulkInserter 11:27:14.104 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.104 TRACE ...e/io/HootApiDbBulkInserter.cpp( 72) urlStr: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 211) Closing writer... 11:27:14.104 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.104 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.104 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.104 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.104 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.104 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 211) Closing writer... 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_nodes_-1 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_ways_-1 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_way_nodes_-1 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_relations_-1 11:27:14.104 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_relation_members_-1 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.105 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.105 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.105 TRACE ...ore/io/OsmMapWriterFactory.cpp( 67) names[i]: hoot::HootApiDbWriter 11:27:14.105 DEBUG ...ore/io/OsmMapWriterFactory.cpp( 71) Using output writer: hoot::HootApiDbWriter 11:27:14.105 TRACE ...ot/core/io/HootApiDbWriter.cpp( 87) Closing database writer... 11:27:14.105 TRACE ...ot/core/io/HootApiDbWriter.cpp( 105) Finalizing write operation... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.105 TRACE ...ore/io/OsmMapWriterFactory.cpp( 52) url: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.105 TRACE ...ore/io/OsmMapWriterFactory.cpp( 55) writerOverride: 11:27:14.105 TRACE ...cpp/hoot/core/util/Factory.cpp( 79) baseName: hoot::OsmMapWriter 11:27:14.105 TRACE ...ore/io/OsmMapWriterFactory.cpp( 67) names[i]: hoot::HootApiDbBulkInserter 11:27:14.105 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.105 TRACE ...e/io/HootApiDbBulkInserter.cpp( 72) urlStr: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 211) Closing writer... 11:27:14.105 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.105 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 211) Closing writer... 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_nodes_-1 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_ways_-1 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_way_nodes_-1 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_relations_-1 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp( 231) No data for table current_relation_members_-1 11:27:14.105 TRACE ...re/io/OsmApiDbBulkInserter.cpp(1016) Resetting variables... 11:27:14.105 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 86) Closing database connection... 11:27:14.105 TRACE .../cpp/hoot/core/io/OsmApiDb.cpp( 175) Resetting queries... 11:27:14.105 TRACE ...ore/io/OsmMapWriterFactory.cpp( 67) names[i]: hoot::HootApiDbWriter 11:27:14.105 DEBUG ...ore/io/OsmMapWriterFactory.cpp( 71) Using output writer: hoot::HootApiDbWriter 11:27:14.105 DEBUG ...ot/core/io/HootApiDbWriter.cpp( 126) Opening database writer for: hootapidb://hoot:hoottest@localhost:5432/hoot/DcGISRoads2... 11:27:14.105 DEBUG ...ot/core/io/HootApiDbWriter.cpp( 186) mapName: DcGISRoads2 11:27:14.105 DEBUG ...cpp/hoot/core/io/HootApiDb.cpp(1118) Opening database connection: hootapidb://localhost:5432/hoot/DcGISRoads2... 11:27:14.105 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... 11:27:14.115 DEBUG ...ain/cpp/hoot/core/io/ApiDb.cpp( 200) Successfully opened db: hootapidb://localhost:5432/hoot/DcGISRoads2 11:27:14.115 DEBUG ...ain/cpp/hoot/core/io/ApiDb.cpp( 201) Postgres database version: PostgreSQL 9.5.22 on x86_64-pc-linux-gnu (Ubuntu 9.5.22-1.pgdg16.04+1), compiled by gcc (Ubuntu 5.4.0-6ubuntu1~16.04.12) 5.4.0 20160609, 64-bit 11:27:14.116 TRACE ...ot/core/io/HootApiDbWriter.cpp( 87) Closing database writer... 11:27:14.116 TRACE ...ot/core/io/HootApiDbWriter.cpp( 105) Finalizing write operation... 11:27:14.116 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 168) Closing database connection... 11:27:14.116 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 474) Flushing bulk inserts... 11:27:14.116 TRACE ...cpp/hoot/core/io/HootApiDb.cpp( 500) Flushing bulk deletes... 11:27:14.116 TRACE ...cpp/hoot/core/io/HootApiDb.cpp(1141) Resetting queries... Error running convert: ERROR: syntax error at end of input LINE 1: EXECUTE ^ (42601) QPSQL: Unable to create query ```As I can tell, hoot successfully connects to the database. The same error rises when I try to get stats from a remote db. It is worth mentioning that Hootenanny works fine when I perform conflation on osm files without database. The same error accures when I try run other command which include hootapidb and different pbf and osm files.
hoot versionoutput:Hootenanny 0.2.54 Built By: rpmbuildTo Reproduce Steps to reproduce the behavior:
--network host. With this it will be able to access the DB on the host machine.Expected behavior A table on the host machine with imported osm file
Host machine:
Additional context My goal is to make hootenanny work with already created DB with already preloaded pbf. I would like to use this DB as the imput for conflate command, but cannot perform some basic commands with hootapidb. Thank you in advance!