search

ngds / install-and-run

Use this repository's issue tracker to post comments, bug reports, and help questions on installing and running NGDS CKAN.
4 stars 3 forks source link

pycsw install seems to be missing #22

Closed smrgeoinfo closed 9 years ago

smrgeoinfo commented 9 years ago

http://ckanext-spatial.readthedocs.org/en/latest/csw.htmlhttp://ckanext-spatial.readthedocs.org/en/latest/csw.html

Or is the pycsw install buried somewhere?

the v1 installer from last May (https://github.com/ngds/install-and-run/archive/v1.zip) has shell script procedures for installing pycsw; not sure how this would translate to rpm

FuhuXia commented 9 years ago

One parameter in production.ini is related the external url. Its value is unique to each and every instance, in order to make the datapusher and geoserver preview works. It has to be set manually. Hopefully the user has to do it only once... until next time we change some config in production.ini, which does not happen often unless we are introducing some new extensions/plugins.

lukejbuckley commented 9 years ago

To be clear…

I need to

  1.  replace production.ini with production.ini.rpmnew
  2.  then edit production.ini to fix the ckan.hostname
  3.  fix proxyBaseUrl in global.xml

Luke Buckley | Computer Software Engineer MBMG | Montana Tech 1300 W Park St, NRB 329 A | Butte, MT 59701

From: Fuhu Xia [mailto:notifications@github.com] Sent: Tuesday, March 10, 2015 1:13 PM To: ngds/install-and-run Cc: Buckley, Luke Subject: Re: [install-and-run] pycsw install seems to be missing (#22)

One parameter in production.ini is related the external url. Its value is unique to each and every instance, in order to make the datapusher and geoserver preview works. It has to be set manually. Hopefully the user has to do it only once... until next time we change some config in production.ini, which does not happen often unless we are introducing some new extensions/plugins.

— Reply to this email directly or view it on GitHubhttps://github.com/ngds/install-and-run/issues/22#issuecomment-78126033.

FuhuXia commented 9 years ago

@lukejbuckley That is correct. Any time if there is a production.ini.rpmnew file shows up after rpm update, you need to re-do the (1) and (2). Step(3) need to be done once. And if you are coming from rpm version 300 or before, you need to do the manual database changes as listed.

lukejbuckley commented 9 years ago

@fuhuxia @ccaudill

Following is the contents of my global.xml (/var/lib/tomcat6/webapps/geoserver/data) file. There is no proxyBaseUrl element with a value of 127.0.0.1 to replace. Do I need to add it somewhere? Or ignore the step?

Alexandria Egypt Work claudius.ptolomaeus@gmail.com The ancient geographes INC Claudius Ptolomaeus Chief geographer false false 5 7 0.5 0.75 false true true false 10 5 30000 UNBOUNDED 10240 UTF-8 8 http://geoserver.org false false 78 0 true 1024 From: Fuhu Xia [mailto:notifications@github.com] Sent: Tuesday, March 10, 2015 6:49 PM To: ngds/install-and-run Cc: Buckley, Luke Subject: Re: [install-and-run] pycsw install seems to be missing (#22) @lukejbuckleyhttps://github.com/lukejbuckley That is correct. Any time if there is a production.ini.rpmnew file shows up after rpm update, you need to re-do the (1) and (2). Step(3) need to be done once. And if you are coming from rpm version 300 or before, you need to do the manual database changes as listed. — Reply to this email directly or view it on GitHubhttps://github.com/ngds/install-and-run/issues/22#issuecomment-78180051.
FuhuXia commented 9 years ago

Thanks for the catch. we have updated the global.xml file after rpm 300 but i missed the step in instrucitons for 200 and previous. 

      service tomcat6 stop
      chown tomcat:tomcat /var/tmp/geoserver.global.xml
      cp -f /var/tmp/geoserver.global.xml /var/lib/tomcat6/webapps/geoserver/data/global.xml

make changes to proxyBaseUrl per instruction, then

      service tomcat6 start

I will update the Readme file later.

lukejbuckley commented 9 years ago

@fuhuxiz @ccaudill

That part worked.

I still get the following when I try to start the db statements

[root@MBMGGIN tmp]# sudo -u postgres createdb -O ckan_default pycsw -E utf-8 createdb: database creation failed: ERROR: database "pycsw" already exists

From: Fuhu Xia [mailto:notifications@github.com] Sent: Wednesday, March 11, 2015 9:54 AM To: ngds/install-and-run Cc: Buckley, Luke Subject: Re: [install-and-run] pycsw install seems to be missing (#22)

Thanks for the catch. we have updated the global.xml file after rpm 300 but i missed the step in instrucitons for 200 and previous.

  service tomcat6 stop

  chown tomcat:tomcat /var/tmp/geoserver.global.xml

  cp -f /var/tmp/geoserver.global.xml /var/lib/tomcat6/webapps/geoserver/data/global.xml

make changes to proxyBaseUrl per instruction, then

  service tomcat6 start

I will update the Readme file later.

— Reply to this email directly or view it on GitHubhttps://github.com/ngds/install-and-run/issues/22#issuecomment-78291796.

FuhuXia commented 9 years ago

@lukejbuckley It is ok, not sure which version you are updating from, but the pycsw table has be created by rpm or some manual process. Anyway, just move on the finish all the commands. It does not hurt anything to run those commands on existing database.

lukejbuckley commented 9 years ago

@ccaudill

All the steps are finished on my end. Test away.

From: Christy Caudill [mailto:notifications@github.com] Sent: Tuesday, March 10, 2015 1:00 PM To: ngds/install-and-run Cc: Buckley, Luke Subject: Re: [install-and-run] pycsw install seems to be missing (#22)

@lukejbuckleyhttps://github.com/lukejbuckley https://github.com/ngds/install-and-run#updating-ngds Looks like pretty great instructions (if a lot of manual work). If you have the time, it would be great if you could do this update so we can continue testing. Thanks for your support!

— Reply to this email directly or view it on GitHubhttps://github.com/ngds/install-and-run/issues/22#issuecomment-78123439.

ccaudill commented 9 years ago

Thanks @lukejbuckley @FuhuXia I'm testing a harvest from the MT node this morning.

ccaudill commented 9 years ago

@FuhuXia Latest attempt to harvest.... For the last half hour or so, the message in the UI http://159.87.39.5/harvest/admin/montana-geological-survey-ngds-node has said "Running" with nothing being harvested in. Here is the harvester log:

[root@localhost ~]# tail -n 50 /var/log/harvester_run.log 2015-03-12 07:30:20,129 DEBUG [ckanext.harvest.model] Harvest tables defined in memory 2015-03-12 07:30:20,133 DEBUG [ckanext.harvest.model] Harvest tables already exist 2015-03-12 07:30:20,222 INFO [ckanext.harvest.logic.action.update] Harvest job run: {} 2015-03-12 07:30:20,297 INFO [ckanext.harvest.logic.action.update] No new harvest jobs.

Traceback (most recent call last): File "/usr/bin/ckan", line 59, in load_entry_point('PasteScript', 'console_scripts', 'paster')() File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 104, in run invoke(command, command_name, options, args[1:]) File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 143, in invoke exit_code = runner.run(args) File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 238, in run result = self.command() File "/usr/lib/ckan/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 122, in command self.run_harvester() File "/usr/lib/ckan/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 287, in run_harvester jobs = get_action('harvest_jobs_run')(context,{}) File "/usr/lib/ckan/src/ckan/ckan/logic/init.py", line 419, in wrapped result = _action(context, data_dict, **kw) File "/usr/lib/ckan/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 338, in harvest_jobs_run raise Exception('There are no new harvesting jobs') Exception: There are no new harvesting jobs 2015-03-12 07:45:15,698 INFO [ckanext.facets.helpers] Metadata plugin custom facets loaded 2015-03-12 07:45:15,703 DEBUG [ckanext.ngds.sysadmin.model.db] Sysadmin configuration table already exists 2015-03-12 07:45:19,709 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory 2015-03-12 07:45:19,730 DEBUG [ckanext.spatial.model.package_extent] Spatial tables already exist 2015-03-12 07:45:19,746 DEBUG [ckanext.harvest.model] Harvest tables defined in memory 2015-03-12 07:45:19,749 DEBUG [ckanext.harvest.model] Harvest tables already exist 2015-03-12 07:45:19,835 INFO [ckanext.harvest.logic.action.update] Harvest job run: {} 2015-03-12 07:45:19,906 INFO [ckanext.harvest.logic.action.update] No new harvest jobs.

Traceback (most recent call last): File "/usr/bin/ckan", line 59, in load_entry_point('PasteScript', 'console_scripts', 'paster')() File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 104, in run invoke(command, command_name, options, args[1:]) File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 143, in invoke exit_code = runner.run(args) File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 238, in run result = self.command() File "/usr/lib/ckan/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 122, in command self.run_harvester() File "/usr/lib/ckan/src/ckanext-harvest/ckanext/harvest/commands/harvester.py", line 287, in run_harvester jobs = get_action('harvest_jobs_run')(context,{}) File "/usr/lib/ckan/src/ckan/ckan/logic/init.py", line 419, in wrapped result = _action(context, data_dict, **kw) File "/usr/lib/ckan/src/ckanext-harvest/ckanext/harvest/logic/action/update.py", line 338, in harvest_jobs_run raise Exception('There are no new harvesting jobs') Exception: There are no new harvesting jobs

FuhuXia commented 9 years ago

the log looks fine. please post the other two logs to see if there is any error. tail -n 20 /var/log/gather-consumer.log tail -n 20 /var/log/fetch-consumer.log

I also tried on my local. It shows http://mbmggin.mtech.edu/csw contains no record.

ccaudill commented 9 years ago

[root@localhost ~]# tail -n 20 /var/log/gather-consumer.log params) File "/usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1584, in _execute_clauseelement compiled_sql, distilled_params File "/usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1698, in _execute_context context) File "/usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/base.py", line 1691, in _execute_context context) File "/usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/default.py", line 331, in do_execute cursor.execute(statement, parameters) sqlalchemy.exc.OperationalError: (OperationalError) server closed the connection unexpectedly This probably means the server terminated abnormally before or while processing the request. 'SELECT harvest_job.id AS harvest_job_id, harvest_job.created AS harvest_job_created, harvest_job.gather_started AS harvest_job_gather_started, harvest_job.gather_finished AS harvest_job_gather_finished, harvest_job.finished AS harvest_job_finished, harvest_job.source_id AS harvest_job_source_id, harvest_job.status AS harvest_job_status \nFROM harvest_job \nWHERE harvest_job.id = %(id_1)s \n LIMIT %(param_1)s' {'id_1': u'fa6603ef-61ab-43f3-89e2-5eb49e5e1bcf', 'param_1': 1} 2015-03-12 07:15:30,728 INFO [ckanext.facets.helpers] Metadata plugin custom facets loaded 2015-03-12 07:15:30,737 DEBUG [ckanext.ngds.sysadmin.model.db] Sysadmin configuration table already exists 2015-03-12 07:15:33,430 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory 2015-03-12 07:15:33,463 DEBUG [ckanext.spatial.model.package_extent] Spatial tables already exist 2015-03-12 07:15:33,483 DEBUG [ckanext.harvest.model] Harvest tables defined in memory 2015-03-12 07:15:33,489 DEBUG [ckanext.harvest.model] Harvest tables already exist 2015-03-12 07:15:33,605 DEBUG [ckanext.harvest.queue] Gather queue consumer registered

ccaudill commented 9 years ago

[root@localhost ~]# tail -n 20 /var/log/fetch-consumer.log 2015-02-18 15:21:28,049 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory 2015-02-18 15:21:28,074 DEBUG [ckanext.spatial.model.package_extent] Spatial tables already exist 2015-02-18 15:21:28,092 DEBUG [ckanext.harvest.model] Harvest tables defined in memory 2015-02-18 15:21:28,095 DEBUG [ckanext.harvest.model] Harvest tables already exist 2015-02-18 15:21:28,206 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered 2015-02-18 15:21:28,897 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory 2015-02-18 15:21:28,929 DEBUG [ckanext.spatial.model.package_extent] Spatial tables already exist 2015-02-18 15:21:28,946 DEBUG [ckanext.harvest.model] Harvest tables defined in memory 2015-02-18 15:21:28,950 DEBUG [ckanext.harvest.model] Harvest tables already exist 2015-02-18 15:21:29,052 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered 2015-02-18 15:21:29,834 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory 2015-02-18 15:21:29,857 DEBUG [ckanext.spatial.model.package_extent] Spatial tables already exist 2015-02-18 15:21:29,874 DEBUG [ckanext.harvest.model] Harvest tables defined in memory 2015-02-18 15:21:29,878 DEBUG [ckanext.harvest.model] Harvest tables already exist 2015-02-18 15:21:29,982 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered 2015-02-18 15:21:33,898 DEBUG [ckanext.spatial.model.package_extent] Spatial tables defined in memory 2015-02-18 15:21:33,924 DEBUG [ckanext.spatial.model.package_extent] Spatial tables already exist 2015-02-18 15:21:33,943 DEBUG [ckanext.harvest.model] Harvest tables defined in memory 2015-02-18 15:21:33,948 DEBUG [ckanext.harvest.model] Harvest tables already exist 2015-02-18 15:21:34,058 DEBUG [ckanext.harvest.queue] Fetch queue consumer registered

FuhuXia commented 9 years ago

If the harvest job status still stays at running, you can clear and re-harvest the job. It does not hurt to run a command supervisorctl restart all before doing that.

ccaudill commented 9 years ago

So the error message after it ran was "No records received from the CSW server". What do you think is going on here? There's clearly records (44) in this CKAN instance: http://mbmggin.mtech.edu/dataset

But it is strange that http://mbmggin.mtech.edu/csw?request=GetRecords&service=CSW&version=2.0.2 Doesn't give you the response you'd expect, which is this: http://catalog.stategeothermaldata.org/geoportal/csw?request=GetRecords&service=CSW&version=2.0.2 telling you the number of total records from the CSW endpoint.

Does this point to a problem with the pyCSW?

ccaudill commented 9 years ago

@FuhuXia pyCSW will harvest in everything, correct? Resources which were published on the node as well as those which were harvested into that node? It seems that harvests are working, but we can't seem to get a CKAN-CKAN harvest working. MT node only has harvested resources, and no published resources.

ccaudill commented 9 years ago

@FuhuXia Could you tell us what the DB, username, and password would be for Postgres in the install in the rpm? We'd like to take a look and see if the data is getting populated correctly there, and if there is something lost between Postgres and CKAN's data store.

FuhuXia commented 9 years ago

@ccaudill You can see there are two commands in the pycsw cron jobs. https://github.com/ngds/install-and-run/blob/master/rpm_install/etc/cron.d/ckan-pycsw#L2 the first ckan-pycsw load command will load any harvested data. The second datastore-pycsw load command will load local data that fit in this criteria. https://github.com/ngds/ckanext-datastorecsw/blob/master/bin/datastore_pycsw.py#L74 If a published data does not appear in the query result, it wont get into csw records.

All DB info is in /etc/ckan/production.ini file https://github.com/ngds/install-and-run/blob/master/rpm_install/etc/ckan/production.ini#L48

FuhuXia commented 9 years ago

@ccaudill For the the MT server published dataset no going to csw issue, can you run the command and provide the output? /usr/lib/ckan/bin/paster --plugin=ckanext-spatial ckan-pycsw load -p /etc/ckan/pycsw.cfg && /usr/lib/ckan/bin/paster --plugin=ckanext-datastorecsw datastore-pycsw load -p /etc/ckan/pycsw.cfg

FuhuXia commented 9 years ago

@smrazgs Let us split the command into two commands and run them one after the other. Please post results, you can trim off repetitive ones, if any.

/usr/lib/ckan/bin/paster --plugin=ckanext-spatial ckan-pycsw load -p /etc/ckan/pycsw.cfg

/usr/lib/ckan/bin/paster --plugin=ckanext-datastorecsw datastore-pycsw load -p /etc/ckan/pycsw.cfg

smrgeoinfo commented 9 years ago

I ran the commands on 159.87.39.5 First one seemed to work OK, lots of inserts with GUIDS, and fails on duplicate GUIDS

Second one gagged, here's the console output:

[root@localhost ~]# /usr/lib/ckan/bin/paster --plugin=ckanext-datastorecsw datastore-pycsw load -p /etc/ckan/pycsw.cfg

/usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/reflection.py:47: SAWarning: Did not recognize type 'tsvector' of column 'anytext_tsvector' ret = fn(self, con, _args, *_kw)

/usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/reflection.py:47: SAWarning: Did not recognize type 'geometry' of column 'wkb_geometry' ret = fn(self, con, _args, *_kw)

Started gathering CKAN datasets identifiers: 2015-03-17 12:58:57.517949

Starting new HTTP connection (1): localhost

Starting new HTTP connection (1): localhost

Traceback (most recent call last): File "/usr/lib/ckan/bin/paster", line 9, in 

load_entry_point('PasteScript==1.7.5', 'console_scripts', 'paster')()

File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 104, in run invoke(command, command_name, options, args[1:])

File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 143, in invoke exit_code = runner.run(args)

File "/usr/lib/ckan/lib/python2.6/site-packages/paste/script/command.py", line 238, in run result = self.command()

File "/usr/lib/ckan/src/ckanext-datastorecsw/ckanext/csw/commands/csw.py", line 56, in command datastore_pycsw.load(config, ckan_url)

File "/usr/lib/ckan/src/ckanext-datastorecsw/bin/datastore_pycsw.py", line 158, in load package_id = parse_resource(result, ckan_url)

File "/usr/lib/ckan/src/ckanext-datastorecsw/bin/datastore_pycsw.py", line 114, in parse_resource return package_id[0]

IndexError: list index out of range

[root@localhost ~]#

smrgeoinfo commented 9 years ago

on 159.87.39.4:

login as: root root@159.87.39.4's password: Last login: Thu Feb 26 14:22:35 2015 from 10.208.1.175 [root@localhost ~]# /usr/lib/ckan/bin/paster --plugin=ckanext-spatial ckan-pycsw load -p /etc/ckan/pycsw.cfg /usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/reflection.py:47: SAWarning: Did not recognize type 'tsvector' of column 'anytext_tsvector' ret = fn(self, con, _args, _kw) /usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/reflection.py:47: SAWarning: Did not recognize type 'geometry' of column 'wkb_geometry' ret = fn(self, con, _args, _kw) Started gathering CKAN datasets identifiers: 2015-03-17 13:05:38.919103 Starting new HTTP connection (1): localhost Starting new HTTP connection (1): localhost Gather finished (38 datasets): 2015-03-17 13:05:39.031257 Starting new HTTP connection (1): localhost Inserted 15e331cc-d6fa-4986-b339-4beec0bb996b Starting new HTTP connection (1): localhost Inserted d2141d76-a7b3-4e88-8127-3e8c2509906f Starting new HTTP connection (1): localhost Inserted 7286ad2a-e9df-4f1f-9cb4-b3c55c3c2a67 Starting new HTTP connection (1): localhost

...lots of the same kind, different guids...

Starting new HTTP connection (1): localhost Inserted d66f2c4e-8746-47a8-bb34-4f07e1e52eaa Starting new HTTP connection (1): localhost Inserted 178c6d5b-568a-4988-8bf1-da52caae24ff [root@localhost ~]# [root@localhost ~]# [root@localhost ~]# /usr/lib/ckan/bin/paster --plugin=ckanext-datastorecsw datastore-pycsw load -p /etc/ckan/pycsw.cfg /usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/reflection.py:47: SAWarning: Did not recognize type 'tsvector' of column 'anytext_tsvector' ret = fn(self, con, _args, _kw) /usr/lib/ckan/lib/python2.6/site-packages/sqlalchemy/engine/reflection.py:47: SAWarning: Did not recognize type 'geometry' of column 'wkb_geometry' ret = fn(self, con, _args, _kw) Started gathering CKAN datasets identifiers: 2015-03-17 13:06:00.479655 Starting new HTTP connection (1): localhost ..5 more the same... Starting new HTTP connection (1): localhost Starting new HTTP connection (1): localhost Gather finished (3 datasets): 2015-03-17 13:06:00.896071 Starting new HTTP connection (1): localhost Inserted ef0f103f-4b3a-4421-8804-98d34e61c93e Starting new HTTP connection (1): localhost Inserted 9e7d925b-0a6b-49bd-b06b-380da3c75449 Starting new HTTP connection (1): localhost Inserted 52378851-ea66-40d9-9388-9aa912e308ce [root@localhost ~]#

smrgeoinfo commented 9 years ago

this wfs request now works: http://159.87.39.4/csw?request=GetRecords&version=2.0.2&service=CSW&typeNames=csw:Record&resultType=results&outputFormat=application/xml&ElementSetName=brief

41 records found

also works on 159.87.39.5 with 9096 records found...

ccaudill commented 9 years ago

It looks like Postgres is not listening...

[root@localhost ~]# netstat -antup | grep 5432 tcp 0 0 0.0.0.0:5432 0.0.0.0:* LISTEN 12579/postmaster tcp 111 0 127.0.0.1:56744 127.0.0.1:5432 CLOSE_WAIT 2761/demo tcp 111 0 127.0.0.1:52643 127.0.0.1:5432 CLOSE_WAIT 1412/python tcp 111 0 127.0.0.1:52581 127.0.0.1:5432 CLOSE_WAIT 1404/python tcp 111 0 127.0.0.1:52618 127.0.0.1:5432 CLOSE_WAIT 1406/python tcp 111 0 127.0.0.1:52594 127.0.0.1:5432 CLOSE_WAIT 1405/python tcp 111 0 127.0.0.1:52628 127.0.0.1:5432 CLOSE_WAIT 1408/python tcp 111 0 127.0.0.1:52579 127.0.0.1:5432 CLOSE_WAIT 1404/python tcp 111 0 127.0.0.1:52654 127.0.0.1:5432 CLOSE_WAIT 1409/python tcp 111 0 127.0.0.1:52629 127.0.0.1:5432 CLOSE_WAIT 1408/python tcp 111 0 127.0.0.1:52913 127.0.0.1:5432 CLOSE_WAIT 2761/demo tcp 111 0 127.0.0.1:52537 127.0.0.1:5432 CLOSE_WAIT 1411/python tcp 111 0 127.0.0.1:52616 127.0.0.1:5432 CLOSE_WAIT 1406/python tcp 111 0 127.0.0.1:52663 127.0.0.1:5432 CLOSE_WAIT 1407/python tcp 111 0 127.0.0.1:52542 127.0.0.1:5432 CLOSE_WAIT 1404/python tcp 111 0 127.0.0.1:52660 127.0.0.1:5432 CLOSE_WAIT 1407/python tcp 111 0 127.0.0.1:52541 127.0.0.1:5432 CLOSE_WAIT 1412/python tcp 111 0 127.0.0.1:52610 127.0.0.1:5432 CLOSE_WAIT 1410/python tcp 111 0 127.0.0.1:52608 127.0.0.1:5432 CLOSE_WAIT 1410/python tcp 111 0 127.0.0.1:52578 127.0.0.1:5432 CLOSE_WAIT 1404/python tcp 111 0 127.0.0.1:52611 127.0.0.1:5432 CLOSE_WAIT 1410/python tcp 111 0 127.0.0.1:52630 127.0.0.1:5432 CLOSE_WAIT 1408/python tcp 111 0 127.0.0.1:52631 127.0.0.1:5432 CLOSE_WAIT 1408/python tcp 111 0 127.0.0.1:52857 127.0.0.1:5432 CLOSE_WAIT 2760/demo tcp 111 0 127.0.0.1:52539 127.0.0.1:5432 CLOSE_WAIT 1405/python tcp 111 0 127.0.0.1:52595 127.0.0.1:5432 CLOSE_WAIT 1405/python tcp 111 0 127.0.0.1:52912 127.0.0.1:5432 CLOSE_WAIT 2761/demo tcp 0 0 127.0.0.1:5432 127.0.0.1:56759 ESTABLISHED 12688/postgres tcp 111 0 127.0.0.1:52587 127.0.0.1:5432 CLOSE_WAIT 1411/python tcp 111 0 127.0.0.1:52540 127.0.0.1:5432 CLOSE_WAIT 1409/python tcp 111 0 127.0.0.1:53382 127.0.0.1:5432 CLOSE_WAIT 2760/demo tcp 111 0 127.0.0.1:52662 127.0.0.1:5432 CLOSE_WAIT 1407/python tcp 111 0 127.0.0.1:52859 127.0.0.1:5432 CLOSE_WAIT 2760/demo tcp 111 0 127.0.0.1:52858 127.0.0.1:5432 CLOSE_WAIT 2760/demo tcp 111 0 127.0.0.1:52545 127.0.0.1:5432 CLOSE_WAIT 1407/python tcp 111 0 127.0.0.1:52580 127.0.0.1:5432 CLOSE_WAIT 1404/python tcp 111 0 127.0.0.1:52653 127.0.0.1:5432 CLOSE_WAIT 1409/python tcp 111 0 127.0.0.1:52620 127.0.0.1:5432 CLOSE_WAIT 1406/python tcp 0 0 127.0.0.1:56759 127.0.0.1:5432 ESTABLISHED 2760/demo tcp 111 0 127.0.0.1:52640 127.0.0.1:5432 CLOSE_WAIT 1412/python tcp 111 0 127.0.0.1:52538 127.0.0.1:5432 CLOSE_WAIT 1410/python tcp 111 0 127.0.0.1:52596 127.0.0.1:5432 CLOSE_WAIT 1405/python tcp 111 0 127.0.0.1:52544 127.0.0.1:5432 CLOSE_WAIT 1408/python tcp 111 0 127.0.0.1:52911 127.0.0.1:5432 CLOSE_WAIT 2761/demo tcp 111 0 127.0.0.1:52609 127.0.0.1:5432 CLOSE_WAIT 1410/python tcp 111 0 127.0.0.1:52589 127.0.0.1:5432 CLOSE_WAIT 1411/python tcp 111 0 127.0.0.1:52652 127.0.0.1:5432 CLOSE_WAIT 1409/python tcp 111 0 127.0.0.1:52617 127.0.0.1:5432 CLOSE_WAIT 1406/python tcp 111 0 127.0.0.1:52641 127.0.0.1:5432 CLOSE_WAIT 1412/python tcp 111 0 127.0.0.1:52543 127.0.0.1:5432 CLOSE_WAIT 1406/python tcp 111 0 127.0.0.1:52586 127.0.0.1:5432 CLOSE_WAIT 1411/python tcp 111 0 127.0.0.1:52655 127.0.0.1:5432 CLOSE_WAIT 1409/python tcp 111 0 127.0.0.1:53381 127.0.0.1:5432 CLOSE_WAIT 2760/demo tcp 111 0 127.0.0.1:52661 127.0.0.1:5432 CLOSE_WAIT 1407/python tcp 111 0 127.0.0.1:52642 127.0.0.1:5432 CLOSE_WAIT 1412/python tcp 111 0 127.0.0.1:52597 127.0.0.1:5432 CLOSE_WAIT 1405/python tcp 111 0 127.0.0.1:52588 127.0.0.1:5432 CLOSE_WAIT 1411/python tcp 0 0 :::5432 :::* LISTEN 12579/postmaster

ccaudill commented 9 years ago

FYI - It seems that there are some firewall issues inherent with CentOS that we didn't have with Ubuntu. I had to run the following command for Postgres to allow a connection and to not be rejected to traffic: iptables -I INPUT 1 -p tcp --dport 5432 -j ACCEPT (ran it in 10.208.3.122/159.87.39.4)

ccaudill commented 9 years ago

You actually have to run these commands so that it sticks:

iptables -I INPUT 1 -p tcp --dport 5432 -j ACCEPT service iptables save service iptables restart

This should be in the rpm??

-- You also have to edit the pg_hba.conf file and the postgresql.conf file....

ccaudill commented 9 years ago

@FuhuXia @smrazgs @dan-olaru-reisys It FINALLY harvested in!!! http://10.208.3.104/harvest/azgs-ginstack-test-harvest/job/14da6a65-257a-428b-ac64-feae305125c1 The fact that the CentOS firewall is set to automatically stop traffic to/from Postgres in a GINstack build seems to have been the problem.

@lukejbuckley - Might you have time to try and run these commands, then we'll try a harvest again from your node?

iptables -I INPUT 1 -p tcp --dport 5432 -j ACCEPT service iptables save service iptables restart

Then, you have to get into the /var/lib/pgsql/9.1/data/postgresql.conf and make it as such:

- Connection Settings -

listen_addresses = '[asterisk]' # what IP address(es) to listen on;

comma-separated list of addresses;

                # defaults to 'localhost', '*' = all
                # (change requires restart)

port = 5432 # (change requires restart)

We also had to edit the /var/lib/pgsql/9.1/data/pg_hba.conf but hopefully you won't have to do that too.

lukejbuckley commented 9 years ago

@ccaudill I ran the commands and restarted the machine.

From: Christy Caudill [mailto:notifications@github.com] Sent: Friday, March 20, 2015 9:15 AM To: ngds/install-and-run Cc: Buckley, Luke Subject: Re: [install-and-run] pycsw install seems to be missing (#22)

@FuhuXiahttps://github.com/FuhuXia @smrazgshttps://github.com/smrazgs @dan-olaru-reisyshttps://github.com/dan-olaru-reisys It FINALLY harvested in!!! http://10.208.3.104/harvest/azgs-ginstack-test-harvest/job/14da6a65-257a-428b-ac64-feae305125c1 The fact that the CentOS firewall is set to automatically stop traffic to/from Postgres in a GINstack build seems to have been the problem.

@lukejbuckleyhttps://github.com/lukejbuckley - Might you have time to try and run these commands, then we'll try a harvest again from your node?

iptables -I INPUT 1 -p tcp --dport 5432 -j ACCEPT service iptables save service iptables restart

Then, you have to get into the /var/lib/pgsql/9.1/data/postgresql.conf and make it as such:

listen_addresses = '' # what IP address(es) to listen on;

comma-separated list of addresses;

defaults to 'localhost', '' = all

(change requires restart)

port = 5432 # (change requires restart)

We also had to edit the /var/lib/pgsql/9.1/data/pg_hba.conf but hopefully you won't have to do that too.

— Reply to this email directly or view it on GitHubhttps://github.com/ngds/install-and-run/issues/22#issuecomment-84041495.

ccaudill commented 9 years ago

@FuhuXia Okay, since Luke has run those commands, and from http://mbmggin.mtech.edu/csw?request=GetCapabilities&Service=CSW&Version=2.0.2 I'm still getting the error message, "Connection refused Is the server running on host "127.0.0.1" and accepting TCP/IP connections on port 5432?" It is clear that in the rpm for a GINstack, the postgresql.conf and the pg_hba.conf have to be edited to allow traffic. That seems like it's the answer to solving this and should be in a new build of the rpm. Can you please confirm?

FuhuXia commented 9 years ago

@ccaudill For the port 5432 firewall rule change, do you have to make the change to all ngds servers, or just one particular server? If I set up some fresh instances on amazon with the rpm installed, can you replicate the issue on the fresh instances?

The reason I want to check it this way is that we definitely don't want to enable this firewall rule on any public ngds server. We are using a default postgres db user credential. If the port is open, anyone is going to have full access to ngds database. In the rpm script, we only have port 80 open, as coded in here https://github.com/ngds/install-and-run/blob/master/rpm_install/tmp/after_rpm.sh#L203.

ccaudill commented 9 years ago

Thanks @FuhuXia The port has to be open to 5432 for the harvest to work, and that is not how it comes in the install. Sure, if you set up some fresh ones, I can recreate this. It is simply about editing those two files to allow traffic on 5432 and to allow anyone to access the data. This is what we want - it is the idea that public access is allowable to these servers - anyone should be able to harvest from them without permission and without editing any files.

FuhuXia commented 9 years ago

Harvesting is done via web api at the port 80. The local communication between ckan and postgresql does not need firewall rule change on port 5432. Ok, I will launch an fresh ngds server and we can work on that one to solve the mystery PostgreSQL port issue.

ccaudill commented 9 years ago

We did this test systematically, and all of those things had to happen before a harvest was successful. It will be interesting with a fresh server - let me know when I can help.

FuhuXia commented 9 years ago

@ccaudill Ok, this instance was started with centos 6.4 with rpm 303 a few weeks ago. I just updated it to rpm 307. It can harvest datasets in, and can be harvested out. No special firewall about port 5432. Can you go through the site and point out which part is not functioning as expected? Let me know if you need ckan or ssh access. http://184.73.219.216/dataset

ccaudill commented 9 years ago

I can see that I actually get a response from http://184.73.219.216/csw?request=GetCapabilities&Service=CSW&Version=2.0.2 so it seems to be working from all your installs, but not ours or our partners installs. Should we ask (as soon as the issue https://github.com/ngds/ckanext-geoserver/issues/7 is resolved) to completely empty their server and install this new build? Since I do get a response from the CSW from your new build and do not from our partners at MT, I'm guessing the harvest will work fine from yours - I'm testing that now. But why are we having these problems if I just updated our servers too from the rpm a few weeks ago? I guess we can wipe ours a try a fresh install too?

FuhuXia commented 9 years ago

@ccaudill If we can find an easy and reliable way to replicate the issue yuou are experiencing on a freshly installed instance, it makes much easier for me to fix it. This way you don't have to wipe your existing servers. If not, then starting new might make more sense.

ccaudill commented 9 years ago

thanks @FuhuXia, we'll try that!

ccaudill commented 9 years ago

@FuhuXia I wonder if you could provide me the postgresql.conf and the pg_hba.conf files from your fresh install, just so I can compare with what we had on a fresh install?

smrgeoinfo commented 9 years ago

@ccaudill @FuhuXia this issue thread is pretty far from the topic line above. Please start new issues when discussing a different problem-- that will make it easier to find things in the issue tracker. You could reference the beginning of the discussion in another thread for situations like this where a problem evolves.

ccaudill commented 9 years ago

@FuhuXia Can you comment on Steve's from 3/17 above? It seems that those paster commands got the CSW working. Are those something that can do into the harvest process?

Comments after that regarding Postgres and harvesting can be carried on at https://github.com/ngds/install-and-run/issues/25 as they don't really have bearing on just whether or not the pyCSW is being properly populated. ... Sorry for going so off-topic there!

smrgeoinfo commented 9 years ago

pycsw is working on the current rpm install e.g. http://uat-ngds.reisys.com/csw?request=GetRecords&version=2.0.2&service=CSW&typeNames=csw:Record&resultType=results&outputFormat=application/xml&ElementSetName=brief

The issue is not the pycsw installation but the firewall and security issues in the OS. Close this issue, and see #25

ccaudill commented 9 years ago

http://final.geothermaldata.org/harvest/admin/geological-survey-of-alabama-gin-stack @dan-olaru-reisys This test harvest has been RUnning for an hour and a half and nothing has harvested. Might you have any suggestions? This is the important test before we can move forward with a v2

smrgeoinfo commented 9 years ago

Christy-- there are 10 records showing as harvested fro this source now, consistent with what I get with a csw request: http://ginstack.gsa.state.al.us/csw?request=GetRecords&version=2.0.2&service=CSW&typeNames=csw:Record&outputSchema=http://www.isotc211.org/2005/gmd&resultType=results&outputFormat=application/xml&ElementSetName=full&maxRecords=20

I think its working

ccaudill commented 9 years ago

WOOHOO! I'm not sure why it took so long, but it is working!