search

Esri / geoportal-server

Geoportal Server is a standards-based, open source product that enables discovery and use of geospatial resources including data and services.
https://gptogc.esri.com/geoportal
Apache License 2.0
244 stars 149 forks source link

Harvesting from ESRI Open Data site into Geoportal v.1.2.9 #311

Closed cybersea closed 4 years ago

cybersea commented 5 years ago

Can you provide some tips/suggestions for harvesting from an ESRI Open data site (example: https://dnr-msp-wa.opendata.arcgis.com/) with Geoportal server v.1.2.9. I used the DCAT harvest protocol and the test was successful, but it harvested zero records. Any tips are welcome to debugging this.

Is DCAT the best way to go? Or is there a better approach?

If DCAT is the best method, is there a way to selectively harvest records from a site with DCAT (like the approach used for CSW)?

pandzel-zz commented 5 years ago

Latest commit to the code fixes issue with harvesting mentioned DCAT endpoint.

cybersea commented 5 years ago

Thanks for the quick fix folks!

I'm still interested to hear if there is a mechanism within the Geoportal server to selectively harvest a subset of records from a DCAT feed.

cybersea commented 5 years ago

Hi Folks,

@pandzel I really want to deploy this fix on our site, but I'm getting errors when deploying the war file I built from the source code.

Would it be possible for you to build it on your end and share it? Any assistance is greatly appreciated.

pandzel-zz commented 5 years ago

@cybersea

@zguo will make a separate build for you and he will share binaries. Meanwhile, could you provide more details what's the nature of errors you are experiencing?

cybersea commented 5 years ago

Thanks @pandzel and @zguo

When deploying the war, which I built with Maven on Mac, the application does not start. I tried it both on the local machine where I built it, and on a CentOS server.

The catalina.out file had the following error: Jan07, 2019 11:55:12 AM org.apache.catalina.core.StandardContext start SEVERE: Error listenerStart

After adding these two lines to the /WEB-INF/classes/logging.properties org.apache.catalina.core.ContainerBase.[Catalina].level = INFO org.apache.catalina.core.ContainerBase.[Catalina].handlers = java.util.logging.ConsoleHandler

The catalina.out file shows a bit more detail

`Jan 07, 2019 11:55:12 AM org.apache.catalina.core.StandardContext listenerStart SEVERE: Exception sending context initialized event to listener instance of class com.sun.faces.config.ConfigureListener com.sun.faces.config.ConfigurationException: It appears the JSP version of the container is older than 2.1 and unable to locate the EL RI expression factory, com.sun.el.ExpressionFactoryImpl. If not using JSP or the EL RI, make sure the context initialization parameter, com.sun.faces.expressionFactory, is properly set. at com.sun.faces.config.ConfigureListener.registerELResolverAndListenerWithJsp(ConfigureListener.java:639) at com.sun.faces.config.ConfigureListener.contextInitialized(ConfigureListener.java:238) at org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:3972) at org.apache.catalina.core.StandardContext.start(StandardContext.java:4467) at org.apache.catalina.manager.ManagerServlet.start(ManagerServlet.java:1251) at org.apache.catalina.manager.HTMLManagerServlet.start(HTMLManagerServlet.java:612) at org.apache.catalina.manager.HTMLManagerServlet.doGet(HTMLManagerServlet.java:136) at javax.servlet.http.HttpServlet.service(HttpServlet.java:617) at javax.servlet.http.HttpServlet.service(HttpServlet.java:717) at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290) at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:558) at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) at java.lang.Thread.run(Thread.java:748)

Jan 07, 2019 11:55:12 AM org.apache.catalina.core.StandardContext start SEVERE: Error listenerStart Jan 07, 2019 11:55:12 AM org.apache.catalina.core.StandardContext start SEVERE: Context [/geoportal-1.2.9-SNAPSHOT] startup failed due to previous errors`

pandzel-zz commented 5 years ago

What is a version of the Apache Tomcat (if this is Tomcat) you are trying to deploy the war file to? I am asking because the stack trace you provided suggests you are using web server which doesn't meet requirements.

cybersea commented 5 years ago

Tomcat 6.0.24, JVM 1.8.0_191

The prebuilt v.1.2.9 WAR we downloaded from your wiki works on this server.

cybersea commented 5 years ago

@pandzel I tested on the Mac with Tomcat 7.0.92 and the geoportal application started. So, looks like we need to updgrade to Tomcat 7. Thanks for your help.

It's a bit of a mystery that your pre-built binaries are able to work on Tomcat 6.

cybersea commented 5 years ago

Hi @zguo Are you able to provide the binaries for this fix from @pandzel in the near future? I'm having trouble with the Tomcat 7 upgrade on our server and hoping to test out your pre-built version to see if the the fixes work for our application.

zguo commented 5 years ago

Hi @cybersea we will make a build and sent it to you in the next day also.

zguo commented 5 years ago

Hi @cybersea here is the link to the war file that contain the dcat fix, I did a quick test, I was able to harvest some dcat records. https://esri.box.com/s/ij0r48w4tx4j01vnpr5z4xv7uaj0pk50

cybersea commented 5 years ago

Thanks very much @zguo. I was able to deploy it successfully.

Can you tell me how you are building the war file, including the version of Java? Yours does not throw errors in Tomcat6 and your war file about half the size of the war file I built from your source code (using Maven per the wiki instructions).

zguo commented 5 years ago

Glad it works for you. Internally we are still use the ant script that was created a long time ago to run the build, the build environment has java 6.

cybersea commented 5 years ago

Thanks @zguo. Can you tell me the specific Java version so I can duplicate?

zguo commented 5 years ago

java version "1.6.0_43"

cybersea commented 4 years ago

Thanks for all the help on this folks. We were able to get this version running and successfully harvested the problematic DCAT source.