search

AtlasOfLivingAustralia / bie-index

Taxonomic search services
https://bie-ws.ala.org.au/ws
Other
1 stars 17 forks source link

bulklookup.json failing with a Logger error #229

Closed ansell closed 10 months ago

ansell commented 5 years ago

Calls for facet results from biocache-service (prod-bdown-b5) are causing an error on bie related to the bulklookup.json service in some cases:

tomcat bie-index.log on bie-b4-16

2019-02-04 11:54:55.245 ERROR --- [1-8080-exec-163] o.g.web.errors.GrailsExceptionResolver   : MissingMethodException occurred when processing request: [POST] /ws/species/guids/bulklookup.json
No signature of method: ch.qos.logback.classic.Logger.error() is applicable for argument types: (java.util.LinkedHashMap, java.io.IOException) values: [[error:Failed calling web service. Server returned HTTP response code: 40
0 for URL: http://localhost:8080/solr/bie/select?q=guid%3A%28%22Stylidium+despectum%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F7984728%7C%22Other+common+names%3A+small+triggerplant%22%7CPlantae%7CStylidiaceae%22%2C
%22Thelymitra+holmesii%7CNZOR-4-61973%7C%7CPlantae%7COrchidaceae%22%2C%22Tipimegus+kalkro%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A0a9bd236-47f9-43ad-bb7f-725f3fbcfeba%7C%7CAnimalia%7CPhoxocephalidae%22%2C%22Vicia+hir
suta%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2897943%7CHairy+vetch%7CPlantae%7CFabaceae%22%2C%22Xanthosia+huegelii%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2894397%7C%7CPlantae%7CApiaceae%22%2C%22Xe
rochrysum+palustre%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2890165%7C%7CPlantae%7CAsteraceae%22%2C%22Zygnema%7CNZOR-4-120163%7C%7CPlantae%7CZygnemataceae%22%2C%22ARACHNIDA%7Curn%3Alsid%3Abiodiversity.org.au%3Aaf
d.taxon%3A00a0a9ea-1ff1-4da7-8a52-1568791c5b1e%7C%7CAnimalia%7C%22%2C%22ASILIDAE%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3Aad83345e-f08a-4aaa-917e-f1f7f5faac8b%7CRobber+Flies%7CAnimalia%7CAsilidae%22%2C%22Acacia+retino
des%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2891480%7C%7CPlantae%7CFabaceae%22%2C%22Acatapaustus+metallopa%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A5b9fb3bb-d8f8-410d-83d8-9df1821ec68b%7C%7CAnimalia%7CNo
lidae%22%2C%22Accipiter+%28Leucospiza%29+novaehollandiae%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A77bf4e01-90bc-4763-b70b-6e699e61a44e%7CGrey+Goshawk%7CAnimalia%7CAccipitridae%22%2C%22Agrostis+stolonifera%7Chttp%3A%2F
%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2906084%7Ccreeping+bent%7CPlantae%7CPoaceae%22%2C%22Allittia+cardiocarpa%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2914001%7CSwamp+Daisy%7CPlantae%7CAsteraceae%22%2C%22Amp
erea+xiphoclada%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2912548%7C%7CPlantae%7CEuphorbiaceae%22%2C%22Amphibromus+fluitans%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2897596%7C%7CPlantae%7CPoaceae%22%2
C%22Amphidinium%7CNZOR-4-96740%7C%7CChromista%7CGymnodiniidae%22%2C%22Anestia+ombrophanes%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A8f0d7066-fe80-47ed-a26d-4567d15b5d85%7C%7CAnimalia%7CErebidae%22%2C%22Anthela+acuta%7C
urn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3Ac433bfc1-2cc2-4471-b24f-4890ee3b4c7f%7C%7CAnimalia%7CAnthelidae%22%2C%22Ardea+%28Mesophoyx%29%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A4a2d016d-355a-42da-b440-e300f6553f6
b%7C%7CAnimalia%7CArdeidae%22%29+OR+linkIdentifier%3A%28%22Stylidium+despectum%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F7984728%7C%22Other+common+names%3A+small+triggerplant%22%7CPlantae%7CStylidiaceae%22%2C%22Th
elymitra+holmesii%7CNZOR-4-61973%7C%7CPlantae%7COrchidaceae%22%2C%22Tipimegus+kalkro%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A0a9bd236-47f9-43ad-bb7f-725f3fbcfeba%7C%7CAnimalia%7CPhoxocephalidae%22%2C%22Vicia+hirsuta%
7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2897943%7CHairy+vetch%7CPlantae%7CFabaceae%22%2C%22Xanthosia+huegelii%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2894397%7C%7CPlantae%7CApiaceae%22%2C%22Xerochr
ysum+palustre%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2890165%7C%7CPlantae%7CAsteraceae%22%2C%22Zygnema%7CNZOR-4-120163%7C%7CPlantae%7CZygnemataceae%22%2C%22ARACHNIDA%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.tax
on%3A00a0a9ea-1ff1-4da7-8a52-1568791c5b1e%7C%7CAnimalia%7C%22%2C%22ASILIDAE%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3Aad83345e-f08a-4aaa-917e-f1f7f5faac8b%7CRobber+Flies%7CAnimalia%7CAsilidae%22%2C%22Acacia+retinodes%7
Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2891480%7C%7CPlantae%7CFabaceae%22%2C%22Acatapaustus+metallopa%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A5b9fb3bb-d8f8-410d-83d8-9df1821ec68b%7C%7CAnimalia%7CNolidae
%22%2C%22Accipiter+%28Leucospiza%29+novaehollandiae%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A77bf4e01-90bc-4763-b70b-6e699e61a44e%7CGrey+Goshawk%7CAnimalia%7CAccipitridae%22%2C%22Agrostis+stolonifera%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2906084%7Ccreeping+bent%7CPlantae%7CPoaceae%22%2C%22Allittia+cardiocarpa%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2914001%7CSwamp+Daisy%7CPlantae%7CAsteraceae%22%2C%22Amperea+xiphoclada%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2912548%7C%7CPlantae%7CEuphorbiaceae%22%2C%22Amphibromus+fluitans%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2897596%7C%7CPlantae%7CPoaceae%22%2C%22Amphidinium%7CNZOR-4-96740%7C%7CChromista%7CGymnodiniidae%22%2C%22Anestia+ombrophanes%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A8f0d7066-fe80-47ed-a26d-4567d15b5d85%7C%7CAnimalia%7CErebidae%22%2C%22Anthela+acuta%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3Ac433bfc1-2cc2-4471-b24f-4890ee3b4c7f%7C%7CAnimalia%7CAnthelidae%22%2C%22Ardea+%28Mesophoyx%29%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A4a2d016d-355a-42da-b440-e300f6553f6b%7C%7CAnimalia%7CArdeidae%22%29&fq=idxtype%3ATAXON&rows=20&wt=json URL= http://localhost:8080/solr/bie/select., ...], ...]
Possible solutions: error(java.lang.String), error(java.lang.String, [Ljava.lang.Object;), error(java.lang.String, java.lang.Object), error(java.lang.String, java.lang.Throwable), error(org.slf4j.Marker, java.lang.String), error(java.lang.String, java.lang.Object, java.lang.Object). Stacktrace follows:

groovy.lang.MissingMethodException: No signature of method: ch.qos.logback.classic.Logger.error() is applicable for argument types: (java.util.LinkedHashMap, java.io.IOException) values: [[error:Failed calling web service. Server returned HTTP response code: 400 for URL: http://localhost:8080/solr/bie/select?q=guid%3A%28%22Stylidium+despectum%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F7984728%7C%22Other+common+names%3A+small+triggerplant%22%7CPlantae%7CStylidiaceae%22%2C%22Thelymitra+holmesii%7CNZOR-4-61973%7C%7CPlantae%7COrchidaceae%22%2C%22Tipimegus+kalkro%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A0a9bd236-47f9-43ad-bb7f-725f3fbcfeba%7C%7CAnimalia%7CPhoxocephalidae%22%2C%22Vicia+hirsuta%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2897943%7CHairy+vetch%7CPlantae%7CFabaceae%22%2C%22Xanthosia+huegelii%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2894397%7C%7CPlantae%7CApiaceae%22%2C%22Xerochrysum+palustre%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2890165%7C%7CPlantae%7CAsteraceae%22%2C%22Zygnema%7CNZOR-4-120163%7C%7CPlantae%7CZygnemataceae%22%2C%22ARACHNIDA%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A00a0a9ea-1ff1-4da7-8a52-1568791c5b1e%7C%7CAnimalia%7C%22%2C%22ASILIDAE%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3Aad83345e-f08a-4aaa-917e-f1f7f5faac8b%7CRobber+Flies%7CAnimalia%7CAsilidae%22%2C%22Acacia+retinodes%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2891480%7C%7CPlantae%7CFabaceae%22%2C%22Acatapaustus+metallopa%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A5b9fb3bb-d8f8-410d-83d8-9df1821ec68b%7C%7CAnimalia%7CNolidae%22%2C%22Accipiter+%28Leucospiza%29+novaehollandiae%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A77bf4e01-90bc-4763-b70b-6e699e61a44e%7CGrey+Goshawk%7CAnimalia%7CAccipitridae%22%2C%22Agrostis+stolonifera%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2906084%7Ccreeping+bent%7CPlantae%7CPoaceae%22%2C%22Allittia+cardiocarpa%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2914001%7CSwamp+Daisy%7CPlantae%7CAsteraceae%22%2C%22Amperea+xiphoclada%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2912548%7C%7CPlantae%7CEuphorbiaceae%22%2C%22Amphibromus+fluitans%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2897596%7C%7CPlantae%7CPoaceae%22%2C%22Amphidinium%7CNZOR-4-96740%7C%7CChromista%7CGymnodiniidae%22%2C%22Anestia+ombrophanes%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A8f0d7066-fe80-47ed-a26d-4567d15b5d85%7C%7CAnimalia%7CErebidae%22%2C%22Anthela+acuta%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3Ac433bfc1-2cc2-4471-b24f-4890ee3b4c7f%7C%7CAnimalia%7CAnthelidae%22%2C%22Ardea+%28Mesophoyx%29%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A4a2d016d-355a-42da-b440-e300f6553f6b%7C%7CAnimalia%7CArdeidae%22%29+OR+linkIdentifier%3A%28%22Stylidium+despectum%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F7984728%7C%22Other+common+names%3A+small+triggerplant%22%7CPlantae%7CStylidiaceae%22%2C%22Thelymitra+holmesii%7CNZOR-4-61973%7C%7CPlantae%7COrchidaceae%22%2C%22Tipimegus+kalkro%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A0a9bd236-47f9-43ad-bb7f-725f3fbcfeba%7C%7CAnimalia%7CPhoxocephalidae%22%2C%22Vicia+hirsuta%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2897943%7CHairy+vetch%7CPlantae%7CFabaceae%22%2C%22Xanthosia+huegelii%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2894397%7C%7CPlantae%7CApiaceae%22%2C%22Xerochrysum+palustre%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2890165%7C%7CPlantae%7CAsteraceae%22%2C%22Zygnema%7CNZOR-4-120163%7C%7CPlantae%7CZygnemataceae%22%2C%22ARACHNIDA%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A00a0a9ea-1ff1-4da7-8a52-1568791c5b1e%7C%7CAnimalia%7C%22%2C%22ASILIDAE%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3Aad83345e-f08a-4aaa-917e-f1f7f5faac8b%7CRobber+Flies%7CAnimalia%7CAsilidae%22%2C%22Acacia+retinodes%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2891480%7C%7CPlantae%7CFabaceae%22%2C%22Acatapaustus+metallopa%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A5b9fb3bb-d8f8-410d-83d8-9df1821ec68b%7C%7CAnimalia%7CNolidae%22%2C%22Accipiter+%28Leucospiza%29+novaehollandiae%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A77bf4e01-90bc-4763-b70b-6e699e61a44e%7CGrey+Goshawk%7CAnimalia%7CAccipitridae%22%2C%22Agrostis+stolonifera%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2906084%7Ccreeping+bent%7CPlantae%7CPoaceae%22%2C%22Allittia+cardiocarpa%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2914001%7CSwamp+Daisy%7CPlantae%7CAsteraceae%22%2C%22Amperea+xiphoclada%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2912548%7C%7CPlantae%7CEuphorbiaceae%22%2C%22Amphibromus+fluitans%7Chttp%3A%2F%2Fid.biodiversity.org.au%2Fnode%2Fapni%2F2897596%7C%7CPlantae%7CPoaceae%22%2C%22Amphidinium%7CNZOR-4-96740%7C%7CChromista%7CGymnodiniidae%22%2C%22Anestia+ombrophanes%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A8f0d7066-fe80-47ed-a26d-4567d15b5d85%7C%7CAnimalia%7CErebidae%22%2C%22Anthela+acuta%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3Ac433bfc1-2cc2-4471-b24f-4890ee3b4c7f%7C%7CAnimalia%7CAnthelidae%22%2C%22Ardea+%28Mesophoyx%29%7Curn%3Alsid%3Abiodiversity.org.au%3Aafd.taxon%3A4a2d016d-355a-42da-b440-e300f6553f6b%7C%7CAnimalia%7CArdeidae%22%29&fq=idxtype%3ATAXON&rows=20&wt=json URL= http://localhost:8080/solr/bie/select., ...], ...]
Possible solutions: error(java.lang.String), error(java.lang.String, [Ljava.lang.Object;), error(java.lang.String, java.lang.Object), error(java.lang.String, java.lang.Throwable), error(org.slf4j.Marker, java.lang.String), error(java.lang.String, java.lang.Object, java.lang.Object)
        at au.org.ala.bie.SearchService.doPostWithParams(SearchService.groovy:1228)
        at au.org.ala.bie.SearchService.getTaxaBatch(SearchService.groovy:715)
        at au.org.ala.bie.SearchService.getTaxa(SearchService.groovy:689)
        at au.org.ala.bie.SearchController.bulkGuidLookup(SearchController.groovy:116)
        at org.grails.core.DefaultGrailsControllerClass$MethodHandleInvoker.invoke(DefaultGrailsControllerClass.java:223)
        at org.grails.core.DefaultGrailsControllerClass.invoke(DefaultGrailsControllerClass.java:188)
        at org.grails.web.mapping.mvc.UrlMappingsInfoHandlerAdapter.handle(UrlMappingsInfoHandlerAdapter.groovy:90)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
        at org.springframework.boot.web.filter.ApplicationContextHeaderFilter.doFilterInternal(ApplicationContextHeaderFilter.java:55)
        at org.grails.web.servlet.mvc.GrailsWebRequestFilter.doFilterInternal(GrailsWebRequestFilter.java:77)
        at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199)
        at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199)
        at au.org.ala.cas.client.UriFilter.doFilter(UriFilter.java:199)
        at org.jasig.cas.client.session.SingleSignOutFilter.doFilter(SingleSignOutFilter.java:97)
        at org.grails.web.filters.HiddenHttpMethodFilter.doFilterInternal(HiddenHttpMethodFilter.java:67)
        at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:117)
        at org.springframework.boot.web.support.ErrorPageFilter.access$000(ErrorPageFilter.java:61)
        at org.springframework.boot.web.support.ErrorPageFilter$1.doFilterInternal(ErrorPageFilter.java:92)
        at org.springframework.boot.web.support.ErrorPageFilter.doFilter(ErrorPageFilter.java:110)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

nginx access log on bie-b4-16:

13.237.6.165 - - [04/Feb/2019:11:54:55 +1100] "POST /ws/species/guids/bulklookup.json HTTP/1.1" 500 23709 "-" "Java/1.8.0_191" "-" request_time=0.396 upstream_response_time=0.396 upstream_connect_time=0.000 upstream_header_time=0.395 upstream_cache_status=-

logs on prod-bdown-b5:

==> /var/log/tomcat7/biocache-service.log <==
2019-02-04 11:54:55,245 [http-bio-127.0.0.1-8080-exec-1] WARN org.springframework.web.client.RestTemplate  (RestTemplate.java:559) - POST request for "https://bie.ala.org.au/ws/species/guids/bulklookup.json" resulted in 500 (Internal Server Error); invoking error handler
2019-02-04 11:54:55,246 [http-bio-127.0.0.1-8080-exec-1] ERROR au.org.ala.biocache.service.SpeciesLookupRestService  (SpeciesLookupRestService.java:163) - Requested URI: https://bie.ala.org.au/ws/species/guids/bulklookup.json
2019-02-04 11:54:55,247 [http-bio-127.0.0.1-8080-exec-1] ERROR au.org.ala.biocache.service.SpeciesLookupRestService  (SpeciesLookupRestService.java:164) - With POST body: guid=Stylidium despectum|http://id.biodiversity.org.au/node/apni/7984728|"Other common names: small triggerplant"|Plantae|Stylidiaceae&guid=Thelymitra holmesii|NZOR-4-61973||Plantae|Orchidaceae&guid=Tipimegus kalkro|urn:lsid:biodiversity.org.au:afd.taxon:0a9bd236-47f9-43ad-bb7f-725f3fbcfeba||Animalia|Phoxocephalidae&guid=Vicia hirsuta|http://id.biodiversity.org.au/node/apni/2897943|Hairy vetch|Plantae|Fabaceae&guid=Xanthosia huegelii|http://id.biodiversity.org.au/node/apni/2894397||Plantae|Apiaceae&guid=Xerochrysum palustre|http://id.biodiversity.org.au/node/apni/2890165||Plantae|Asteraceae&guid=Zygnema|NZOR-4-120163||Plantae|Zygnemataceae&guid=ARACHNIDA|urn:lsid:biodiversity.org.au:afd.taxon:00a0a9ea-1ff1-4da7-8a52-1568791c5b1e||Animalia|&guid=ASILIDAE|urn:lsid:biodiversity.org.au:afd.taxon:ad83345e-f08a-4aaa-917e-f1f7f5faac8b|Robber Flies|Animalia|Asilidae&guid=Acacia retinodes|http://id.biodiversity.org.au/node/apni/2891480||Plantae|Fabaceae&guid=Acatapaustus metallopa|urn:lsid:biodiversity.org.au:afd.taxon:5b9fb3bb-d8f8-410d-83d8-9df1821ec68b||Animalia|Nolidae&guid=Accipiter (Leucospiza) novaehollandiae|urn:lsid:biodiversity.org.au:afd.taxon:77bf4e01-90bc-4763-b70b-6e699e61a44e|Grey Goshawk|Animalia|Accipitridae&guid=Agrostis stolonifera|http://id.biodiversity.org.au/node/apni/2906084|creeping bent|Plantae|Poaceae&guid=Allittia cardiocarpa|http://id.biodiversity.org.au/node/apni/2914001|Swamp Daisy|Plantae|Asteraceae&guid=Amperea xiphoclada|http://id.biodiversity.org.au/node/apni/2912548||Plantae|Euphorbiaceae&guid=Amphibromus fluitans|http://id.biodiversity.org.au/node/apni/2897596||Plantae|Poaceae&guid=Amphidinium|NZOR-4-96740||Chromista|Gymnodiniidae&guid=Anestia ombrophanes|urn:lsid:biodiversity.org.au:afd.taxon:8f0d7066-fe80-47ed-a26d-4567d15b5d85||Animalia|Erebidae&guid=Anthela acuta|urn:lsid:biodiversity.org.au:afd.taxon:c433bfc1-2cc2-4471-b24f-4890ee3b4c7f||Animalia|Anthelidae&guid=Ardea (Mesophoyx)|urn:lsid:biodiversity.org.au:afd.taxon:4a2d016d-355a-42da-b440-e300f6553f6b||Animalia|Ardeidae&guid=Austrolestes annulosus|urn:lsid:biodiversity.org.au:afd.taxon:2fe21bed-d7f5-4e45-8deb-5d4e5b08fc4d||Animalia|Lestidae&guid=Austrolestes leda|urn:lsid:biodiversity.org.au:afd.taxon:b1cf9e51-2ece-4ee0-a048-1bf88187a362||Animalia|Lestidae&guid=Austrostipa rudis (Spreng.) S.W.L.Jacobs & J.Everett subsp. rudis|NZOR-4-122127||Plantae|Poaceae&guid=Austrostipa rudis subsp. australis|http://id.biodiversity.org.au/node/apni/2919659|Veined Spear-grass|Plantae|Poaceae&guid=Avena sterilis|http://id.biodiversity.org.au/node/apni/2915849|Sterile Oats|Plantae|Poaceae&guid=Bacteriastrum|NZOR-4-33162||Chromista|Chaetocerotaceae&guid=Caladenia tentaculata|http://id.biodiversity.org.au/name/apni/92457||Plantae|&guid=Calamanthus fuliginosus albiloris|urn:lsid:biodiversity.org.au:afd.taxon:73cc8ce6-9490-412e-a67a-e45aa1118049||Animalia|Acanthizidae&guid=Calamanthus pyrrhopygius|ALA_Calamanthus_pyrrhopygius|Chestnut-rumped Heathwren||&guid=Caloplaca|NZOR-4-49137||Fungi|Teloschistaceae
2019-02-04 11:54:55,248 [http-bio-127.0.0.1-8080-exec-1] ERROR au.org.ala.biocache.service.SpeciesLookupRestService  (SpeciesLookupRestService.java:165) - RestTemplate error: 500 Internal Server Error
org.springframework.web.client.HttpServerErrorException: 500 Internal Server Error
    at org.springframework.web.client.DefaultResponseErrorHandler.handleError(DefaultResponseErrorHandler.java:91)
    at org.springframework.web.client.RestTemplate.handleResponseError(RestTemplate.java:566)
    at org.springframework.web.client.RestTemplate.doExecute(RestTemplate.java:524)
    at org.springframework.web.client.RestTemplate.execute(RestTemplate.java:482)
    at org.springframework.web.client.RestTemplate.postForObject(RestTemplate.java:327)
    at au.org.ala.biocache.service.SpeciesLookupRestService.getNameDetailsForGuids(SpeciesLookupRestService.java:158)
    at au.org.ala.biocache.service.SpeciesLookupRestService.getSpeciesDetails(SpeciesLookupRestService.java:190)
    at au.org.ala.biocache.dao.SearchDAOImpl.writeTaxonDetailsToStream(SearchDAOImpl.java:868)
    at au.org.ala.biocache.dao.SearchDAOImpl.writeFacetToStream(SearchDAOImpl.java:815)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
    at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:198)
    at com.sun.proxy.$Proxy44.writeFacetToStream(Unknown Source)
    at au.org.ala.biocache.web.OccurrenceController.downloadFacet(OccurrenceController.java:916)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)
    at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)
    at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:104)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandleMethod(RequestMappingHandlerAdapter.java:743)
    at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:672)
    at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:82)
    at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:933)
    at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:867)
    at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:951)
    at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:842)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:624)
    at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:827)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at au.org.ala.biocache.web.SimpleCORSFilter.doFilter(SimpleCORSFilter.java:33)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at au.org.ala.biocache.web.LoggerConfigFilter.doFilter(LoggerConfigFilter.java:26)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.obtainContent(SiteMeshFilter.java:129)
    at com.opensymphony.sitemesh.webapp.SiteMeshFilter.doFilter(SiteMeshFilter.java:77)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:88)
    at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:106)
    at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
    at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
    at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:221)
    at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:122)
    at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:505)
    at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
    at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
    at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
    at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:436)
    at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1078)
    at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:625)
    at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
    at java.lang.Thread.run(Thread.java:748)
2019-02-04 11:55:03,611 [http-bio-127.0.0.1-8080-exec-1] INFO au.org.ala.biocache.dao.JsonPersistentQueueDAOImpl  (JsonPersistentQueueDAOImpl.java:236) - Deleting /data/cache/downloads/offline1549241691634.json false

==> /var/log/nginx/access.log <==
13.54.102.100 - - [04/Feb/2019:11:55:03 +1100] "GET /ws/occurrences/facets/download?facets=names_and_lsid&lookup=true&count=true&&q=geospatial_kosher%3A%22true%22&fq=*%3A*+AND+-%28occurrence_status%3A%22absent%22%29&wkt=POLYGON((145.073:-38.218,145.459:-38.218,145.459:-38.041,145.073:-38.041,145.073:-38.218)) HTTP/1.1" 200 1240328 "-" "Jakarta Commons-HttpClient/3.1" "13.54.102.100" request_time=12.003 upstream_response_time=12.004 upstream_connect_time=0.000 upstream_header_time=1.352 upstream_cache_status=-

I could replicate the issue using the biocache-ws.ala.org.au URL in the GET request above.

adam-collins commented 10 months ago

no longer an issue