search

GSA / data.gov

Main repository for the data.gov service
https://data.gov
Other
627 stars 99 forks source link

Solr 504 Gateway Time-out #3636

Closed robert-bryson closed 2 years ago

robert-bryson commented 2 years ago

Solr returns a 504 Gateway Time-out with query https://default-solr-2e001a6400465b1e-solrcloud.54a8e908-f9ea-49ea-8891-d927fd60261e.ssb.data.gov/solr/ckan/select?q=%3A&start=311111 (or any query that hits deep in the pagination).

How to reproduce

  1. Hit solr with above query

Expected behavior

Responds successfully and quickly

Actual behavior

504 Gateway Time-out error

Sketch

This was surfaced with deep paginations of the cloud.gov catalog site (like this for example) taking a long time to return (around 1min).

Probably related to https://github.com/GSA/datagov-deploy/issues/3642

mogul commented 2 years ago

We've increased the ingress-nginx timeout, which we believe was the source of the problem. However, we have yet to verify this as we need to complete load-testing first.

nickumia-reisys commented 2 years ago

Most recent result from the above query... Not sure how to interpret it yet.

image

nickumia-reisys commented 2 years ago

I checked this link, https://catalog-stage-datagov.app.cloud.gov/dataset/?page=1800

It still takes a long time to load, but it actually loads.

FuhuXia commented 2 years ago

I checked this link, https://catalog-stage-datagov.app.cloud.gov/dataset/?page=1800

It still takes a long time to load, but it actually loads.

On staging it timed 20+ seconds for most of deep pagination pages (> 15000). On prod it was about 40 seconds, probably due to prod Solr was busy at re-indexing.

FuhuXia commented 2 years ago

We rarely see the 500 error. We will mark resolved.