NASA-PDS / ds-view

Data Set View application (pds.nasa.gov/ds-view)
Apache License 2.0
0 stars 0 forks source link

Memory leak in Solr connections leading to Tomcat crash #14

Closed jordanpadams closed 10 months ago

jordanpadams commented 10 months ago

Checked for duplicates

Yes - I've already checked

🐛 Describe the bug

When we got bombarded by openai crawling, Tomcat was spinning up too many open threads and crashing daily.

🕵️ Expected behavior

I expected we could take the traffic.

📜 To Reproduce

See SA logs and convo on Slack.

See these errors in catalina.out

"Connection evictor" #762 daemon prio=5 os_prio=0 tid=0x00007fc2700c9000 nid=0xd79b waiting on condition [0x00007fc248002000]
   java.lang.Thread.State: TIMED_WAITING (sleeping)
    at java.lang.Thread.sleep(Native Method)
    at org.apache.http.impl.client.IdleConnectionEvictor$1.run(IdleConnectionEvictor.java:66)
    at java.lang.Thread.run(Thread.java:748)

And when you take ds-view offline, it is explicitly noting connection evictor issues related to [ds-view], noting there is a memory leak spawning new threads and not closing them.

🖥 Environment Info

Chrome

📚 Version of Software Used

v2.14.3

🩺 Test Data / Additional context

No response

🦄 Related requirements

No response

⚙️ Engineering Details

No response