Slow server response time in development

[thatbudakguy]

When loading pages, including via turbo, the app noticeably takes a few seconds. A fresh build of Geoblacklight, by contrast, is very snappy – page loads execute much faster.

The browser console notes that the server response time is slow:

What's causing us to be different from vanilla Geoblacklight here?

[jmartin-sul]

TLDR: I did some testing for this issue Monday/yesterday/this morning, met with @thatbudakguy yesterday for a bit on my inconclusive results, and we decided that things were probably fine (pending the testing I did this morning on earthworks-uat, which didn't turn up anything concerning). Earthworks dev on laptop is a little slow, but not horribly so, and the bl8 branch didn't show any obvious performance issues in the UAT env (which was more of a concern than a somewhat slow dev environment).

more detail:

• i tested 4 scenarios
  • earthworks on my localhost; using sqlite, solr_wrapper, no redis
  • earthworks on my localhost; using docker (pg, solr, redis)
  • a fresh geoblacklight instance on my localhost, using the EngineCart generated test app
  • earthworks as deployed to our UAT env this morning
• for each test, i
  • did some things to try to reduce confounding factors from things like caching of already requested assets by the browser, or caching of frequently accessed data in memory by the OS
  • i tested in a private browser window
  • i restarted the browser before each test setup
  • i did a hard refresh for every page load (almost all of my testing was against the front page of the app in question, just visiting /)
  • visited the page i was testing about 20 times in a row, letting it fully finish loading before the next refresh
• for localhost earthworks tests, i loaded the fixture data per the README (but just what was loaded by bin/rake geoblacklight:solr:seed -- i didn't load any OpenGeoMetadata).
• i captured request/response data (which includes page timing) using the browser debugger (FF developer edition v130.0b2), persisting the request capture between page loads, and saving the capture locally as a HAR file. originally, i was going to save server log data in case that needed investigation, but helpfully in dev mode, rails sends back a header called server-timing, with info breaking down how long each part of the response construction took (e.g. sql.active_record duration, render_partial.action_view duration, etc)

And basically, I found that the first page load in my localhost testing took a while, but then the rest of the page loads were fine (based on both the numbers i captured and the subjective feel for me during testing). GeoBlacklight was actually worse on the initial page load (14.2 seconds!) in my test run, but was faster than Earthworks for all the other page loads.

I did notice tons of 404 errors for requests for openstreetmap PNGs in my local instance, but that didn't seem to cause any actual functional or performance trouble? e.g. https://a.tile.openstreetmap.fr/hot/1/2/1.png

I spent a little too much time learning about .har files and learning the basics of jq (HAR files are JSON) so that I could easily output stats without having to compile them by hand. Hopefully that knowledge pays off on some future debugging expedition 😅 (I wasn't able to find off the shelf HAR analysis tools like one can find for e.g. the callgrind format, though for basic visual analysis outside of a text editor, Firefox apparently lets you load a HAR file for viewing in the browser debugger).

I'm not going to upload the .har files, because 1) I'm guessing no one is actually going to look at them to do further analysis, and 2) I don't want to spend time sanitizing them (though I doubt there's anything more sensitive than some cookies for my local EW and GBL app instances).

But in the interest of providing some concrete timing details that someone besides me could judge, and saving some useful commands that might come in handy later for me or other jq noobs, here are the numbers I came up with this morning:

# wait times for overall page loads, with URLs in output (as a sanity check on the results)
cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks - docker (pg, solr, redis) - 127.0.0.1_Archive [24-08-06 23-28-58].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | { url: .request.url, waitTime: .timings.wait }])'
2683
[{"url":"http://127.0.0.1:3000/","waitTime":2345},{"url":"http://127.0.0.1:3000/","waitTime":125},{"url":"http://127.0.0.1:3000/","waitTime":113},{"url":"http://127.0.0.1:3000/","waitTime":109},{"url":"http://127.0.0.1:3000/","waitTime":106},{"url":"http://127.0.0.1:3000/","waitTime":114},{"url":"http://127.0.0.1:3000/","waitTime":101},{"url":"http://127.0.0.1:3000/","waitTime":147},{"url":"http://127.0.0.1:3000/","waitTime":110},{"url":"http://127.0.0.1:3000/","waitTime":114},{"url":"http://127.0.0.1:3000/","waitTime":103},{"url":"http://127.0.0.1:3000/","waitTime":99},{"url":"http://127.0.0.1:3000/","waitTime":147},{"url":"http://127.0.0.1:3000/","waitTime":95},{"url":"http://127.0.0.1:3000/","waitTime":97},{"url":"http://127.0.0.1:3000/","waitTime":107},{"url":"http://127.0.0.1:3000/","waitTime":103},{"url":"http://127.0.0.1:3000/","waitTime":174},{"url":"http://127.0.0.1:3000/","waitTime":105},{"url":"http://127.0.0.1:3000/","waitTime":104},{"url":"http://127.0.0.1:3000/","waitTime":103},{"url":"http://127.0.0.1:3000/","waitTime":100}]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks, sqlite, solr_wrapper, no redis - 127.0.0.1_Archive [2024-08-06 18-25-03].har' | jq  --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | { url: .request.url, waitTime: .timings.wait }])'
2341
[{"url":"http://127.0.0.1:3000/","waitTime":1885},{"url":"http://127.0.0.1:3000/","waitTime":97},{"url":"http://127.0.0.1:3000/","waitTime":96},{"url":"http://127.0.0.1:3000/","waitTime":97},{"url":"http://127.0.0.1:3000/","waitTime":95},{"url":"http://127.0.0.1:3000/","waitTime":97},{"url":"http://127.0.0.1:3000/","waitTime":94},{"url":"http://127.0.0.1:3000/","waitTime":100},{"url":"http://127.0.0.1:3000/","waitTime":96},{"url":"http://127.0.0.1:3000/","waitTime":89},{"url":"http://127.0.0.1:3000/","waitTime":91},{"url":"http://127.0.0.1:3000/","waitTime":133},{"url":"http://127.0.0.1:3000/","waitTime":94},{"url":"http://127.0.0.1:3000/","waitTime":94},{"url":"http://127.0.0.1:3000/","waitTime":103},{"url":"http://127.0.0.1:3000/","waitTime":96},{"url":"http://127.0.0.1:3000/","waitTime":84},{"url":"http://127.0.0.1:3000/","waitTime":98},{"url":"http://127.0.0.1:3000/","waitTime":95},{"url":"http://127.0.0.1:3000/","waitTime":95},{"url":"http://127.0.0.1:3000/","waitTime":84}]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu doing searches and visiting results and clicking around Archive [2024-08-08 12-02-01].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("^https:..earthworks-uat.stanford.edu.catalog"))) | { url: .request.url, waitTime: .timings.wait }])'
-1
[{"url":"https://earthworks-uat.stanford.edu/catalog?f%5Bgbl_resourceClass_sm%5D%5B%5D=Maps","waitTime":159},{"url":"https://earthworks-uat.stanford.edu/catalog/suggest?q=groundwater","waitTime":44},{"url":"https://earthworks-uat.stanford.edu/catalog/dec38cf0-183a-42c7-92fa-8addd3a64c25/track?counter=2&document_id=dec38cf0-183a-42c7-92fa-8addd3a64c25&search_id=1809","waitTime":19},{"url":"https://earthworks-uat.stanford.edu/catalog/dec38cf0-183a-42c7-92fa-8addd3a64c25","waitTime":102},{"url":"https://earthworks-uat.stanford.edu/catalog/dec38cf0-183a-42c7-92fa-8addd3a64c25/relations","waitTime":284},{"url":"https://earthworks-uat.stanford.edu/catalog/3a0b094c-0d3d-402f-b688-6d01f4e59f43/track?counter=4&document_id=3a0b094c-0d3d-402f-b688-6d01f4e59f43&search_id=1809","waitTime":16},{"url":"https://earthworks-uat.stanford.edu/catalog/3a0b094c-0d3d-402f-b688-6d01f4e59f43","waitTime":93},{"url":"https://earthworks-uat.stanford.edu/catalog/3a0b094c-0d3d-402f-b688-6d01f4e59f43/relations","waitTime":296},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-py212tb3061/track?counter=8&document_id=stanford-py212tb3061&search_id=1809","waitTime":20},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-py212tb3061","waitTime":96},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-py212tb3061/relations","waitTime":266},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-jx177gz2729/track?counter=8&document_id=stanford-jx177gz2729&search_id=1812","waitTime":22},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-jx177gz2729","waitTime":113},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-jx177gz2729/relations","waitTime":283},{"url":"https://earthworks-uat.stanford.edu/catalog?f%5Bdct_subject_sm%5D%5B%5D=Inland+Waters&f%5Bdct_subject_sm%5D%5B%5D=Hydrogeology&q=groundwater&search_field=all_fields&bbox=-130.429687+28.459033+-108.676758+42.682435","waitTime":396},{"url":"https://earthworks-uat.stanford.edu/catalog?f%5Bdct_subject_sm%5D%5B%5D=Inland+Waters&f%5Bdct_subject_sm%5D%5B%5D=Hydrogeology&q=groundwater&search_field=all_fields&bbox=-135.263672+28.536275+-113.510742+42.747012","waitTime":177},{"url":"https://earthworks-uat.stanford.edu/catalog?f%5Bdct_subject_sm%5D%5B%5D=Inland+Waters&f%5Bdct_subject_sm%5D%5B%5D=Hydrogeology&q=groundwater&search_field=all_fields&bbox=-135.307617+32.249974+-113.554687+45.828799","waitTime":166},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-hp422bd8542/track?counter=9&document_id=stanford-hp422bd8542&search_id=1809","waitTime":21},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-hp422bd8542","waitTime":98},{"url":"https://earthworks-uat.stanford.edu/catalog/stanford-hp422bd8542/relations","waitTime":283}]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu_Archive [2024-08-08 09-36-26].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | { url: .request.url, waitTime: .timings.wait }])'
3215
[{"url":"https://earthworks-uat.stanford.edu/","waitTime":119},{"url":"https://earthworks-uat.stanford.edu/","waitTime":124},{"url":"https://earthworks-uat.stanford.edu/","waitTime":135},{"url":"https://earthworks-uat.stanford.edu/","waitTime":358},{"url":"https://earthworks-uat.stanford.edu/","waitTime":128},{"url":"https://earthworks-uat.stanford.edu/","waitTime":123},{"url":"https://earthworks-uat.stanford.edu/","waitTime":136},{"url":"https://earthworks-uat.stanford.edu/","waitTime":116},{"url":"https://earthworks-uat.stanford.edu/","waitTime":131},{"url":"https://earthworks-uat.stanford.edu/","waitTime":116},{"url":"https://earthworks-uat.stanford.edu/","waitTime":123},{"url":"https://earthworks-uat.stanford.edu/","waitTime":132},{"url":"https://earthworks-uat.stanford.edu/","waitTime":112},{"url":"https://earthworks-uat.stanford.edu/","waitTime":127},{"url":"https://earthworks-uat.stanford.edu/","waitTime":134},{"url":"https://earthworks-uat.stanford.edu/","waitTime":127},{"url":"https://earthworks-uat.stanford.edu/","waitTime":127},{"url":"https://earthworks-uat.stanford.edu/","waitTime":133},{"url":"https://earthworks-uat.stanford.edu/","waitTime":116},{"url":"https://earthworks-uat.stanford.edu/","waitTime":124},{"url":"https://earthworks-uat.stanford.edu/","waitTime":122}]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu_Archive [2024-08-08 12-00-20].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | { url: .request.url, waitTime: .timings.wait }])'
789
[{"url":"https://earthworks-uat.stanford.edu/","waitTime":135},{"url":"https://earthworks-uat.stanford.edu/","waitTime":87},{"url":"https://earthworks-uat.stanford.edu/","waitTime":96},{"url":"https://earthworks-uat.stanford.edu/","waitTime":94},{"url":"https://earthworks-uat.stanford.edu/","waitTime":105},{"url":"https://earthworks-uat.stanford.edu/","waitTime":83},{"url":"https://earthworks-uat.stanford.edu/","waitTime":78},{"url":"https://earthworks-uat.stanford.edu/","waitTime":71},{"url":"https://earthworks-uat.stanford.edu/","waitTime":87},{"url":"https://earthworks-uat.stanford.edu/","waitTime":90},{"url":"https://earthworks-uat.stanford.edu/","waitTime":84},{"url":"https://earthworks-uat.stanford.edu/","waitTime":80},{"url":"https://earthworks-uat.stanford.edu/","waitTime":77},{"url":"https://earthworks-uat.stanford.edu/","waitTime":77},{"url":"https://earthworks-uat.stanford.edu/","waitTime":81},{"url":"https://earthworks-uat.stanford.edu/","waitTime":84},{"url":"https://earthworks-uat.stanford.edu/","waitTime":88},{"url":"https://earthworks-uat.stanford.edu/","waitTime":85},{"url":"https://earthworks-uat.stanford.edu/","waitTime":78},{"url":"https://earthworks-uat.stanford.edu/","waitTime":81},{"url":"https://earthworks-uat.stanford.edu/","waitTime":90},{"url":"https://earthworks-uat.stanford.edu/","waitTime":91}]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/geoblacklight, enginecart - 127.0.0.1_Archive [24-08-06 23-42-16].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | { url: .request.url, waitTime: .timings.wait }])'
1781
[{"url":"http://127.0.0.1:3000/","waitTime":14200},{"url":"http://127.0.0.1:3000/","waitTime":79},{"url":"http://127.0.0.1:3000/","waitTime":155},{"url":"http://127.0.0.1:3000/","waitTime":80},{"url":"http://127.0.0.1:3000/","waitTime":78},{"url":"http://127.0.0.1:3000/","waitTime":76},{"url":"http://127.0.0.1:3000/","waitTime":74},{"url":"http://127.0.0.1:3000/","waitTime":76},{"url":"http://127.0.0.1:3000/","waitTime":89},{"url":"http://127.0.0.1:3000/","waitTime":84},{"url":"http://127.0.0.1:3000/","waitTime":74},{"url":"http://127.0.0.1:3000/","waitTime":76},{"url":"http://127.0.0.1:3000/","waitTime":74},{"url":"http://127.0.0.1:3000/","waitTime":69},{"url":"http://127.0.0.1:3000/","waitTime":83},{"url":"http://127.0.0.1:3000/","waitTime":76},{"url":"http://127.0.0.1:3000/","waitTime":53},{"url":"http://127.0.0.1:3000/","waitTime":74},{"url":"http://127.0.0.1:3000/","waitTime":70},{"url":"http://127.0.0.1:3000/","waitTime":69},{"url":"http://127.0.0.1:3000/","waitTime":75},{"url":"http://127.0.0.1:3000/","waitTime":73}]

# wait times for overall page loads, without URLs in output
cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks - docker (pg, solr, redis) - 127.0.0.1_Archive [24-08-06 23-28-58].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | .timings.wait])'
2683
[2345,125,113,109,106,114,101,147,110,114,103,99,147,95,97,107,103,174,105,104,103,100]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks, sqlite, solr_wrapper, no redis - 127.0.0.1_Archive [2024-08-06 18-25-03].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | .timings.wait])'
2341
[1885,97,96,97,95,97,94,100,96,89,91,133,94,94,103,96,84,98,95,95,84]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu doing searches and visiting results and clicking around Archive [2024-08-08 12-02-01].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("^https:..earthworks-uat.stanford.edu.catalog"))) | .timings.wait])'
-1
[159,44,19,102,284,16,93,296,20,96,266,22,113,283,396,177,166,21,98,283]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu_Archive [2024-08-08 09-36-26].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | .timings.wait])'
3215
[119,124,135,358,128,123,136,116,131,116,123,132,112,127,134,127,127,133,116,124,122]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu_Archive [2024-08-08 12-00-20].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | .timings.wait])'
789
[135,87,96,94,105,83,78,71,87,90,84,80,77,77,81,84,88,85,78,81,90,91]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/geoblacklight, enginecart - 127.0.0.1_Archive [24-08-06 23-42-16].har' | jq --compact-output '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.request.url | test("/$"))) | .timings.wait])'
1781
[14200,79,155,80,78,76,74,76,89,84,74,76,74,69,83,76,53,74,70,69,75,73]

# slow requests of all kinds (not necessarily top level user requested page), with request URLs and HTTP response codes
cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks - docker (pg, solr, redis) - 127.0.0.1_Archive [24-08-06 23-28-58].har' | jq '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.timings.wait > 250)) | { url: .request.url, waitTime: .timings.wait, httpRespCode: .response.status }])'
2683
[
  {
    "url": "http://127.0.0.1:3000/",
    "waitTime": 2345,
    "httpRespCode": 200
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 3179,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/2/0.png",
    "waitTime": 3193,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 3183,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 3182,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/-2/0.png",
    "waitTime": 3177,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 3190,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 3180,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 3176,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 363,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 363,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 363,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 362,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 364,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 364,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 340,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 340,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 340,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 351,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 366,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 367,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 367,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 351,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 347,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 355,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 356,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 356,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 355,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 370,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 361,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 361,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 370,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 369,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 350,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 349,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 349,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 371,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 371,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 372,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 371,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 372,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 364,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 342,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 342,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 364,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 363,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 355,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/2/0.png",
    "waitTime": 3202,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 355,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 355,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/-2/0.png",
    "waitTime": 3201,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 3190,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 352,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 354,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 354,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 354,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 352,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 358,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 357,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 356,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 358,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 356,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 367,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 351,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 341,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 341,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 351,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 363,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 381,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 382,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 363,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 377,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 358,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 358,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 357,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 376,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 347,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 347,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 346,
    "httpRespCode": 404
  }
]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks, sqlite, solr_wrapper, no redis - 127.0.0.1_Archive [2024-08-06 18-25-03].har' | jq '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.timings.wait > 250)) | { url: .request.url, waitTime: .timings.wait, httpRespCode: .response.status }])'
2341
[
  {
    "url": "http://127.0.0.1:3000/",
    "waitTime": 1885,
    "httpRespCode": 200
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 3175,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/2/0.png",
    "waitTime": 3182,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 3171,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 3176,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/-2/0.png",
    "waitTime": 3177,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 3174,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 3172,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 3184,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 3250,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/2/0.png",
    "waitTime": 3173,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 3281,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 3281,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/-2/0.png",
    "waitTime": 3173,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 3281,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 3250,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 3250,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 337,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 337,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 337,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 332,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 338,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 336,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 335,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 335,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 335,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 335,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 332,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 340,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 338,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 337,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 337,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 342,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 341,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 337,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 336,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 341,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 340,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 336,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 3198,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/2/0.png",
    "waitTime": 3193,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 3181,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 3181,
    "httpRespCode": 404
  },
  {
    "url": "https://c.tile.openstreetmap.fr/hot/1/-2/0.png",
    "waitTime": 3193,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 3181,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 3197,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 3198,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 343,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 342,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 334,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 334,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 335,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 335,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 336,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 336,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 334,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 333,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/3/1.png",
    "waitTime": 338,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-1/0.png",
    "waitTime": 336,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 339,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 340,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/3/0.png",
    "waitTime": 340,
    "httpRespCode": 404
  },
  {
    "url": "https://b.tile.openstreetmap.fr/hot/1/-2/1.png",
    "waitTime": 336,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/-1/1.png",
    "waitTime": 341,
    "httpRespCode": 404
  },
  {
    "url": "https://a.tile.openstreetmap.fr/hot/1/2/1.png",
    "waitTime": 341,
    "httpRespCode": 404
  }
]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu doing searches and visiting results and clicking around Archive [2024-08-08 12-02-01].har' | jq '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.timings.wait > 250)) | { url: .request.url, waitTime: .timings.wait, httpRespCode: .response.status }])'
-1
[
  {
    "url": "https://earthworks-uat.stanford.edu/catalog/dec38cf0-183a-42c7-92fa-8addd3a64c25/relations",
    "waitTime": 284,
    "httpRespCode": 200
  },
  {
    "url": "https://earthworks-uat.stanford.edu/?search_field=all_fields&q=groundwater",
    "waitTime": 290,
    "httpRespCode": 200
  },
  {
    "url": "https://earthworks-uat.stanford.edu/catalog/3a0b094c-0d3d-402f-b688-6d01f4e59f43/relations",
    "waitTime": 296,
    "httpRespCode": 200
  },
  {
    "url": "https://earthworks-uat.stanford.edu/catalog/stanford-py212tb3061/relations",
    "waitTime": 266,
    "httpRespCode": 200
  },
  {
    "url": "https://earthworks-uat.stanford.edu/?f%5Bdct_subject_sm%5D%5B%5D=Inland+Waters&f%5Bdct_subject_sm%5D%5B%5D=Groundwater&q=groundwater&search_field=all_fields",
    "waitTime": 259,
    "httpRespCode": 200
  },
  {
    "url": "https://earthworks-uat.stanford.edu/catalog/stanford-jx177gz2729/relations",
    "waitTime": 283,
    "httpRespCode": 200
  },
  {
    "url": "https://earthworks-uat.stanford.edu/catalog?f%5Bdct_subject_sm%5D%5B%5D=Inland+Waters&f%5Bdct_subject_sm%5D%5B%5D=Hydrogeology&q=groundwater&search_field=all_fields&bbox=-130.429687+28.459033+-108.676758+42.682435",
    "waitTime": 396,
    "httpRespCode": 200
  },
  {
    "url": "https://earthworks-uat.stanford.edu/?bbox=-135.307617+32.249974+-113.554687+45.828799&f%5Bdct_subject_sm%5D%5B%5D=Hydrogeology&q=groundwater&search_field=all_fields",
    "waitTime": 294,
    "httpRespCode": 200
  },
  {
    "url": "https://earthworks-uat.stanford.edu/catalog/stanford-hp422bd8542/relations",
    "waitTime": 283,
    "httpRespCode": 200
  }
]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu_Archive [2024-08-08 09-36-26].har' | jq '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.timings.wait > 250)) | { url: .request.url, waitTime: .timings.wait, httpRespCode: .response.status }])'
3215
[
  {
    "url": "https://earthworks-uat.stanford.edu/",
    "waitTime": 358,
    "httpRespCode": 200
  }
]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/earthworks-uat.stanford.edu_Archive [2024-08-08 12-00-20].har' | jq '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.timings.wait > 250)) | { url: .request.url, waitTime: .timings.wait, httpRespCode: .response.status }])'
789
[]

cat ~/'Documents/notes, scratch, etc/earthworks1113-slow-server-response-time-in-dev/geoblacklight, enginecart - 127.0.0.1_Archive [24-08-06 23-42-16].har' | jq '(.log.pages[] | .pageTimings.onLoad), ([(.log.entries[] | select(.timings.wait > 250)) | { url: .request.url, waitTime: .timings.wait, httpRespCode: .response.status }])'
1781
[
  {
    "url": "http://127.0.0.1:3000/",
    "waitTime": 14200,
    "httpRespCode": 200
  }
]