AyOK-Code / oscn

1 stars 1 forks source link

SOS Performance Fixes #253

Closed sgelbart closed 2 weeks ago

sgelbart commented 3 weeks ago

Summary

Cut the local runtime from 5+ hours to closer to two. The changes may be more noticeable on the server as it has to go over the wire for the database.

Changes

1) Insert ZipCodes for addresses in bulk This was being done row by row and due to the large amount of zip codes was still taking some time

2) Cache larger lookups in memory. The variable containing the large lookup (1390000 records) seem to be pretty well optimized in memory but take a few minutes to load.

Testing info:

irb(main):002:0> Time.now
=> 2024-08-21 18:53:31.129373628 +0000
irb(main):003:0> test = ::OkSos::Entity.pluck(:filing_number, :id).to_h
D, [2024-08-21T18:53:40.920200 #2] DEBUG -- :   OkSos::Entity Pluck (6723.8ms)  SELECT "ok_sos_entities"."filing_number", "ok_sos_entities"."id" FROM "ok_sos_entities"
=>                                                                                                                                     
{1912071632=>630019,                                                                                                                   
...                                                                                                                                    
irb(main):004:0> Time.now
=> 2024-08-21 18:56:45.623127182 +0000
  irb(main):007:0> ObjectSpace.memsize_of(test)
=> 67108960
irb(main):008:0> puts 'RAM USAGE: ' + `pmap #{Process.pid} | tail -1`[10,40].strip
RAM USAGE: 1345428K

Tested on performance-2xl dyno

github-actions[bot] commented 3 weeks ago

Simplecov Coverage

group name covered percent covered lines lines of code
Total 57.627 2229 3868