search

MetPX / sarracenia

https://MetPX.github.io/sarracenia
GNU General Public License v2.0
46 stars 22 forks source link

Fix HTTP performance regression #1265

Closed reidsunderland closed 1 month ago

reidsunderland commented 1 month ago

This fixes the performance regression in the HTTP transfer class.

Instead of calling build_opener (which is very slow) for every __open__, it is only done in connect now.

I also made another change, which removes the setting of self.connected = False in the __open__ method. With that there, it forces a "re-connect" (and build_opener call) every time __open__ was called. Now connect gets called only when check_is_connected returns False, which will mainly happen if there is a change to the "sendTo" URL (if it was downloading from google.com, then we switch to downloading from example.com, it will re-connect).

If we have a feed that downloads from many different domains/sources, we would still see the performance issue, because of the re-connecting. I left a note in the comments saying that I think build_opener could actually only be done even less frequently, in __init__. But I was worried that that was too large of a change to make.

This change is just one commit, so it should be easily cherry-pickable if we want to make a 3.00.55post2 release

github-actions[bot] commented 1 month ago

Test Results

249 tests   248 :white_check_mark:  1m 29s :stopwatch:   1 suites    1 :zzz:   1 files      0 :x:

Results for commit 0fe61aa1.