search

internetarchive / dweb-mirror

Offline Internet Archive project
https://www-dweb-mirror.dev.archive.org/
GNU Affero General Public License v3.0
263 stars 27 forks source link

Really slow doing back or reload on the Bali collection or Search - why ... #270

Closed mitra42 closed 4 years ago

mitra42 commented 4 years ago

New ~/archiveorg Search "Bali" Click Crawl Reload (and poss even retype URL) - multi-minute

-- Also seen in details/Bali when hit <- back button or retype URL

mitra42 commented 4 years ago

Note - also an error at 

DAC.ArchiveMember.js in `new ArchiveMember`
const conforming = unexpanded ? o : ArchiveMember.processMetadataFjords(o, rules.member); // If claiming unexpanded dont check data
     so worth a breakpoint there.
mitra42 commented 4 years ago

Its reasonably fast now -

mitra42 commented 4 years ago

Reopen - actually much slower than should be . Trace info suggests 4s (on Mac) doing addCrawlInfo

mitra42 commented 4 years ago

Got it down to 450ms mostly by adding fetch_metadata in addDownloadedInfoToMembers above the parallel to stop it otherwise doing fetch_metadata in two places.