Right now the images for both the user's Avatars and Community icons can be massive, sometimes as large as 1080p or even larger. When crawling a site, the crawler should resize said icons to only be 16x16 (as required by the search results) and store them locally. Said images should be retained for as long as the cache-control headers specify in the response when fetching them.
Right now the images for both the user's Avatars and Community icons can be massive, sometimes as large as 1080p or even larger. When crawling a site, the crawler should resize said icons to only be 16x16 (as required by the search results) and store them locally. Said images should be retained for as long as the cache-control headers specify in the response when fetching them.