We would like to have a job to progressively refresh favicons. Each domain should be re-fetched and stored once a week.
We expect to have ~1 million domain stored in the storage. This means that we need to refresh 1 favicon per minute.
Recommended approach:
create a listFiles(chunkSize, olderThan) function in both local and s3 drivers that should be able to fetch all files by chunk (maybe 1000 by 1000) and filter those that are older than a specified timestamp.
Once https://github.com/twentyhq/favicon/issues/13 is complete
We would like to have a job to progressively refresh favicons. Each domain should be re-fetched and stored once a week. We expect to have ~1 million domain stored in the storage. This means that we need to refresh 1 favicon per minute.
Recommended approach: