mattdebont
commented
5 years ago - Calculate sitemap.xml on request;
- Always up to date with latest in database
- Table scan dynamodb and create xml on request
- Trivial attack vector
- Can force us to spend time re-generating xml over and over again, forcing us to scale up for robot scans of the system (there are many)
- It doesn't change apart from uploads so this is wasted, need to cache
- Trivial attack vector
- Update sitemap.xml as part of upload
- 1 to 1 operation, simple to understand in principle
- What happens with manual deletes? Need another chain to handle removing entries from the hub manually
- Updates many times per ingested asset
- Calculate and cache sitemap statically
- Daily sitemap update at x time
- Cache result to S3
- Run from DynamoDB or Elasticsearch
- arguably dynamodb is the better place for this, start from single source of truth, if something is in dynamodb it is 'published' no matter if its in the search results or not
- use published since last run? to reduce complexity?
- what about manual deletes again?
- https://medium.com/vevo-engineering/scalable-sitemap-solutions-with-aws-lambda-js-fd926c861c0d
- lambda job to update the sitemap, run daily? at x in the morning
- retireve all datahub assets
- for each asset add to sitemap-assets.xml
- for each resource in asset add to sitemap-resources.xml
- store in s3? have the app just front this delivery?
- update sitemap.xml to point to these (up to 50,000 records / 50MB) per file as per https://www.sitemaps.org/protocol.html#index should be fine to deal with compressed here so only real limit is likely the 50000 records potentially
- retireve all datahub assets
- lambda job to update the sitemap, run daily? at x in the morning
jonparsonsjncc
completer
To enable Google etc. to search, we need to generate a sitemap, or a least a browse, so that every asset is reachable from the homepage.