Currently the crawler items (the meta data for a crawled job) are stored as embedded documents in the crawler entity. A crawler might be import a lot of jobs which could pose problems if we continue to embed the items.
So the items must be stored in a dedicated mongo collection and stored as referenced entities in the crawler entity.
cbleek
commented
5 years ago
Mongo documents have a maximum size.
Currently the crawler items (the meta data for a crawled job) are stored as embedded documents in the crawler entity. A crawler might be import a lot of jobs which could pose problems if we continue to embed the items.
So the items must be stored in a dedicated mongo collection and stored as referenced entities in the crawler entity.