Currently we drop resources larger than 2MB because of limitations with the database - regardless of which backing store we end up with in the future, we are likely always going to need to chunk blobs.
We should
Configure a max object size for downloaded resources (set probably around 10MB for default)
Split resources smaller than this into <1MB chunks and store them in the DB in a way that can be put back together later e.g.
resource:{
url net.URL
data []byte
nextChunk int //id to resource chunk containing next part of the data
}
Currently we drop resources larger than 2MB because of limitations with the database - regardless of which backing store we end up with in the future, we are likely always going to need to chunk blobs.
We should
Split resources smaller than this into <1MB chunks and store them in the DB in a way that can be put back together later e.g.