datatogether / archivertools

Python package for scraping websites into the Data Together pipeline via morph.io
GNU Affero General Public License v3.0
6 stars 1 forks source link

addFile's implementation is subject to sqlite's per-row size limit #12

Open jeffreyliu opened 6 years ago

jeffreyliu commented 6 years ago

Since addFile is just storing in the sqlite db as a BLOB object, it's subject to the default limit of ~1 Gb. To support larger files, we'd need to either separate the file in chunks, or use an alternate method of storing or uploading the files onto Data Together.