Quid about compression ?

niconoe / DarwinSQL

DarwinCore Archives expressed as SQLite

BSD 2-Clause "Simplified" License

1 stars 0 forks source link

Quid about compression ? #3

Open niconoe opened 6 years ago

niconoe commented 6 years ago

DwCA are generally zipped. We want to provide a solution there, so a converted DwCA doesn't suddenly becomes 10x larger. There seems to be no perfect solution is SQLite3 for this. Possible approach:

zip the whole database?
see if index can decrease the size of databases?
...

niconoe commented 6 years ago

First test with a manual archive conversion:

A 13.3Mb initial zipped archive
Once uncompressed, it reveals a 85Mb occurrence.txt file
occurrence.txt is imported in SQLite => 98Mb
recompress manually occurrence.txt => 6.1Mb
compress manually SQLite file => 7.9Mb

niconoe commented 6 years ago

Other findings:

sqlite3 cannot directly open zipped databases: 'Error: file is encrypted or is not a database'
creating indexes increase the DB size (which makes sense)