search

freme-project / e-Entity

Apache License 2.0
1 stars 1 forks source link

upload compressed data sets #66

Open ghsnd opened 8 years ago

ghsnd commented 8 years ago

An idea for a possible future feature: allow to upload compressed datasets. For instance the traditional RDF serializations compressed with gzip, xz, ..., but also something like HDT or compressed HDT. This allows larger datasets to be uploaded.

jnehring commented 8 years ago

Good idea. As far as I know GZIP compression is a standard mechanism of HTTP. There are many code examples that enable GZIP compression in Spring MVC. This works on a filter level across all endpoints. Maybe we can add this as a general feature so you can compress data you upload to FREME for enrichment also to save bandwidth.

ghsnd commented 8 years ago

Good idea. As far as I know GZIP compression is a standard mechanism of HTTP. There are many code examples that enable GZIP compression in Spring MVC. This works on a filter level across all endpoints. Maybe we can add this as a general feature so you can compress data you upload to FREME for enrichment also to save bandwidth.

Exactly.

But what I also mean is (offline) RDF-specific compression, to reduce the size of a large dataset drastically. For instance the aforementioned HDT format is a very compact serialization of RDF, and as a plus, it can be further compressed using general compressors like gzip. So if I have a dataset in this format, it would be handy if I can upload it and the server then stores the triples into whatever format. The LDF server even serves HDT files out of the box.

jnehring commented 8 years ago

The specification of FREME 0.5 is already over. I put it on the Future feature list so we can keep this feature in mind in a future version of FREME.