ckan / ckanext-dcat

CKAN ♥ DCAT
164 stars 142 forks source link

Increase harvester get content chunk size #217

Closed seitenbau-govdata closed 2 years ago

seitenbau-govdata commented 2 years ago

We have some RDF endpoints which delivers a big dump file with thousands of datasets resulting in a large file of up to 100 MB. And we have noticed that getting the whole content takes a very long time and sometime even gets interrupted. The reason is the small chunk size of 1 KB. So we like to increase the chunk size up to 512 KB.

amercader commented 2 years ago

That sounds very reasonable