search

GeoscienceAustralia / agdc

Repository for Australian Geoscience Data Cube (AGDC) code
BSD 3-Clause "New" or "Revised" License
29 stars 24 forks source link

Increase GDAL memory sizes for improved compression and ingest speed. #53

Closed jeremyh closed 9 years ago

jeremyh commented 9 years ago

These commits are from December, but we never followed through to merge them.

The default GDAL memory sizes are tiny. By bumping them up, compression improves significantly (output files were half the size), and ingest time on our test data dropped from four minutes to around one minute in the NCI environment.

The output files are identical at the pixel level (the improved compression is presumably due to a much larger window size).

wenjunwu commented 9 years ago

Have we merged yet? If not, let’s merge it now.

From: Jeremy Hooke [mailto:notifications@github.com] Sent: Thursday, 14 May 2015 11:40 AM To: GeoscienceAustralia/agdc Subject: [agdc] Increase GDAL memory sizes for improved compression and ingest speed. (#53)

These commits are from December, but we never followed through to merge them.

The default GDAL memory sizes are tiny. By bumping them up, compression improves significantly (output files were half the size), and ingest time on our test data dropped from four minutes to around one minute in the NCI environment.

The output files are identical at the pixel level (the improved compression is presumably due to a much larger window size).

You can view, comment on, or merge this pull request online at:

https://github.com/GeoscienceAustralia/agdc/pull/53

Commit Summary

File Changes

Patch Links:

— Reply to this email directly or view it on GitHubhttps://github.com/GeoscienceAustralia/agdc/pull/53.

Geoscience Australia Disclaimer: This e-mail (and files transmitted with it) is intended only for the person or entity to which it is addressed. If you are not the intended recipient, then you have received this e-mail by mistake and any use, dissemination, forwarding, printing or copying of this e-mail and its file attachments is prohibited. The security of emails transmitted cannot be guaranteed; by forwarding or replying to this email, you acknowledge and accept these risks.