PydPiper output MINC files can be compressed further to save disk space

yohanyee commented 5 years ago

for example, using one of the large grid files associated with a nonlinear transformation:

export MINC_COMPRESS=4 mincresample -2 -like ${xfm_grid_file} ${xfm_grid_file} ${compressed_xfm_grid_file}

In my case, this brought the size down from 416M to 322M .

gdevenyi commented 5 years ago

Usually this environment variable is exported when minc toolkit is activated. We set it to 9

bcdarwin commented 5 years ago

It's currently set to 4 on our systems if you're exporting usual modules (if you hand-roll an environment all bets are off). I think this was determined years ago so perhaps we should consider increasing it as CPU time seems relatively cheap relative to disk space/access.

From: Gabriel A. Devenyi notifications@github.com Sent: April 8, 2019 11:11:15 AM To: Mouse-Imaging-Centre/pydpiper Cc: Subscribed Subject: Re: [Mouse-Imaging-Centre/pydpiper] PydPiper output MINC files can be compressed further to save disk space (#423)

Usually this environment variable is exported when minc toolkit is activated. We set it to 9

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHubhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_Mouse-2DImaging-2DCentre_pydpiper_issues_423-23issuecomment-2D480873510&d=DwMCaQ&c=Sj806OTFwmuG2UO1EEDr-2uZRzm2EPz39TfVBG2Km-o&r=WbPKw40NU3g_RTKn7pWL3cSAdk6QRKr3kMreWPZzNcg&m=a6dHoz7-Wdu35nwTjjjsgSq2bGM623O0addgFVt-Ln0&s=M1FHp6a6GHpcV0LWvSPxC4cPwyQylUel_4aH4W_HU9w&e=, or mute the threadhttps://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_AAKBNGh4JBunbtYlKA7cRJPCbr2mRGoZks5ve1wTgaJpZM4chL0U&d=DwMCaQ&c=Sj806OTFwmuG2UO1EEDr-2uZRzm2EPz39TfVBG2Km-o&r=WbPKw40NU3g_RTKn7pWL3cSAdk6QRKr3kMreWPZzNcg&m=a6dHoz7-Wdu35nwTjjjsgSq2bGM623O0addgFVt-Ln0&s=mW08gzrgMtPsB8WAoszOvzUH4HXee6Mgk73DGyM7f5Q&e=.

This e-mail may contain confidential, personal and/or health information(information which may be subject to legal restrictions on use, retention and/or disclosure) for the sole use of the intended recipient. Any review or distribution by anyone other than the person for whom it was originally intended is strictly prohibited. If you have received this e-mail in error, please contact the sender and delete all copies.

bcdarwin commented 5 years ago

Seems the displacement fields aren't being compressed -- perhaps ITK-related.

gdevenyi commented 5 years ago

Ah yes indeed, I enabled that in itk with a patch recently

gdevenyi commented 5 years ago

https://github.com/InsightSoftwareConsortium/ITK/pull/185

What we do now for using ANTs is build minc-toolkit-v2 without ANTs and then build the latest ANTs (with MINC enabled) and load overtop to provide the tools.

For now minc-toolkit-v2 is stuck on ITKv4 until development work to get to ITKv5 can be done and ANTs has moved on to be v5 based.

bcdarwin commented 4 years ago

As a workaround until this can be enabled, we could add minccompress steps to compress the ANTS output files in place.

Mouse-Imaging-Centre / pydpiper

PydPiper output MINC files can be compressed further to save disk space #423