Working directory with "." character causes issues with command line parsing

[gdevenyi]

Run into a super odd bug here.

I was playing around with files in /tmp

> pwd
/tmp/tmp.BURqyslKWN

Trying this script:

#!/bin/bash
if [[ -n ${__mb_debug:-} ]]; then
  set -x
fi
set -euo pipefail
export ITK_GLOBAL_DEFAULT_NUMBER_OF_THREADS=${THREADS_PER_COMMAND:-$(nproc)}
export ITK_USE_THREADPOOL=1
export ITK_GLOBAL_DEFAULT_THREADER=Pool

tmpdir=$(mktemp -d)

movingfile=$1
fixedfile=$2
outputdir=$3

movingmask="NULL"
fixedmask="NULL"

antsRegistration --dimensionality 3 --verbose --minc \
  --output [${outputdir}/$(basename ${movingfile})-$(basename ${fixedfile})] \
  --use-histogram-matching 0 \
  --initial-moving-transform [${fixedfile},${movingfile},1] \
  --transform Translation[0.5] \
  --metric Mattes[${fixedfile},${movingfile},1,32,Regular,0.5] \
  --convergence [2025x2025x2025x2025x2025x2025,1e-6,10] \
  --shrink-factors 16x15x14x13x12x11 \
  --smoothing-sigmas 13.5891488046x12.7398270043x11.890505204x11.0411834037x10.1918616035x9.34253980317mm \
  --masks [NULL,NULL] \
  --transform Rigid[0.25] \
  --metric Mattes[${fixedfile},${movingfile},1,37,Regular,0.5] \
  --convergence [2025x2025x2025x2025x2025x2025,1e-6,10] \
  --shrink-factors 13x12x11x10x9x8 \
  --smoothing-sigmas 11.0411834037x10.1918616035x9.34253980317x8.49321800288x7.64389620259x6.7945744023mm \
  --masks [NULL,NULL] \
  --transform Similarity[0.125] \
  --metric Mattes[${fixedfile},${movingfile},1,64,Regular,0.5] \
  --convergence [2025x2025x2025x2025x2025x2025,1e-6,10] \
  --shrink-factors 10x9x8x7x6x5 \
  --smoothing-sigmas 8.49321800288x7.64389620259x6.7945744023x5.94525260202x5.09593080173x4.24660900144mm \
  --masks [NULL,NULL] \
  --transform Affine[0.1] \
  --metric Mattes[${fixedfile},${movingfile},1,256,Regular,0.5] \
  --convergence [2025x2025x2025x2025x2025x2025,1e-6,10] \
  --shrink-factors 7x6x5x4x3x2 \
  --smoothing-sigmas 5.94525260202x5.09593080173x4.24660900144x3.39728720115x2.54796540086x1.69864360058mm \
  --masks [NULL,NULL] \
  --transform Affine[0.1] \
  --metric Mattes[${fixedfile},${movingfile},1,256,None] \
  --convergence [2025x2025x2025x750x250,1e-6,20] \
  --shrink-factors 4x3x2x1x1 \
  --smoothing-sigmas 3.39728720115x2.54796540086x1.69864360058x0.849321800288x0mm \
  --masks [${fixedmask},${movingmask}]

If I run it:

> antsRegistration_affine.sh t1.mnc $mnimodel .
Stage 0
  iterations = 1
  convergence threshold = 1e-06
  convergence window size = 10
  number of levels = 1
  using the Mattes MI metric (number of bins = 32, weight = 1)

ERROR:  The number of shrink factors does not match the number of levels.
Shrink Factors: 6
Number Of Levels: 1

If I change my working directory:

> cd /
> /tmp/tmp.BURqyslKWN/antsRegistration_affine.sh /tmp/tmp.BURqyslKWN/t1.mnc $mnimodel /tmp/tmp.BURqyslKWN/
<registration proceeds>

Very odd.

[gdevenyi]

I think this is the same issue.. https://github.com/ANTsX/ANTs/issues/402

[gdevenyi]

And another one from antsApplyTransforms:

#Working dir /tmp/tmp.BURqyslKWN
Using double precision for computations.
Input scalar image: /tmp/greatzero.mnc
Reference image: t1.mnc
HDF5-DIAG: Error detected in HDF5 (1.10.3) thread 0:
  #000: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5F.c line 370 in itk_H5Fis_hdf5(): unable open file
    major: File accessibilty
    minor: Not an HDF5 file
  #001: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Fint.c line 806 in itk_H5F__is_hdf5(): unable to locate file signature
    major: File accessibilty
    minor: Not an HDF5 file
  #002: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDint.c line 126 in itk_H5FD_locate_signature(): unable to read file signature
    major: Low-level I/O
    minor: Unable to initialize object
  #003: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDint.c line 205 in itk_H5FD_read(): driver read request failed
    major: Virtual File Layer
    minor: Read failed
  #004: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDsec2.c line 716 in H5FD_sec2_read(): file read failed: time = Mon Sep 23 13:00:04 2019
, filename = '2', file descriptor = 3, errno = 21, error message = 'Is a directory', buf = 0x7ffca833dda0, total read size = 8, bytes this sub-read = 8, bytes actually read = 18446744073709551615, offset = 0
    major: Low-level I/O
    minor: Read failed
HDF5-DIAG: Error detected in HDF5 (1.10.3) thread 0:
  #000: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5F.c line 370 in itk_H5Fis_hdf5(): unable open file
    major: File accessibilty
    minor: Not an HDF5 file
  #001: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5Fint.c line 806 in itk_H5F__is_hdf5(): unable to locate file signature
    major: File accessibilty
    minor: Not an HDF5 file
  #002: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDint.c line 126 in itk_H5FD_locate_signature(): unable to read file signature
    major: Low-level I/O
    minor: Unable to initialize object
  #003: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDint.c line 205 in itk_H5FD_read(): driver read request failed
    major: Virtual File Layer
    minor: Read failed
  #004: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/ThirdParty/HDF5/src/itkhdf5/src/H5FDsec2.c line 716 in H5FD_sec2_read(): file read failed: time = Mon Sep 23 13:00:04 2019
, filename = '2', file descriptor = 3, errno = 21, error message = 'Is a directory', buf = 0x7ffca833dd30, total read size = 8, bytes this sub-read = 8, bytes actually read = 18446744073709551615, offset = 0
    major: Low-level I/O
    minor: Read failed
Transform reader for 2 caught an ITK exception:

itk::ExceptionObject (0x220bda0)
Location: "unknown" 
File: /opt/quarantine/ANTs/20190211/src/ANTs-build/ITKv5/Modules/IO/TransformBase/include/itkTransformFileReader.hxx
Line: 127
Description: itk::ERROR: TransformFileReaderTemplate(0x21ec560): Could not create Transform IO object for reading file 2
  Tried to create one of the following:
    HDF5TransformIOTemplate
    HDF5TransformIOTemplate
    MINCTransformIOTemplate
    MINCTransformIOTemplate
    MatlabTransformIOTemplate
    MatlabTransformIOTemplate
    TxtTransformIOTemplate
    TxtTransformIOTemplate
  You probably failed to set a file suffix, or
    set the suffix to an unsupported type.

#Working dir /
> antsApplyTransforms -d 3 -i /tmp/greatzero.mnc -r /tmp/tmp.BURqyslKWN/t1.mnc -t [/tmp/tmp.BURqyslKWN/t1.mnc-mni_icbm152_t1_tal_nlin_sym_09c.mnc0_GenericAffine.xfm,1] -n GenericLabel --verbose -o /tmp/tmp.BURqyslKWN/headmask.mnc
Using double precision for computations.
Input scalar image: /tmp/greatzero.mnc
Reference image: /tmp/tmp.BURqyslKWN/t1.mnc
=============================================================================
The composite transform comprises the following transforms (in order): 
  1. inverse of /tmp/tmp.BURqyslKWN/t1.mnc-mni_icbm152_t1_tal_nlin_sym_09c.mnc0_GenericAffine.xfm (type = AffineTransform)
=============================================================================
Default pixel value: 0
Interpolation type: LabelImageGenericInterpolateImageFunction
Output warped image: /tmp/tmp.BURqyslKWN/headmask.mnc

[cookpa]

Can you try a directory without periods in it?

[cookpa]

Also you don't have spaces around your square brackets - remember all those glob problems we had? Might be that too

[gdevenyi]

Can you try a directory without periods in it?

Yup, that's the problem. It works without periods. I'm guession extension stripping is too aggressive somewhere

[cookpa]

Can you try to find the simplest command that reproduces this?

This works for me on my Mac:

fixedfile=$1
movingfile=$2

cmd="antsRegistration --dimensionality 3 --verbose \
  --initial-moving-transform [ ${fixedfile},${movingfile},1 ] \
  --transform Translation[ 0.5 ] \
  --metric Mattes[ ${fixedfile},${movingfile},1,32,Regular,0.5 ] \
  --convergence [ 10x10x10x10,1e-6,10 ] \
  --shrink-factors 8x4x2x1 \
  --smoothing-sigmas 4x2x1x0mm \
  --output [ movingToFixed, movingToFixedDeformed.nii.gz ]"

echo "--- antsRegistration call ---"
echo $cmd
echo "---"

$cmd

I can then do

cd /tmp/tmpy.mctmpface
./antsCall.sh myImage.nii.gz myImage.nii.gz

Using

ANTs Version: 3.0.0.0.dev119-g13e0e
Compiled: Sep 18 2019 16:48:39

[cookpa]

I still also think glob is a potential culprit. I think number of levels are determined by --convergence, so if you do

--convergence [2025x2025x2025x2025x2025x2025,1e-6,10] 

bash can parse that to a single thing if there's a file named any one of those characters in the brackets. This would also explain why moving the working directory helps.

[gdevenyi]

Well this is super weird.

It seems to be an interaction between the working directory and the square brackets.

If I add the spaces, or I change my WD, I get a proper echo'd command:

antsRegistration --dimensionality 3 --verbose --minc --output [./t1.mnc-mni_icbm152_t1_tal_nlin_sym_09c.mnc] --use-histogram-matching 0 --initial-moving-transform [/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1] --transform Translation[0.5] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,32,Regular,0.5] --convergence [2025x2025x2025x2025x2025x2025,1e-6,10] --shrink-factors 16x15x14x13x12x11 --smoothing-sigmas 13.5891488046x12.7398270043x11.890505204x11.0411834037x10.1918616035x9.34253980317mm --masks [NULL,NULL] --transform Rigid[0.25] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,37,Regular,0.5] --convergence [2025x2025x2025x2025x2025x2025,1e-6,10] --shrink-factors 13x12x11x10x9x8 --smoothing-sigmas 11.0411834037x10.1918616035x9.34253980317x8.49321800288x7.64389620259x6.7945744023mm --masks [NULL,NULL] --transform Similarity[0.125] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,64,Regular,0.5] --convergence [2025x2025x2025x2025x2025x2025,1e-6,10] --shrink-factors 10x9x8x7x6x5 --smoothing-sigmas 8.49321800288x7.64389620259x6.7945744023x5.94525260202x5.09593080173x4.24660900144mm --masks [NULL,NULL] --transform Affine[0.1] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,256,Regular,0.5] --convergence [2025x2025x2025x2025x2025x2025,1e-6,10] --shrink-factors 7x6x5x4x3x2 --smoothing-sigmas 5.94525260202x5.09593080173x4.24660900144x3.39728720115x2.54796540086x1.69864360058mm --masks [NULL,NULL] --transform Affine[0.1] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,256,None] --convergence [2025x2025x2025x750x250,1e-6,20] --shrink-factors 4x3x2x1x1 --smoothing-sigmas 3.39728720115x2.54796540086x1.69864360058x0.849321800288x0mm --masks [NULL,NULL]

But without, I get broken convergence levels:

antsRegistration --dimensionality 3 --verbose --minc --output [./t1.mnc-mni_icbm152_t1_tal_nlin_sym_09c.mnc] --use-histogram-matching 0 --initial-moving-transform [/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1] --transform Translation[0.5] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,32,Regular,0.5] --convergence 0 1 2 --shrink-factors 16x15x14x13x12x11 --smoothing-sigmas 13.5891488046x12.7398270043x11.890505204x11.0411834037x10.1918616035x9.34253980317mm --masks [NULL,NULL] --transform Rigid[0.25] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,37,Regular,0.5] --convergence 0 1 2 --shrink-factors 13x12x11x10x9x8 --smoothing-sigmas 11.0411834037x10.1918616035x9.34253980317x8.49321800288x7.64389620259x6.7945744023mm --masks [NULL,NULL] --transform Similarity[0.125] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,64,Regular,0.5] --convergence 0 1 2 --shrink-factors 10x9x8x7x6x5 --smoothing-sigmas 8.49321800288x7.64389620259x6.7945744023x5.94525260202x5.09593080173x4.24660900144mm --masks [NULL,NULL] --transform Affine[0.1] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,256,Regular,0.5] --convergence 0 1 2 --shrink-factors 7x6x5x4x3x2 --smoothing-sigmas 5.94525260202x5.09593080173x4.24660900144x3.39728720115x2.54796540086x1.69864360058mm --masks [NULL,NULL] --transform Affine[0.1] --metric Mattes[/opt/quarantine/resources/mni_icbm152_nlin_sym_09c_minc2/mni_icbm152_t1_tal_nlin_sym_09c.mnc,/tmp/tmp.BURqyslKWN/t1.mnc,1,256,None] --convergence 0 1 2 --shrink-factors 4x3x2x1x1 --smoothing-sigmas 3.39728720115x2.54796540086x1.69864360058x0.849321800288x0mm --masks [NULL,NULL
``

So, I guess I close this as "weird bashism"

I'll audit my scripts and make pad all the square brackets.

[gdevenyi]

@cookpa Indeed, there are directories named "0" "1" and "2" in that working dir.

[cookpa]

For posterity:

https://github.com/ANTsX/ANTs/wiki/Command-line-parsing-issues

[gdevenyi]

I indeed had been following those issues but thought "that'll never happen to me" :)