Over-writing output fixel directories

[Lestropie]

http://community.mrtrix.org/t/fixel-analysis-pipeline-index-mif-files-are-smaller-than-expected/1010

To me, suggesting that users explicitly make a copy of their fixel template before adding subject data to it (purely in order to retain a "clean" template), then having fixelcorrespondence / warp2metric receive a single fixel image as both input fixel template and output directory target, made more sense logically, and would be robust against this issue.

[Lestropie]

http://community.mrtrix.org/t/tck2fixel-segmentation-fault/2497

[Lestropie]

Integrating into the discussion on this issue the fact that the -force command-line option should work sensibly for fixel directories.

I think what may be needed here is a comprehensive list of commands that output fixel data (whether a complete directory, or a single fixel data file), and consider what the appropriate behaviour should be with equivalent input & output directories / with & without -force. I'll probably also propose how I would prefer the fixel command interfaces to be set up, which I think would resolve both the issues originally listed here and those mentioned in #2167.

[jdtournier]

There's a lot of moving parts here... Agree we need to list at least a few scenarios and try to figure out whether there's a simple strategy that would make sense - at least in principle.

I'm not sure I'm completely on top of the issue, but as far as I can tell there are at least two conflicting situations where -force could legitimately be invoked:

A command that produces a fresh fixel directory is asked to overwrite an existing fixel directory (e.g. fod2fixel)

In this case, there's no reason to expect the new fixel index & directions files to match any existing data file (at least not a priori). We should consider that such an operation should invalidate all existing data files, and so the logical thing to do is to wipe the entire folder clean.

This is quite a drastic operation, and very likely to lead to unexpected loss of data for users - they might not have realised that the command would remove all files, not just the ones that the command would ordinarily produce. Solutions might include (one or more of):

• perform a full comparison of the new and existing index and directions files, retain the originals if they match.
• provide a big warning and request confirmation before wiping any additional data files (if any are found). Could also be done with an additional option to be supplied on top of -force (e.g. -confirm, -clean , -wipe, ...), since requesting confirmation in an interactive manner would get in the way of scripts, etc.
• Abort operation if any additional data files are present that wouldn't otherwise have been generated by the command. This is difficult to do well as a backend operation without the app developer specifying what their outputs are going to be upfront, which is going to be fragile in the long run.
• Don't allow overwrite at all, and request users delete the folder manually if that's what they really want to do. Probably the simplest solution...

A command accepts fixel inputs and outputs, and these are allowed to be the same (e.g. tck2fixel)

These should not modify the index or directions files from the inputs, and just use what's been provided. Current problem seems related to the copy operation that happens even when the same folder is used for both input and output (?), so the main thing is to prevent that, and only update the relevant data files.

In fact, from a certain point of view, I'd probably prefer for commands such as these to only accept a single existing fixel folder that will act as both input and output, since what we're doing here is enhancing an existing fixel dataset with additional data files - not creating a whole new one as such. There's nothing preventing users from copying the whole thing and running the command on the copy if that's what they want to do, so I don't think this would cause any loss of functionality - but it would potentially reduce the complexity of what we need to support, since we would only need to worry about the data files, not the index or directions. Of course, this is dependent on there being no legitimate use case that can only be done with separate input and output folders - which is why your idea of going through the list of commands is worth doing.

I'm probably missing quite a few bits of the picture though, but hopefully someone can fill in the holes in my thinking...

[Lestropie]

I think a lot of the issues around fixel input / output directories arose from giving many commands that output a fixel data file the ability to duplicate the index / directions files as well through specifying both input and output fixel directories. This was a way of reducing the number of explicit steps in the FBA pipeline: there would be no explicit duplication required if one wanted to keep a clean copy of the fixel template before beginning to add data files, by simply using different input & output fixel directories at that step. I still think I was right to object to this at the time...

I'll try to make a list of relevant commands, what their current interfaces look like, and propose a cohesive set of alternatives.

Current interface

• fixel2tsf:

  • Input fixel directory
  • Input tracks
  • Output fixel data file (absolute path) Output fixel data file isn't constrained to be in the input fixel directory

• fixelconvert:

  • Input fixel directory / file
  • Output fixel directory / file No issue here; principally for converting to / from .msf, command could probably be deleted now.

• fixelcorrespondence:

  • Input fixel data file
  • Template fixel directory
  • Output fixel directory
  • Output fixel data file (basename only) I think this command is the main one I've had an objection to. It was certainly

• fixelcrop:

  • Input fixel directory
  • Output fixel directory Modifies index & directions, and creates new versions of all data files

• fixelfilter:

  • Input (either a fixel directory, or a fixel data file)
  • Operation
  • Output (either a fixel directory, or a fixel data file)

• fixelreorient:

  • Input fixel directory
  • Warp image
  • Output fixel directory Unique in that it modifies directions, but not index. Copies all data files from input to output.

• fod2fixel:

  • Input FOD image
  • Output fixel directory Writes brand new index & directions

• peaks2fixel:

  • Peaks image
  • Output fixel directory Writes brand new index & directions

• voxel2fixel:

  • Input 3D image
  • Input fixel directory
  • Output fixel directory
  • Output fixel data file (basename only)

Proposed changes

• fixel2tsf: Output fixel data file could be a basename, with the file pre-supposed to be going into the specified fixel directory (which would by definition be both an input and an output), to be more consistent with other changes below.

• fixelcorrespondence:

  • Input fixel data file
  • Template fixel directory
  • Output fixel data file (basename only) Input fixel directory is assumed from path to input fixel data file, as is currently done; single specified fixel directory is assumed to be both the fixels to which correspondence must be achieved, and the directory into which the output fixel data file will be written.

• fixelfilter: If input is directory, write new output directory, as is current behaviour; if input is single fixel data file, should the output argument be only a basename, with the assumption that it should be written into the same directory as the input fixel data file? (Currently it's interpreted as an absolute image file path)

• fixelreorient: There was I think some argument for over-writing core fixel directory images due to this command, because it can over-write the directions image and the index image can remain untouched and the format is still valid. However from a user's understanding perspective I would personally broadcast the index & directions files as being the two "core compulsory images", and modification of either constitutes a fundamental change to fixel structure, hence necessitating writing a new fixel directory. This command does however already copy all fixel data files from input to output directory.

• voxel2fixel:

  • Input 3D image
  • Template fixel directory
  • Output fixel data file (basename only) Same as fixelcorrespondence: specified fixel directory assumed to be both input & output

---

Agree with your points on overwriting. I was going to go with point 3 (only permit -force if all contents of the existing fixel directory would be overwritten), agree there's a risk of fragility though. I'll have to think more about it, see if there's a way to ensure that worst case scenario the -force option doesn't work.