I put some installation details and usage in the README where the idea was to have a CLI that would let you do this:
# Enable use of the file naming convention you mentioned before
> export CODEX_FILE_FORMAT_VERSION='v0.2'
> codex-processor \
--gpus=[0,1] # Can be a scalar or list of GPUs to use
--input-dir=path/to/raw/stacks \ # Contains "Cyc*_reg*" folders
--config-dir=path/to/configs \ # Defaults to data dir
--output-dir=path/to/output
As far as file naming conventions go, I added environment variables like CODEX_FILE_FORMAT_VERSION and CODEX_CONFIG_VERSION to control changes in configuration schemas and file naming versions, where the file naming schemas are declared like this.
I think the version 0.2 file naming formats there would take care of what you were asking about though I'm unclear on how to write output files normally like reg1_X01_Y01.tif in a format like C###_Z###_T###.tif so right now it's R###_X###_Y###.tif instead. Was the intention really to do away with X/Y coordinates in the resulting files? Let me know what I'm missing.
I tested it so far on Windows, Mac and Linux (via nvidia-docker) with little issue so far except that on Windows you also have to set an environment variable like set CODEX_CPU_ONLY_OPS=CodexFocalPlaneSelector,TranslationApplier since those operations in my code currently use TensorFlow ops with unsupported CUDA functions on Windows.
Also @armish , I know this is a tough commit to review since I rebased a lot into it (I'll keep them shorter if we get a back and forth going) but here are some key parts:
processor.py - This is the main entry point and hopefully it's pretty clear what's going on; The CLI functionality comes from Google's Fire project
pipeline.py - This module contains the core logic for the process; Any and all suggestions there would be great
config.py - Managing the configurations for these things is something I could probably use some insight on. I'm reading the json files directly and I started on jsonschema validations like this one for Experiment.json but haven't finished it yet. I would like to have some better control over configuration validation rather than a bunch of python but it seems like it will be tedious either way.
docker - Any thoughts/suggestions on how to do those better? I tried following what tensorflow does with its docker files but I'm by no means an experiment on it
Or suggestions on anything else would be great too!
Hey @nsamusik ,
I put some installation details and usage in the README where the idea was to have a CLI that would let you do this:
As far as file naming conventions go, I added environment variables like
CODEX_FILE_FORMAT_VERSIONandCODEX_CONFIG_VERSIONto control changes in configuration schemas and file naming versions, where the file naming schemas are declared like this.I think the version 0.2 file naming formats there would take care of what you were asking about though I'm unclear on how to write output files normally like
reg1_X01_Y01.tifin a format likeC###_Z###_T###.tifso right now it'sR###_X###_Y###.tifinstead. Was the intention really to do away with X/Y coordinates in the resulting files? Let me know what I'm missing.I tested it so far on Windows, Mac and Linux (via nvidia-docker) with little issue so far except that on Windows you also have to set an environment variable like
set CODEX_CPU_ONLY_OPS=CodexFocalPlaneSelector,TranslationAppliersince those operations in my code currently use TensorFlow ops with unsupported CUDA functions on Windows.Also @armish , I know this is a tough commit to review since I rebased a lot into it (I'll keep them shorter if we get a back and forth going) but here are some key parts:
Or suggestions on anything else would be great too!