Need for an output_root directory

[stcui007]

In recent CONUS baseline testing, it is realized that a specific sub-directory is needed to store the ngen output data files, which count to over a million. Current form of realization files writes these files to project directory, which makes working in the directory extrmely incovennient. For example, a simple ls command takes many minutes to complete. Then you cannot spot the file you are looking for. In general, such situation would arise for any relatively large basin.

Code exists that accommodate this. But requires the user to create the sub-directory in advance, otherwise, it default to project directory. A updated version of the code added a safety feature for case the user forget to create the needed sub-directory.

To alleviate the problem, all current realization files need to add the following line: "output_root": "./output_dir/" at the top level of the realization json tree. That will make ngen to write all the output files to ./output_dir. You are free to choose your own favorite name instead of "output_dir". At the time of this writing, if you forget to create ./output_dir/in advance before running, the default output directory is the project directory.

The updated code (to be merged) throws an error if you do not created the output_root before starting ngen to remind you to create the subdirectory before things get messy. It may require all updated realization to be merged first.

Current behavior

Large amount of output files when running ngen for CONUS or any large basin.

Expected behavior

Output files are written to a user specified directory to keep the project directory clean.

Steps to replicate behavior (include URLs)

1. Running ngen in framework for a relatively large basin

Screenshots

[aaraney]

Once #744 this should no longer be an issue.

[stcui007]

Should have mentioned #744, but this gives more complete description with proposed solutions.

On Thu, Jul 18, 2024 at 12:27 PM Austin Raney @.***> wrote:

Once #744 https://github.com/NOAA-OWP/ngen/pull/744 this should no longer be an issue.

— Reply to this email directly, view it on GitHub https://github.com/NOAA-OWP/ngen/issues/857#issuecomment-2237128246, or unsubscribe https://github.com/notifications/unsubscribe-auth/ACA4SRMMRJREDBWY24Y3UDDZM73JPAVCNFSM6AAAAABLDB6R5SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZXGEZDQMRUGY . You are receiving this because you authored the thread.Message ID: @.***>

[SnowHydrology]

@aaraney Will #744 also enable a user to change the name of the output files?

[aaraney]

@SnowHydrology, ~not the output file name, the output directory name.~

Potentially? The design we have right now allows it, but that could change.

[SnowHydrology]

Thanks @aaraney. To clarify, the design would currently allow the user to change the output file names, but it may not in the future?

[aaraney]

@SnowHydrology, correct.

[stcui007]

Closed via#855.