search

NCAR / ParallelIO

A high-level Parallel I/O Library for structured grid applications
Apache License 2.0
136 stars 53 forks source link

Needs to be a way to set PIO_Save_Decomps to save decompositions to file #213

Closed edhartnett closed 7 years ago

edhartnett commented 8 years ago

There needs to be a way to set PIO_Save_Decomps. It used to be set by environment variable that is not compatible with the netCDF API. However, since it is useful, I would like to make it available to netCDF users.

We can make this available through the API in some way.

We could make the default a CMake option.

We could also automatically turn this on when the log level is set to non-zero. Would that be useful?

We can include backward support for the environment variable if that is required for current users. I would prefer not to, but it can be done if necessary. I can add a build option to turn off this environment var support and use that build option with netCDF.

jedwards4b commented 8 years ago

I think that the env variable is the best interface for this variable. It can potentially create a lot of files so I don't think that we want it set with log level.

Katetc commented 8 years ago

Yes, I do love the Log features that have been added, btw. But the Save Decomps would really pollute a directory so you should have to turn it on manually. It could be a nasty surprise otherwise.

edhartnett commented 8 years ago

How many files does it produce? Looks like one file per compute class - is that correct?

Katetc commented 8 years ago

For a simple CESM run (F case) I get more than 100 files. Many are duplicates, and the perl script ParallelIO/scripts/prune_decomps.pl gets rid of the duplicates. That usually reduces it down to 10-15 per active component.