generate ncWMS config for multiple ensembles

[corviday]

At present, the ncwms_configurator script collects data on all datafiles in a single ensemble and writes them to an ncWMS configuration file. Different portals use different ensembles, so we also maintain an ensemble called all_files, which is normally the one used to generate ncWMS configurations.

We have a lot of datafiles. At present, a freshly written configuration file from the all_files ensemble is a 12MB text file and takes over an hour for ncWMS to load on our fastest system. It would be nice if we could generate smaller configuration files that still were able to support multiple portals by being able to pass multiple ensembles in to ncwms_configurator.

[jameshiebert]

Seems like an OK idea. Taking a look at the script, it's definitely possible, but a little awkward because you have to supply the script with an output file as well. So then you have to coordinate the output file parameter (as a list) with the ensemble parameter (as a list).

Why not just script this from the command line?

for ensemble in all_files bc_prism some_other_ensemeble; do
   ncwms_configurator create -e ${ensemble} -o ${ensemble}.config
done

[corviday]

Sorry, I think my phrasing was unclear.

What would make working with ncWMS instances easier is the ability to create a single config file that lists all the files from multiple ensembles (ncWMS runs against a single config file at a time). For example, it would be nice to be able to easily generate a configuration file containing all data in the ce_files, extreme_precip, and p2a_files ensembles (7857 files). An ncWMS able to serve all those files could support all PCEX's portals and whatever we end up doing with Plan2Adapt.

At present, I use the all_files ensemble (15316 files) to configure the pcex/p2a ncWMS server. Which is certainly a workable system, but restarting ncWMS takes twice as long as it could if it was serving exactly the files I needed, and once in a while the admin page times out when I try to load it sometimes (this won't affect users, just developers using the admin page to test map generation before making maps accessible to users on a live server).

This is a low-priority issue. It sometimes impacts developers setting up ncWMS access or using the admin interface to test map generation, but as far as I know it does not impact users.

Possible solutions:

• extend ncwms_configurator to generate a multi-ensemble config file
• create a new ensemble with all the ce&p2a files
• continue as we have been
• write a kludgy script that merges config files, dropping duplicates

[jameshiebert]

I have a (completely untested) branch that I think addresses this issue. I realize that this is low priority, but if you want to take a look at this and take it to completion, have at it :)