search

jhamman / chains

A Snakemake workflow for the chain of models approach to hydrologic modeling
Apache License 2.0
2 stars 1 forks source link

Test case broken: file concatenation preprocessing #1

Open gutmann opened 5 years ago

gutmann commented 5 years ago

I've tried to run the test case described in the storylines workflow and I get an error I don't understand : 

$ snakemake hydrology_models --configfile ../storylines_test_data/test_config.yml
... a whole bunch of good output I can paste if you want...
Error in rule downscaling:
    jobid: 0
    output: /glade/scratch/gutmann/storylines/test_data/CNRM-CM5/hist/bcsd/downscaling_data/downscale.CNRM-CM5.hist.bcsd.nc
    log: /glade/scratch/gutmann/storylines/test_data/CNRM-CM5/hist/bcsd/logs/downscale.CNRM-CM5.hist.bcsd.20190312-145812.log.txt

RuleException:
ValueError in line 67 of /gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile:
cannot broadcast shape (6, 2) to shape (365, 6, 365)
  File "/gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile", line 176, in __rule_downscaling
  File "/gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile", line 67, in process_downscaling_dataset
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/backends/api.py", line 658, in open_mfdataset
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 552, in _auto_combine
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 474, in _combine_nd
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 492, in _auto_combine_all_along_first_dim
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 510, in _auto_combine_1d
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 510, in <listcomp>
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 368, in _auto_concat
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 122, in concat
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 307, in _dataset_concat
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/variable.py", line 1978, in concat
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/combine.py", line 300, in ensure_common_dims
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/variable.py", line 1188, in set_dims
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/xarray/core/duck_array_ops.py", line 46, in f
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/site-packages/dask/array/core.py", line 3228, in broadcast_to
  File "/glade/u/home/gutmann/anaconda3/envs/storylines/lib/python3.6/concurrent/futures/thread.py", line 56, in run
Exiting because a job execution failed. Look above for error message
Shutting down, this might take some time.
Exiting because a job execution failed. Look above for error message
An error occurred
Complete log: /gpfs/fs1/work/gutmann/storylines/chains/.snakemake/log/2019-03-12T145813.750351.snakemake.log

I copied over the test case storylines_test_data to my work dir, and changed the test_config.yml working dir from flash to a directory in my scratch space. The key error seems to be ValueError in line 67 of /gpfs/fs1/work/gutmann/storylines/chains/downscaling.snakefile: cannot broadcast shape (6, 2) to shape (365, 6, 365) line 67 is : 

    ds = xr.open_mfdataset(sorted(input_files),
                           preprocess=preproc,
                           engine='netcdf4').load()

Looking at the files, all seem to have a primary variable that is (time, lat, lon) and all seem to have the same lat, lon, bounds dimensions. All bounds_lat, bounds_lon variables seem to have the correct dimensions. The broadcast error above looks like there is a file with bounds_latitude(time, latitude, time), but I don't see anything like that. Any ideas?

gutmann commented 5 years ago

Removing preprocess=preproc seems to fix this... but of course then it isn't casting to a float and will likely need more memory. Is there a better way to do that in xarray now?

Very simply breaking case : 


import xarray as xr

files = ['/gpfs/fs1/work/jhamman/storylines/storylines_test_data/downscaling/bcsd/cnrm-cm5_rcp45_r1i1p1/conus_c5.cnrm-cm5_rcp45_r1i1p1.daily.pr.1950.nc', 
            '/gpfs/fs1/work/jhamman/storylines/storylines_test_data/downscaling/bcsd/cnrm-cm5_rcp45_r1i1p1/conus_c5.cnrm-cm5_rcp45_r1i1p1.daily.pr.1951.nc']

# This works fine
ds = xr.open_mfdataset(files)

def cast_to_float(ds): 
    return ds.astype(np.float32) 

preproc = cast_to_float

# this breaks
ds = xr.open_mfdataset(files, 
                    preprocess=preproc, 
                    engine='netcdf4').load() 

Causes the same Broadcast ValueError crash