search

bird-house / flyingpigeon

WPS processes for climate model data, indices and extreme events
http://flyingpigeon.readthedocs.io/en/latest/
Apache License 2.0
19 stars 15 forks source link

Process pointinspection and Documentation #271

Closed nilshempelmann closed 5 years ago

nilshempelmann commented 5 years ago

@cehbrecht I need help with the workdir settings. If I get it right it needs to be updated in all the eggshell functions as well, since there are many files produced there.

@huard @Zeitsperre If you have time check the Subset Process Documentation.

nilshempelmann commented 5 years ago

missed to commit changes in eggshell

cehbrecht commented 5 years ago

travis and codacy are not happy yet. You could also add a test.

cehbrecht commented 5 years ago

I need help with the workdir settings. If I get it right it needs to be updated in all the eggshell functions as well, since there are many files produced there.

I haven't checked the eggshell ... but it should provide supporting functions not knowing about self.workdir handling ... that is pywps only. So, you need to find the right set of function parameters.

nilshempelmann commented 5 years ago

@huard agree with netcdf outputs. How to configure the output if the number of outputs can vary? Can you point me to an example?

huard commented 5 years ago

@nilshempelmann Do you mean if the number of geometries is larger than one, or if the number of input files is larger than one ? I'll assume it's the former, and then OCGIS supports it out of the box, you just pass a list of geometries to OCGIS and make sure you specify aggregate=True. The idea is to create another dimension for the output where the index is the gemetry index. This allows you to store the lat, lon as well along this dimension. https://github.com/NCPP/ocgis/pull/459 https://ocgis.readthedocs.io/en/latest/appendix.html?highlight=geom_dim#netcdf-output

nilshempelmann commented 5 years ago

@huard The number of netCDF output files vary if input files are model results from different GCMs-RCMs. Input files can be belonging to different modelruns, the process are sorting them and process all different models separatly eggshell.nc.nc_utils.sort_by_filename(resource, historical_concatination=False)
(mulitiple files of the same Modelrun are currently aggregated, including also aggregation of historical and suitable rcp run of the same Model output). Its not recommendet to store multiple models into one netcdf file.

And yes multiple geometries / optional unify results in variation of the number of outputs. So the current solution was to archive all of them and output one singe archive.

We also should keep the DRS-nameing logig and respect the CMIP/Cordex archive standarts. Here they separate the geometries/domains in seperate files. eggshell.nc.nc_utils.drs_filename()

A following process (e.g. visualisation) starts with archive_extract, in case of input=tarfile.

If you have a solution to avoid the archive-tar would be nice.

huard commented 5 years ago

I understand now. You are saying the the current output is a tar file, which allows us to bundle multiple output files together in one single ComplexOutput.

I think now is the time to implement a generic approach for multiple file output using the MetaLink standard.

See https://github.com/geopython/pywps/issues/298 https://github.com/bird-house/emu/issues/64#issuecomment-442547939

I suggest we first test and discuss it in Emu then move the solution to this process.

nilshempelmann commented 5 years ago

And here comes the issue https://github.com/bird-house/flyingpigeon/issues/273 :-)

nilshempelmann commented 5 years ago

Following processes are sucessfully running, but test are failing (related https://github.com/bird-house/emu/issues/70): Subset_countries subset_continets pointinspection

@huard need your help to get the oranus subset processes in place. https://github.com/bird-house/flyingpigeon/issues/274

nilshempelmann commented 5 years ago

@cehbrecht Finally :-) But subset countries and continents output path needs to be fixed in the tests (disabled for the moment)

nilshempelmann commented 5 years ago

@cehbrecht : commented your change request. Feel free to update the Ouranos processes