emerald-beryl-pipeline
Luigi based pipeline to run inversions using our simplified SimPEG wrapper
Installation
Clone this repo, then run pip install .[all]. Note the [all].
Without this, emerald-processing-em and simpeg/simpleem won't be
installed, and you'll have to install some version of these manually.
Documentation
- Documentation on how to run all the pipelines from anotebook
- Documentation on how to extract the API documentation used to generate the front end UI
- This should be used to test any changes / additions to e.g. processing pipeline filters
Running from the command line
To run this pipeline locally:
luigi --module beryl_pipeline.inversion Inversion --inversion-name=file:///some/temp/dirThis assumes you've copied docs/example-real.yml to /some/temp/dir/config.yml.
Cloud computing support
All the pipeline tools use the luigi OpenerTarget, which allows for accessing files on a host of file storage options, including gcs and s3 by just using the appropriate URL schemes.
Note: There are two types of parallelization utilized by this pipeline system: Multiple luigi tasks can be run in parallel if multiple workers are supplied, potentially using multiple computers. Additionally, SimPEG inversions are parallelized using threading, and can utlize any number of CPU:s available on a single computer.
Unit tests
To run the unit tests, first pip install nose2 and then run
nose2 -s tests