NSLS-II / sciprovenance

Files relating to the Provenance Driven LDRD project between NSLS-II and CSI
BSD 2-Clause "Simplified" License
1 stars 2 forks source link

WIP: add demo #10

Closed CJ-Wright closed 5 years ago

pavoljuhas commented 5 years ago

@CJ-Wright - I have put together a draft of ES setup for importing the analysis run_start document that you sent a while ago - see https://github.com/CJ-Wright/sciprovenance/compare/csi_demo...pavoljuhas:add-tomo-search.

This thus far defines a document mapping between the run-start fields and Elasticsearch and has the following output:

$ python dbes_tomo_example.py
Sample ES document:

{'_id': '38d93d78-33a7-4a83-a8fe-c132e1fc1c45',
 'analysis_stage': 'img_sinogram',
 'puid': ['fe416836-a19f-40fa-8d79-c80d4a3dbc65'],
 'uid': '38d93d78-33a7-4a83-a8fe-c132e1fc1c45',
 'usednodes': [{'ndfunc': 'img_sinogram',
                'ndtype': 'shed.translation.ToEventStream'},
               {'ndfunc': 'xpdtools.pipelines.tomo.conditional_squeeze',
                'ndtype': 'rapidz.core.map'},
               {'ndfunc': 'xpdtools.pipelines.tomo.sort_sinogram',
                'ndtype': 'rapidz.core.starmap'},
               {'ndfunc': 'xpdtools.pipelines.tomo.append_data',
                'ndtype': 'rapidz.core.accumulate'},
               {'ndtype': 'rapidz.core.zip'},
               {'ndfunc': 'tomopy.prep.normalize.minus_log',
                'ndtype': 'rapidz.core.map'},
               {'ndfunc': 'xpdtools.pipelines.tomo.reshape',
                'ndtype': 'rapidz.core.map'},
               {'ndfunc': 'data', 'ndtype': 'shed.translation.FromEventStream'},
               {'ndfunc': 'numpy.deg2rad', 'ndtype': 'rapidz.core.map'},
               {'ndfunc': 'data',
                'ndtype': 'shed.translation.FromEventStream'}]}

Would this work as an ES summary of the tomography pipeline?

pavoljuhas commented 5 years ago

For the demo let us disable warnings in notebook - for example by setting export PYTHONWARNINGS=ignore in parent shell.

pavoljuhas commented 5 years ago

@CJ-Wright - please pull to export kwargs to ES

git pull https://github.com/pavoljuhas/sciprovenance add-ndkwargs-field
pavoljuhas commented 5 years ago

@CJ-Wright - are you OK with merging this in? cc: @hjjvandam