xpdAcq / xpdAn

Analysis Pipelines and Tools for XPD

http://xpdacq.github.io/xpdAn/

Other

2 stars 10 forks source link

WIP: write callbacks for exporting data (including to a remote db) #228

Closed CJ-Wright closed 5 years ago

CJ-Wright commented 6 years ago

This would allow us to export our data to a drive automatically, and export our data to an external db automatically (assuming that we can work out the method of getting a connection to our db)

CJ-Wright commented 6 years ago

Needs tests

codecov-io commented 6 years ago

Codecov Report

Merging #228 into master will increase coverage by 0.76%. The diff coverage is 93.52%.

@@            Coverage Diff             @@
##           master     #228      +/-   ##
==========================================
+ Coverage   85.07%   85.83%   +0.76%     
==========================================
  Files          32       32              
  Lines        1360     1497     +137     
==========================================
+ Hits         1157     1285     +128     
- Misses        203      212       +9

Impacted Files	Coverage Δ
xpdan/tests/test_callbacks.py	`100% <100%> (ø)`	:arrow_up:
xpdan/tests/conftest.py	`98.59% <100%> (ø)`	:arrow_up:
xpdan/callbacks.py	`92.65% <91.17%> (-2.02%)`	:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 700f871...cdc112b. Read the comment docs.

CJ-Wright commented 6 years ago

We might consider using the databroker conftests here, so we have access to all the databroker backends. Note that the remote export is not tested, I could add a test without an ssh prefix I guess. I may need to just test this by hand and see how we do.

CJ-Wright commented 5 years ago

This needs to move to rapidz

sbillinge commented 5 years ago

Thanks for your lonely efforts on this. I don't understand it well enough to give you any kind of meaningful comments. Have you discussed with the DAMA guys about plusses and minusses of different transfer methods (rsync vs sthg else). It would sure be nice to be one and done with this.

CJ-Wright commented 5 years ago

We've discussed it in the past and rsync seems the way to go. Testing is a bit of a pain though so I may split this up into two PRs to separate the standard copy from the rsync copy. This way I can test the rsync copy by hand.

CJ-Wright commented 5 years ago

This is passing on the default branch, I need to fix orch to get it working for the bleeding edge branch.

CJ-Wright commented 5 years ago

I fixed orch, but this is still working on the pre-chunked pipelines so the bleeding edge xpdtools causes it to fail. Merging.