materialsproject / jobflow

jobflow is a library for writing computational workflows.
https://materialsproject.github.io/jobflow
Other
90 stars 24 forks source link

Allow SSHTunnel in job stores #477

Closed mjwen closed 9 months ago

mjwen commented 10 months ago

Summary

Allow the use of SSHTunnel in job stores. This will make it possible to use a jobflow.yaml like the below

  docs_store:
    type: MongoStore
    ssh_tunnel:
      type: SSHTunnel
      ...
  additional_stores:
    data:
      type: S3Store
        ...
      index:
        ...
      s3_profile:
        ...
      ssh_tunnel:
        type: SSHTunnel
        ...

MongoStore already have ssh tunnel support. Support for S3Store is coming from this PR https://github.com/materialsproject/maggma/pull/882

The SSHTunnel class is simply added to the all_stores variable. Technically, all_stores no longer only contains stores. So, we might want to change the variable name.

mjwen commented 10 months ago

Test fails because SSHTunnel is relocated in this PR https://github.com/materialsproject/maggma/pull/882

Once that is merged, we should be good.

codecov[bot] commented 9 months ago

Codecov Report

Merging #477 (e8fa70c) into main (43f8463) will increase coverage by 0.00%. The diff coverage is 100.00%.

Additional details and impacted files ```diff @@ Coverage Diff @@ ## main #477 +/- ## ======================================= Coverage 99.86% 99.86% ======================================= Files 20 20 Lines 1510 1512 +2 Branches 416 416 ======================================= + Hits 1508 1510 +2 Misses 2 2 ``` | [Files](https://app.codecov.io/gh/materialsproject/jobflow/pull/477?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=materialsproject) | Coverage Δ | | |---|---|---| | [src/jobflow/core/store.py](https://app.codecov.io/gh/materialsproject/jobflow/pull/477?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=materialsproject#diff-c3JjL2pvYmZsb3cvY29yZS9zdG9yZS5weQ==) | `100.00% <100.00%> (ø)` | |
utf commented 9 months ago

Thanks!