BUG: Using an S3Store additional store in jobflow.yaml

[Andrew-S-Rosen]

I am trying to set up a MinIO bucket as an additional store in my jobflow.yaml. The format I have so far is below. However, this isn't sufficient because the S3Store takes an index as the first argument, which itself is a MongoStore pointing to the collection to use as an index for the data in S3/MinIO. Perhaps it's just not clear to me how to do this (in which case a docs update could help), but I don't think there is an easy way to have the index point to the docs_store.

JOB_STORE:
    docs_store:
      type: MongoStore
      database: rosen_mp
      host: mongodb07.nersc.gov
      port: 27017
      username: <>
      password: <>
      collection_name: jobflow_outputs
    additional_stores:
      data:
        type: S3Store
        index: ??? # here is the issue. how to point to the above MongoStore
        bucket: rosen
        s3_profile:
          aws_access_key_id: <>
          aws_secret_access_key: <>
        compress: true
        endpoint_url: https://minio.materialsproject.org
        s3_workers: 24

[utf]

Yeah, unfortunately, I don't think it is possible to use the simplified format to do this, as the shorthand only supports Store keyword arguments being strings, dicts, floats, or lists and not arbitrary python objects like other stores. We could add in shorthand specifically for S3Store. In the time being, the easiest would be to use the long form specification. I.e., the full yaml output from JobStore.as_dict.

[utf]

The easiest way to do this would be something like:

from monty.serialization import dumpfn

store = JobStore(...)  # construct job store object
jobflow_config = {"JOB_STORE": store}

dumpfn(jobflow_config, "jobflow.yaml")

[Andrew-S-Rosen]

That's what I figured. Thanks for the workaround! That should work!

[utf]

I'm going to close this for now as there is a workaround. But if it bothers you enough to want to submit a PR for this custom case then I'll happily accept it.