Simulation Database

[CFGrote]

Investigate a possible architecture for a simulation database.

• [x] Take a look at nomad.
• [ ] Devise a workflow where data is split up into metadata (including the input parameters), raw data (sim. output), and reduced/analyzed data. Where can we deposit what?
• openpmd is a hierarchical metadata scheme, hence a hierarchical database "scheme" suggests itself, such as mongodb.
• [ ] Devise a pipeline that takes an openpmd dataset, converts to json and pushes to a mongodb instance. For testing, either run a self-hosted mongo server or sign up for atlas

More ideas welcome

[shervin86]

21/04/2020 Plan of the day

Study the MongoDB tutorials and see which features can be interesting for storing and sharing simulations

[shervin86]

Here's an example of openPMD file saved in JSON format provided by Zsolt.

Each field has a "datatype" value, that is useless and should be discarded.

   "attributes" : {
      "softwareVersion" : {
         "datatype" : "STRING",
         "value" : "4.8.3"
      },
      "basePath" : {
         "value" : "/data/%T/",
         "datatype" : "STRING"
      },
      "date" : {
         "datatype" : "STRING",
         "value" : "2020-04-27 14:12:50 +0200"
      },
      "software" : {
         "datatype" : "STRING",
         "value" : "EPOCH"
      },
      "iterationFormat" : {
         "value" : "/data/%T/",
         "datatype" : "STRING"
      },
      "particlesPath" : {
         "datatype" : "STRING",
         "value" : "particles/"
      },
      "author" : {
         "datatype" : "STRING",
         "value" : "Zsolt Lecz<zsolt.lecz@eli-alps.hu>"
      },
      "openPMD" : {
         "value" : "1.1.0",
         "datatype" : "STRING"
      },
      "iterationEncoding" : {
         "datatype" : "STRING",
         "value" : "groupBased"
      },
      "openPMDextension" : {
         "datatype" : "STRING",
         "value" : "BeamPhysics;SpeciesType"
      }

And here's how it should look like ideally:

   "attributes" : {
      "softwareVersion" : "4.8.3",
      "basePath" : "/data/%T/",
      "date" : "2020-04-27 14:12:50 +0200"
      "software" : "EPOCH",
      "iterationFormat" : "/data/%T/",
      "particlesPath" : "particles/"
      "author" : "Zsolt Lecz<zsolt.lecz@eli-alps.hu>",
      "openPMD" : "1.1.0",
      "iterationEncoding" : "groupBased",
      "openPMDextension" : "BeamPhysics;SpeciesType"
      }

So, in order to have an efficient storage in MongoDB, the json should be slimmed