Use json for input files

[mrp089]

Generate an input file that directly contains the Python structure of the 0D solver

See for example https://github.com/Autodesk/nanodesign/tree/dev/tests/samples

[ktbolt]

@JonathanPham The Python json module is good for reading/writing Python data structures. Have a look here https://github.com/Autodesk/nanodesign/blob/dev/nanodesign/converters/viewer/writer.py to get an idea how it works.

[mrp089]

@JonathanPham Can you parse a sample simulation setup with json and post the output here?

[JonathanPham]

From stackoverflow, in order to save a Python object (e.g. a 0D element) to a json file, we need to convert that object into a dict, where the fields of that object will be stored as the keys / values of the dict. But those fields have to be converted into strings and we can't do that with numpy functions (or any function) easily. By numpy function, I mean like np.abs() in StenosisBlock or cubic spline functions (for pulsatile inflows in the inlet block)

[ktbolt]

@JonathanPham Complex data structures like custom objects and numpy objects are not serializable. You will need to use jsonpickle https://github.com/jsonpickle/jsonpickle I think.

[mrp089]

Could we do an intermediate step between "text input file" and "exporting the whole model with json"? We could store all the information we have in the text input file but store it in a more "pythonic" format, e.g.

model['nodes'][0] = [0.0, 0.0, 0.1]
...
model['joint'][0] = {'in': 0, 'out': [1, 2, 3]}
...
model['element'][0] = {'nodes' : [0, 1], 'length': 1.0, 'resistance': 13.37, 'capacitance': 42.0, 'stenosis': 0.1}
...
model['inflow'] = [[0.0, 10.0], [1.0, 10.0]]
model['bcs'] = {...}
model['solver'] = {...}

The downside is that we would have to manually define a way to store this information. However, we already have the 0D and 1D model in more or less this dictionary structure in SimVascular in Python/site-packages/sv_rom_simulation. (We currently proceed to translate that structure into a text file.)

On the upside, we would still have something that's human-readable and can be handled with json. At the same time, it's also flexible and easily expandable. What do you think, @JonathanPham @ktbolt?

[JonathanPham]

@mrp089 The method you suggested would reduce the number of file I/O operations that we would have to do, because we wouldn't need to save the dictionary structure in SimVascular in Python/site-packages/sv_rom_simulation to a text file and later read that same text file in svzerodsolver/solver.py to create the 0d model. We can save this dictionary structure to a json file. Sounds like a good idea.

If we use this method, I don't think we will have to "export the whole 0d model to json". I think the main reason why we wanted to export the 0d model to json was to reduce the number of file I/O operations, but these would already be reduced if we use your approach. If we export the whole 0d model to json, then we would probably have to use jsonpickle as @ktbolt suggested, but then users would have to install jsonpickle in order to use our 0d solver. Probably simpler to not require that (less package dependencies to deal with)...

[mrp089]

Awesome, then let's do that! Can you provide a sample (current) input file together with the dictionary structure you'd like to have? Then I'll modify the export accordingly.

[ktbolt]

What is the difference between "text input file" and "exporting the whole model?

I thought the goal was to just store the input data for the solver? So you just need to store some numpy arrays, float values and string values I think, like @mrp089 example. For example

{
"mesh":  {
    "coordinates":  [ (1.0, 0.0, 0.0), (2.0,0.0,0.0) ]
    "elements":  [ { "nodes": (1,2), "length":3.0 },  { "nodes": (0,1), "length":4.0 } ]
    }
}

You can extract data from the JSON file by just walking though the data structures which don't need to match anything in the solver code.

[mrp089]

Originally, we had this idea where we store the whole Python object structure of the 0D model with json. But it looks like that will be too complicated and maybe not even that useful.

[ktbolt]

It's probably not a good idea to store the whole Python object structure of the 0D model, this could expose implementation details that could change. The JSON file should be easy to understand so that users could generate their own file.

[JonathanPham]

@ktbolt Sounds good, we won't be storing the whole Python object structure of the 0D model to a json file. Instead, we'll just save the contents of the original 0D input file in a more pythonic format, as @mrp089 suggested.

@mrp089 attached is a 0D input file and its related json file. When I read the 0D input file in svzerodsolver.solver.py, I save the information to a dictionary called parameters. To create the attached json file, I simply exported this dict using json.dump(parameters, outfile). Would it be possible for you, @mrp089, to have SimVascular output a similar json file? Then I can read this json file in svzerodsolver.solver.py directly, using

with open(json_file, 'r') as f:
   parameters = json.load(f)

0076_1001_0d.zip

parameters has a messy structure right now, but in the future, we could update it to have a neater format. Would need to update svzerodsolver.solver.py to use the new format

[mrp089]

@JonathanPham Thanks! A couple of things:

1. Why is the inflow sorted as [t0, f0, t1, f1, ..., tn, fn] and not [[t0, t1, ..., tn], [f0, f1, ..., fn]]? Can json not handle arrays of arrays? If that's the case, we should sort it with dictionary keys (or just do that anyways).
2. What are the 0.0s in the RCR BCs? Is it for time-dependent RCRs? If that's the case, I think there should be a better format (see 1). However, I've never seen time-dependent RCR (correct me if I'm wrong) and I'd be happy to remove that alltogether.

Thanks!

[ktbolt]

Wouldn't inflow be better stored as [ [t0,f0], [t1,f1], ... ] ? I assume this is a 2D NumPy array so you can just write it using ndarray.tolist().

[mrp089]

@JonathanPham Here are my input files with new centerlines (after https://github.com/SimVascular/SimVascular/commit/19e6fb6f26600da0cbc99def4a93dcf377ddc7ac): 0d.zip, 1d.zip.

Can you provide the json files with

with open(file_name, 'w') as file:
    json.dump(inp, file, indent=4, sort_keys=True)

Thank you!

[JonathanPham]

@mrp089 Here are the json files corresponding to the 0d input files you sent me: json_0d.zip

[mrp089]

Thanks! I'm able to generate these now: https://github.com/mrp089/SimVascular/tree/0d_json_920

Next step: make it nicer! @JonathanPham Can you derive a new json format that follows the following guidelines:

• no redundant information e.g. joint_outlet_segments, joint_inlet_segments are already in joint_name_to_segments_map
• dictionaries whenever possible to make input clearer e.g. {'Rp': 10.0, 'C': 0.01, 'Rd': 20.0} instead of {'RCR': [10.0, 0.01, 20.0]}
• don't include "none" values e.g. NOBOUND, NONE
• remove unused features e.g. segment-dependent fluid properties (make constant instead), segment_tangent_vectors, lengths
• remove unused names (use IDs if necessary) e.g. joints J0, inlets IN0, outlets OUT0

I already flagged a bunch of stuff as (presumably) UNUSED: https://github.com/mrp089/SimVascular/blob/77e8e8546a9990d3ecca4fd0d5de86138a15d355/Python/site-packages/sv_rom_simulation/io_0d.py#L99 https://github.com/mrp089/SimVascular/blob/77e8e8546a9990d3ecca4fd0d5de86138a15d355/Python/site-packages/sv_rom_simulation/io_0d.py#L113-L120 https://github.com/mrp089/SimVascular/blob/77e8e8546a9990d3ecca4fd0d5de86138a15d355/Python/site-packages/sv_rom_simulation/io_0d.py#L144-L156

[JonathanPham]

@mrp089 Sounds good. I'll create new json files following the guidelines you provided and then send them over to you.

[JonathanPham]

@mrp089 Here are the updated json files for the 0d models, following the guidelines you provided.

Note that the lengths is still in the dicts because we need them to determine the distance along the centerlines for our 0d results file (#15).

json_0d.zip

[mrp089]

That looks awesome @JonathanPham!! I'll modify my code accordingly.

Do you have an svZeroDSolver branch that can read those input files? Then I'll test the whole loop once I'm done.

[mrp089]

One minor edit: I added density and viscosity to the input file. I know it's always the same, but we let people in the GUI choose it so we should pass it on to the solver.

[ktbolt]

@JonathanPham The JSON files look like a dump of internal data structures, will look like a bunch of unrelated data to a user, also seems verbose

    "lengths": {
        "0": 1.855960624456018,
        "1": 1.0243020880514946,
         ...
        "13": 2.545517257688754
    },

    "segment_0d_types": {
        "0": "STENOSIS",
        "1": "STENOSIS",
       ...
        "13": "STENOSIS"
    },

    "segment_0d_values": {
        "0": {
            "R_poiseuille": 0.37021831189620685,
            "stenosis_coefficient": 5.208655764149232e-06
        },
        "1": {
            "R_poiseuille": 0.24008330447702383,
            "stenosis_coefficient": 3.91920639741218e-05
        },
    ...
        "13": {
            "R_poiseuille": 58.968569740744954,
            "stenosis_coefficient": 0.0
        }

where IDs [0, 1, ..., 13] are repeated in several data structures.

Let's group data into objects that are more descriptive of the problem. For example

{

"number_of_segments": 4

"segements": [
    { "id": 1, 
       "length": 1.855960624456018, 
        "type": "STENOSIS",
        "name": "branch0_seg0"
    },
 ]

"boundary_conditions": {

   "inlet": [ 
      {  "type": "FLOW",
         "data": [ (0.0, 3.841951), (0.001, 3.928484) ]
     }
  ]

   "outlet": [ 
      {  "type": "RCR",
         "name": "RCR_0",
         "id": 6,
         "data": { "C": 7.125000000000001e-05, "Pd": 0, "Rd": 13858.0, "Rp": 885.0 }
     }

      {  "type": "RCR",
         "name": "RCR_1",
         "id": 10,
         "data":  { "C": 0.00029221, "Pd": 0.0, "Rd": 3122.0, "Rp": 271.0 } 
      }
   ]

  }

Create a Python class that reads/writes JSON files. Make sure to include error handling using exceptions. We would like to call this from the SV Python API.

Note that you might could use a template to check the input data, something like

RCR_BC = { 'type':str, 'name':str, 'id':int, 'data': dict }
RCR_DATA = { "C": float, "Pd": float, "Rd": float, "Rp": float }

[JonathanPham]

@ktbolt Good idea. I'll go ahead and update the current json format to use the more descriptive format you suggested.

@mrp089 Can you hold off on updating SimVascular until I finish making these new changes?

[mrp089]

Sounds good! @JonathanPham Please use these legacy input files for testing (using the current SV master): 0d.zip

[JonathanPham]

@mrp089 Here are the json files, using the new format, corresponding to the legacy 0d input files you sent previously. These new json files are similar to the ones recommended by @ktbolt, but they are not quite the same.

After you make the changes in SimVascular, you can test out the new 0d json input files using my branch. To use this branch, you'll need to set the use_steady_soltns_as_ics option to False because I haven't updated the initial conditions section to use the new json file yet.

zero_d_json.zip

[mrp089]

Thanks, @JonathanPham! I'll look at it Thursday afternoon. Let me know if they're any changes.

[JonathanPham]

Not sure where to where post this issue, so I'll post it here.

@mrp089 I just noticed that the coronary boundary conditions in the 0d text input files that you sent me in a previous comment are not periodic. I think this might be an issue, because in the 0d solver, we always assume that pulsatile boundary conditions are periodic (meaning that the endpoints of the time series are the same).