UI ingestion with multiple model-graphs

[kityansiu]

We have a need to ingest data that spans multiple model-graphs. Right now the UI will give the following error:

Here's an example of the yaml files that are in this ingestion package. In ADEPT/InstanceData/import.yaml, I have the following:

data-graph: http://rack001/adeptdata
model-graphs:
- http://rack001/model
- http://rack001/mbsemodel
ingestion-steps:
- class: http://arcos.rack/PROV-S#ACTIVITY
  csv: ingest_ACTIVITY.csv
- class: http://arcos.rack/mbse#Engine
  csv: ingest_Engine.csv
- class: http://arcos.rack/mbse#Measurement
  csv: ingest_DesignedPartMeasurement.csv
- class: http://arcos.rack/mbse#Assembly
  csv: ingest_DesignedAssembly.csv
- class: http://arcos.rack/mbse#DesignedPart
  csv: ingest_DesignedPart.csv
- class: http://arcos.rack/REQUIREMENTS#REQUIREMENT
  csv: ingest_REQUIREMENT.csv
- class: http://arcos.rack/mbse#ManufacturedPart
  csv: ingest_Turbine.csv
- class: http://arcos.rack/mbse#Measurement
  csv: ingest_AirfoilMeasurements.csv
- class: http://arcos.rack/mbse#ManufacturedPart
  csv: ingest_Airfoil.csv

And in ADEPT/manifest.yaml, I have the following:

name: 'ADEPT Engine'
description: 'MBE Demo'
footprint:
  model-graphs:
  - http://rack001/model
  - http://rack001/mbsemodel
  data-graphs:
  - http://rack001/adeptdata
steps:
  - data: InstanceData/import.yaml

As an example of desired behavior, I can run rack manifest import manifest.yaml which correctly ingests the data into the data-graph http://rack001/adeptdata while correctly referencing both model-graphs.

[weisenje]

Currently we limit both model manifests and data manifests to a single model graph. To support this use case, I plan to make data manifests (but not model manifests) support multiple model graphs.

You mentioned that rack manifest import manifest.yaml gives the desired behavior. Is it possible that you are using an old version of CLI? My understanding is that the CLI uses the same ingestion routine as the RACK UI, and therefore should give the same error. Further, I believe that rack manifest import now only accepts ingestion packages (zip file), not yaml files.

Also - are you able to give me access to this ingestion package, for testing?

@glguy please comment if needed

[glguy]

In the CLI the model graphs field is defined to accept a list

https://github.com/ge-high-assurance/RACK/blob/8514558e78ca00adfa2a4d570def97cb8668ee0c/cli/rack/__init__.py#L136

https://github.com/ge-high-assurance/RACK/blob/8514558e78ca00adfa2a4d570def97cb8668ee0c/cli/rack/__init__.py#L593

So what Kit is asking for should work there.

[glguy]

The current version of the CLI shouldn't support loading manifest yaml files directly, but it can load single data.yamls

[weisenje]

Thanks @glguy

@kityansiu I have completed the code changes for this feature, but I would like to test it with your ingestion package first before updating RACK. Please point me to it if/when you can. Thanks!

[weisenje]

@kityansiu: this is fixed and pushed to RACK. Please test on your ingestion package when you get a chance.

@glguy: per discussion above, I noticed that the Wiki CLI “rack manifest import” instructions are out of date (still lists parameter as YAML file vs zip file) – you may want to update this

@weisenje: for future reference, keeping model manifest as single model as discussed here

[kityansiu]

I tested this and verified that it's working.

[kityansiu]

The current version of the CLI shouldn't support loading manifest yaml files directly, but it can load single data.yamls

You're right. I was using RACK v12.0 when I ran into this issue.

@glguy: per discussion above, I noticed that the Wiki CLI “rack manifest import” instructions are out of date (still lists parameter as YAML file vs zip file) – you may want to update this

@glguy, can you update the Wiki and the CLI help?

[glguy]

I've updated the wiki to match to documentation in README

[kityansiu]

usage: rack manifest import [-h] [--clear] [--optimize | --no-optimize]
                            [--optimize-url OPTIMIZE_URL]
                            config

positional arguments:
  config                Manifest YAML file
                            ^^^^^^

How about this here?

[glguy]

Well then, I guess the wiki wasn't so much out of date but the tool was... fixing