search

ge-high-assurance / RACK

DARPA's Automated Rapid Certification of Software (ARCOS) project called Rapid Assurance Curation Kit (RACK)
BSD 3-Clause "New" or "Revised" License
20 stars 6 forks source link

Run ./setup-rack.sh as part of release process #645

Closed glguy closed 2 years ago

glguy commented 2 years ago

It seems to me that we should run cli/setup-rack.sh in workflows/actions/download/action.yml so that every fresh started RACK is ready to go with all the baseline RACK ingestion nodegroups.

Is there any reason we aren't already doing this? Is there an important RACK use-case that doesn't have these nodegroups?

Pinging @tuxji @russell-d-e in case either of you knows why we don't do this already.

tuxji commented 2 years ago

We already call the setup-rack script when building a rack box. We just call it from another place:

rack-box/scripts/install.sh
154:./setup-rack.sh

What makes you think a freshly started RACK doesn't have the baseline nodegroups?

glguy commented 2 years ago

@tuxji I've recently had conversations with Abha and Kevin where ingestion of data failed for them. I suggested adding ./setup-rack.sh to their ingestion process just before ./setup-arcos.sh and that fixed the missing nodegroup errors they were having.

I suppose I then leapt to the conclusion that we weren't running that in the setup process. Since you mention that we are I guess some deeper investigation is in order. I'll have to try and reproduce their results today.

tuxji commented 2 years ago

@glguy I checked the output from the last dev build (went to https://github.com/ge-high-assurance/RACK/runs/4942731217?check_suite_focus=true, expanded Build, expanded Build rack box Docker image) and I couldn't see anything out of the normal, but here's part of the output in case it helps:

    docker: Running hooks in /etc/ca-certificates/update.d...
    docker:
    docker: done.
    docker: done.
    docker: Adding user `ubuntu' ...
    docker: Adding new group `ubuntu' (1000) ...
    docker: Adding new user `ubuntu' (1000) with group `ubuntu' ...
    docker: Creating home directory `/home/ubuntu' ...
    docker: Copying files from `/etc/skel' ...
    docker: Adding system user `fuseki' (UID 105) ...
    docker: Adding new group `fuseki' (GID 106) ...
    docker: Adding new user `fuseki' (UID 105) with group `fuseki' ...
    docker: Not creating home directory `/home/fuseki'.
    docker: /home/ubuntu/semtk-opensource /home/ubuntu/semtk-opensource
    docker: Returning (already ran .env)
    docker: /home/ubuntu/semtk-opensource
    docker: Web build ID=built-a6d915045b8a-2022-01-25-201553
    docker: sparqlGraph/main-oss/sparqlgraphconfigOss.js
    docker: sparqlGraph/main-oss/KDLEasyLoggerConfigOss.js
    docker: sparqlForm/main-oss/sparqlformconfig.js
    docker: sparqlForm/main-oss/KDLEasyLoggerConfig.js
    docker: ./updateWebapps.sh done
    docker: Waiting for Fuseki at http://localhost:3030...
    docker: Waiting for nodeGroupService at http://localhost:12059...
    docker: Processing ./wheels/Pillow-9.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    docker: Processing ./wheels/PyYAML-5.4-cp38-cp38-manylinux1_x86_64.whl
    docker: Processing ./wheels/ase-3.21.1-py3-none-any.whl
    docker: Processing ./wheels/attrs-19.3.0-py2.py3-none-any.whl
    docker: Processing ./wheels/certifi-2020.6.20-py2.py3-none-any.whl
    docker: Requirement already satisfied: chardet==3.0.4 from file:///home/ubuntu/RACK/cli/wheels/chardet-3.0.4-py2.py3-none-any.whl in /usr/lib/python3/dist-packages (3.0.4)
    docker: Processing ./wheels/colorama-0.4.3-py2.py3-none-any.whl
    docker: Processing ./wheels/cycler-0.11.0-py3-none-any.whl
    docker: Processing ./wheels/fonttools-4.29.0-py3-none-any.whl
    docker: Processing ./wheels/idna-2.10-py2.py3-none-any.whl
    docker: Processing ./wheels/importlib_metadata-1.7.0-py2.py3-none-any.whl
    docker: Processing ./wheels/jsonschema-3.2.0-py2.py3-none-any.whl
    docker: Processing ./wheels/kiwisolver-1.3.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
    docker: Processing ./wheels/matplotlib-3.5.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
    docker: Processing ./wheels/numpy-1.22.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    docker: Processing ./wheels/packaging-21.3-py3-none-any.whl
    docker: Processing ./wheels/plotly-4.14.3-py2.py3-none-any.whl
    docker: Processing ./wheels/pyparsing-3.0.7-py3-none-any.whl
    docker: Processing ./wheels/pyrsistent-0.16.0-cp38-cp38-linux_x86_64.whl
    docker: Processing ./wheels/python_dateutil-2.8.1-py2.py3-none-any.whl
    docker: Processing ./wheels/rack-0.0.0-py3-none-any.whl
    docker: Processing ./wheels/requests-2.25.1-py2.py3-none-any.whl
    docker: Processing ./wheels/retrying-1.3.3-py3-none-any.whl
    docker: Processing ./wheels/scipy-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
    docker: Processing ./wheels/semtk_python3-0.0.1-py3-none-any.whl
    docker: Processing ./wheels/setuptools-60.5.0-py3-none-any.whl
    docker: Processing ./wheels/six-1.15.0-py2.py3-none-any.whl
    docker: Processing ./wheels/tabulate-0.8.7-py3-none-any.whl
    docker: Processing ./wheels/urllib3-1.26.5-py2.py3-none-any.whl
    docker: Processing ./wheels/zipp-3.1.0-py3-none-any.whl
    docker: Installing collected packages: Pillow, PyYAML, numpy, scipy, cycler, pyparsing, packaging, fonttools, kiwisolver, six, python-dateutil, matplotlib, ase, attrs, certifi, colorama, idna, zipp, importlib-metadata, setuptools, pyrsistent, jsonschema, retrying, plotly, rack, urllib3, requests, semtk-python3, tabulate
    docker:   Attempting uninstall: six
    docker:     Found existing installation: six 1.14.0
    docker:     Not uninstalling six at /usr/lib/python3/dist-packages, outside environment /usr
    docker:     Can't uninstall 'six'. No files were found to uninstall.
    docker:   Attempting uninstall: certifi
    docker:     Found existing installation: certifi 2019.11.28
    docker:     Not uninstalling certifi at /usr/lib/python3/dist-packages, outside environment /usr
    docker:     Can't uninstall 'certifi'. No files were found to uninstall.
    docker:   Attempting uninstall: idna
    docker:     Found existing installation: idna 2.8
    docker:     Not uninstalling idna at /usr/lib/python3/dist-packages, outside environment /usr
    docker:     Can't uninstall 'idna'. No files were found to uninstall.
    docker:   Attempting uninstall: setuptools
    docker:     Found existing installation: setuptools 45.2.0
    docker:     Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
    docker:     Can't uninstall 'setuptools'. No files were found to uninstall.
    docker:   Attempting uninstall: urllib3
    docker:     Found existing installation: urllib3 1.25.8
    docker:     Not uninstalling urllib3 at /usr/lib/python3/dist-packages, outside environment /usr
    docker:     Can't uninstall 'urllib3'. No files were found to uninstall.
    docker:   Attempting uninstall: requests
    docker:     Found existing installation: requests 2.22.0
    docker:     Not uninstalling requests at /usr/lib/python3/dist-packages, outside environment /usr
    docker:     Can't uninstall 'requests'. No files were found to uninstall.
    docker: Successfully installed Pillow-9.0.0 PyYAML-5.4 ase-3.21.1 attrs-19.3.0 certifi-2020.6.20 colorama-0.4.3 cycler-0.11.0 fonttools-4.29.0 idna-2.10 importlib-metadata-1.7.0 jsonschema-3.2.0 kiwisolver-1.3.2 matplotlib-3.5.1 numpy-1.22.1 packaging-21.3 plotly-4.14.3 pyparsing-3.0.7 pyrsistent-0.16.0 python-dateutil-2.8.1 rack-0.0.0 requests-2.25.1 retrying-1.3.3 scipy-1.7.3 semtk-python3-0.0.1 setuptools-60.5.0 six-1.15.0 tabulate-0.8.7 urllib3-1.26.5 zipp-3.1.0
    docker: Clearing...                                                 Clearing graph
    docker: Success Update succeeded
    docker: OK
    docker: Clearing...                                                 Clearing graph
    docker: Success Update succeeded
    docker: Clearing graph
    docker: Success Update succeeded
    docker: Clearing graph
    docker: Success Update succeeded
    docker: Clearing graph
    docker: Success Update succeeded
    docker: Clearing graph
    docker: Success Update succeeded
    docker: OK
    docker: Deleting demoNodegroup...                          OK
    docker: Ingesting ../RACK-Ontology/OwlModels/AGENTS.owl... OK
    docker: Ingesting ../RACK-Ontology/OwlModels/ANALYSIS.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/BASELINE.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/CONFIDENCE.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/DOCUMENT.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/FILE.owl...   OK
    docker: Ingesting ../RACK-Ontology/OwlModels/HARDWARE.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/HAZARD.owl... OK
    docker: Ingesting ../RACK-Ontology/OwlModels/MODEL.owl...  OK
    docker: Ingesting ../RACK-Ontology/OwlModels/PROCESS.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/PROV-S.owl... OK
    docker: Ingesting ../RACK-Ontology/OwlModels/REQUIREMENTS.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/REVIEW.owl... OK
    docker: Ingesting ../RACK-Ontology/OwlModels/SECURITY.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/SOFTWARE.owl...OK
    docker: Ingesting ../RACK-Ontology/OwlModels/SYSTEM.owl... OK
    docker: Ingesting ../RACK-Ontology/OwlModels/TESTING.owl...OK
    docker: Storing nodegroups...                                       OK
    docker: Storing nodegroups...                                       OK
==> docker: Provisioning with shell script: scripts/clean.sh
==> docker: debconf: delaying package configuration, since apt-utils is not installed
    docker: (Reading database ... 24287 files and directories currently installed.)
    docker: Preparing to unpack .../libudev1_245.4-4ubuntu3.15_amd64.deb ...
    docker: Unpacking libudev1:amd64 (245.4-4ubuntu3.15) over (245.4-4ubuntu3.14) ...
    docker: Setting up libudev1:amd64 (245.4-4ubuntu3.15) ...
    docker: Processing triggers for libc-bin (2.31-0ubuntu9.2) ...
==> docker: Committing the container
    docker: Image ID: sha256:c913c0e68ac20ee303bc19499b1a02d36a5aee451c8f6083c11e5c1de8c59652
==> docker: Killing the container: a6d915045b8a38532b8a11bc519bf888f57fbdbb88935cd1b0e00a39c96358ba
==> docker: Running post-processor: docker-tag
    docker (docker-tag): Tagging image: sha256:c913c0e68ac20ee303bc19499b1a02d36a5aee451c8f6083c11e5c1de8c59652
    docker (docker-tag): Repository: gehighassurance/rack-box:dev
Build 'docker' finished after 5 minutes 11 seconds.
kquick commented 2 years ago

I can confirm that using the dev image last week, I had to manually run setup-owl.sh and setup-rack.sh to be able to successfully run ingestion. Is it possible that setup-owl.sh reverts existing setup such that setup-rack.sh would need to be run again?

On Wed, Jan 26, 2022 at 9:56 AM John Interrante @.***> wrote:

@glguy https://github.com/glguy I checked the output from the last dev build (went to https://github.com/ge-high-assurance/RACK/runs/4942731217?check_suite_focus=true, expanded Build, expanded Build rack box Docker image) and I couldn't see anything out of the normal, but here's part of the output in case it helps:

docker: Running hooks in /etc/ca-certificates/update.d...
docker:
docker: done.
docker: done.
docker: Adding user `ubuntu' ...
docker: Adding new group `ubuntu' (1000) ...
docker: Adding new user `ubuntu' (1000) with group `ubuntu' ...
docker: Creating home directory `/home/ubuntu' ...
docker: Copying files from `/etc/skel' ...
docker: Adding system user `fuseki' (UID 105) ...
docker: Adding new group `fuseki' (GID 106) ...
docker: Adding new user `fuseki' (UID 105) with group `fuseki' ...
docker: Not creating home directory `/home/fuseki'.
docker: /home/ubuntu/semtk-opensource /home/ubuntu/semtk-opensource
docker: Returning (already ran .env)
docker: /home/ubuntu/semtk-opensource
docker: Web build ID=built-a6d915045b8a-2022-01-25-201553
docker: sparqlGraph/main-oss/sparqlgraphconfigOss.js
docker: sparqlGraph/main-oss/KDLEasyLoggerConfigOss.js
docker: sparqlForm/main-oss/sparqlformconfig.js
docker: sparqlForm/main-oss/KDLEasyLoggerConfig.js
docker: ./updateWebapps.sh done
docker: Waiting for Fuseki at http://localhost:3030...
docker: Waiting for nodeGroupService at http://localhost:12059...
docker: Processing ./wheels/Pillow-9.0.0-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
docker: Processing ./wheels/PyYAML-5.4-cp38-cp38-manylinux1_x86_64.whl
docker: Processing ./wheels/ase-3.21.1-py3-none-any.whl
docker: Processing ./wheels/attrs-19.3.0-py2.py3-none-any.whl
docker: Processing ./wheels/certifi-2020.6.20-py2.py3-none-any.whl
docker: Requirement already satisfied: chardet==3.0.4 from file:///home/ubuntu/RACK/cli/wheels/chardet-3.0.4-py2.py3-none-any.whl in /usr/lib/python3/dist-packages (3.0.4)
docker: Processing ./wheels/colorama-0.4.3-py2.py3-none-any.whl
docker: Processing ./wheels/cycler-0.11.0-py3-none-any.whl
docker: Processing ./wheels/fonttools-4.29.0-py3-none-any.whl
docker: Processing ./wheels/idna-2.10-py2.py3-none-any.whl
docker: Processing ./wheels/importlib_metadata-1.7.0-py2.py3-none-any.whl
docker: Processing ./wheels/jsonschema-3.2.0-py2.py3-none-any.whl
docker: Processing ./wheels/kiwisolver-1.3.2-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
docker: Processing ./wheels/matplotlib-3.5.1-cp38-cp38-manylinux_2_5_x86_64.manylinux1_x86_64.whl
docker: Processing ./wheels/numpy-1.22.1-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
docker: Processing ./wheels/packaging-21.3-py3-none-any.whl
docker: Processing ./wheels/plotly-4.14.3-py2.py3-none-any.whl
docker: Processing ./wheels/pyparsing-3.0.7-py3-none-any.whl
docker: Processing ./wheels/pyrsistent-0.16.0-cp38-cp38-linux_x86_64.whl
docker: Processing ./wheels/python_dateutil-2.8.1-py2.py3-none-any.whl
docker: Processing ./wheels/rack-0.0.0-py3-none-any.whl
docker: Processing ./wheels/requests-2.25.1-py2.py3-none-any.whl
docker: Processing ./wheels/retrying-1.3.3-py3-none-any.whl
docker: Processing ./wheels/scipy-1.7.3-cp38-cp38-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
docker: Processing ./wheels/semtk_python3-0.0.1-py3-none-any.whl
docker: Processing ./wheels/setuptools-60.5.0-py3-none-any.whl
docker: Processing ./wheels/six-1.15.0-py2.py3-none-any.whl
docker: Processing ./wheels/tabulate-0.8.7-py3-none-any.whl
docker: Processing ./wheels/urllib3-1.26.5-py2.py3-none-any.whl
docker: Processing ./wheels/zipp-3.1.0-py3-none-any.whl
docker: Installing collected packages: Pillow, PyYAML, numpy, scipy, cycler, pyparsing, packaging, fonttools, kiwisolver, six, python-dateutil, matplotlib, ase, attrs, certifi, colorama, idna, zipp, importlib-metadata, setuptools, pyrsistent, jsonschema, retrying, plotly, rack, urllib3, requests, semtk-python3, tabulate
docker:   Attempting uninstall: six
docker:     Found existing installation: six 1.14.0
docker:     Not uninstalling six at /usr/lib/python3/dist-packages, outside environment /usr
docker:     Can't uninstall 'six'. No files were found to uninstall.
docker:   Attempting uninstall: certifi
docker:     Found existing installation: certifi 2019.11.28
docker:     Not uninstalling certifi at /usr/lib/python3/dist-packages, outside environment /usr
docker:     Can't uninstall 'certifi'. No files were found to uninstall.
docker:   Attempting uninstall: idna
docker:     Found existing installation: idna 2.8
docker:     Not uninstalling idna at /usr/lib/python3/dist-packages, outside environment /usr
docker:     Can't uninstall 'idna'. No files were found to uninstall.
docker:   Attempting uninstall: setuptools
docker:     Found existing installation: setuptools 45.2.0
docker:     Not uninstalling setuptools at /usr/lib/python3/dist-packages, outside environment /usr
docker:     Can't uninstall 'setuptools'. No files were found to uninstall.
docker:   Attempting uninstall: urllib3
docker:     Found existing installation: urllib3 1.25.8
docker:     Not uninstalling urllib3 at /usr/lib/python3/dist-packages, outside environment /usr
docker:     Can't uninstall 'urllib3'. No files were found to uninstall.
docker:   Attempting uninstall: requests
docker:     Found existing installation: requests 2.22.0
docker:     Not uninstalling requests at /usr/lib/python3/dist-packages, outside environment /usr
docker:     Can't uninstall 'requests'. No files were found to uninstall.
docker: Successfully installed Pillow-9.0.0 PyYAML-5.4 ase-3.21.1 attrs-19.3.0 certifi-2020.6.20 colorama-0.4.3 cycler-0.11.0 fonttools-4.29.0 idna-2.10 importlib-metadata-1.7.0 jsonschema-3.2.0 kiwisolver-1.3.2 matplotlib-3.5.1 numpy-1.22.1 packaging-21.3 plotly-4.14.3 pyparsing-3.0.7 pyrsistent-0.16.0 python-dateutil-2.8.1 rack-0.0.0 requests-2.25.1 retrying-1.3.3 scipy-1.7.3 semtk-python3-0.0.1 setuptools-60.5.0 six-1.15.0 tabulate-0.8.7 urllib3-1.26.5 zipp-3.1.0
docker: Clearing...                                                 Clearing graph
docker: Success Update succeeded
docker: OK
docker: Clearing...                                                 Clearing graph
docker: Success Update succeeded
docker: Clearing graph
docker: Success Update succeeded
docker: Clearing graph
docker: Success Update succeeded
docker: Clearing graph
docker: Success Update succeeded
docker: Clearing graph
docker: Success Update succeeded
docker: OK
docker: Deleting demoNodegroup...                          OK
docker: Ingesting ../RACK-Ontology/OwlModels/AGENTS.owl... OK
docker: Ingesting ../RACK-Ontology/OwlModels/ANALYSIS.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/BASELINE.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/CONFIDENCE.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/DOCUMENT.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/FILE.owl...   OK
docker: Ingesting ../RACK-Ontology/OwlModels/HARDWARE.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/HAZARD.owl... OK
docker: Ingesting ../RACK-Ontology/OwlModels/MODEL.owl...  OK
docker: Ingesting ../RACK-Ontology/OwlModels/PROCESS.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/PROV-S.owl... OK
docker: Ingesting ../RACK-Ontology/OwlModels/REQUIREMENTS.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/REVIEW.owl... OK
docker: Ingesting ../RACK-Ontology/OwlModels/SECURITY.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/SOFTWARE.owl...OK
docker: Ingesting ../RACK-Ontology/OwlModels/SYSTEM.owl... OK
docker: Ingesting ../RACK-Ontology/OwlModels/TESTING.owl...OK
docker: Storing nodegroups...                                       OK
docker: Storing nodegroups...                                       OK

==> docker: Provisioning with shell script: scripts/clean.sh ==> docker: debconf: delaying package configuration, since apt-utils is not installed docker: (Reading database ... 24287 files and directories currently installed.) docker: Preparing to unpack .../libudev1_245.4-4ubuntu3.15_amd64.deb ... docker: Unpacking libudev1:amd64 (245.4-4ubuntu3.15) over (245.4-4ubuntu3.14) ... docker: Setting up libudev1:amd64 (245.4-4ubuntu3.15) ... docker: Processing triggers for libc-bin (2.31-0ubuntu9.2) ... ==> docker: Committing the container docker: Image ID: sha256:c913c0e68ac20ee303bc19499b1a02d36a5aee451c8f6083c11e5c1de8c59652 ==> docker: Killing the container: a6d915045b8a38532b8a11bc519bf888f57fbdbb88935cd1b0e00a39c96358ba ==> docker: Running post-processor: docker-tag docker (docker-tag): Tagging image: sha256:c913c0e68ac20ee303bc19499b1a02d36a5aee451c8f6083c11e5c1de8c59652 docker (docker-tag): Repository: gehighassurance/rack-box:dev Build 'docker' finished after 5 minutes 11 seconds.

— Reply to this email directly, view it on GitHub https://github.com/ge-high-assurance/RACK/issues/645#issuecomment-1022447426, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAGAHXNQ4ZL7SUPSGTMVZGLUYAYVVANCNFSM5MZNKVKA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you are subscribed to this thread.Message ID: @.***>

-- Kevin Quick Galois, Inc.

kquick commented 2 years ago

Sorry, I was confused: with the rack-dev docker image I cannot run ingestion on the latest data import package without first running setup-owl.sh and setup-arcos.sh, but I did not need to run setup-rack.sh.

glguy commented 2 years ago

I think that we can close this issue and something else was going on with @AbhaMoitra 's case. I can confirm the Apache data loads without having to run setup-rack.sh manually.