Open amykglen opened 1 month ago
I suppose maybe I should have created a separate issue for KG2c development work for 2.10.0, but I didn't - tagging that work here. It should be pretty minimal anyway
Going ahead with loading KG2.10.0c into neo4j
KG2.10.0c is up and running in plover, and all plover regression pytests are passing
just waiting for the Neo4j to be loaded so we can do a little manual inspection before giving Chunyu the green light
KG2.10.0c up and running on http://kg2canonicalized.rtx.ai:7474/browser/
awesome! just poked around a bit and it's looking good to me
one thing to note is that the acetaminophen cluster (and probably a number of other chemical entities) added a lot of members due to the SRI's addition of drug/chemical conflations - now things like Tylenol
and acetaminophen 500 mg oral tablet
are included in the acetaminophen cluster (yay)
I'd say we're ready for Chunyu to start building the refreshed xDTD - do you agree, @sundareswarpullela?
@chunyuma KG2.10.0c neo4j is up and running on http://kg2canonicalized.rtx.ai:7474/browser/, @amykglen and I think we are ready for a refreshed KG2.10.0c xDTD
fyi I kicked off the NGD rebuild - I'm doing it on buildkg2c.rtx.ai
because I think we're well overdue for a 'full' rebuild (which requires more memory than a partial rebuild, which is what we typically do)
I updated ARAX_decorator to handle the two new edge properties (knowledge_level and agent_type) and verified that they appear to be coming through correctly in TRAPI responses! here's an example of the TRAPI attributes
on an edge returned from KG2 (in the kg2.10.0c
branch):
'attributes': [{'attribute_source': 'infores:rtx-kg2',
'attribute_type_id': 'biolink:original_predicate',
'attributes': None,
'description': 'The IDs of the original RTX-KG2pre edge(s) '
'corresponding to this edge prior to any '
'synonymization or remapping.',
'original_attribute_name': None,
'value': ['DrugCentral:294---DrugCentral:indication---None---None---None---DOID:4730---DrugCentral:'],
'value_type_id': 'metatype:String',
'value_url': None},
{'attribute_source': 'infores:rtx-kg2',
'attribute_type_id': 'biolink:knowledge_level',
'attributes': None,
'description': None,
'original_attribute_name': None,
'value': 'knowledge_assertion',
'value_type_id': 'metatype:String',
'value_url': None},
{'attribute_source': 'infores:rtx-kg2',
'attribute_type_id': 'biolink:agent_type',
'attributes': None,
'description': None,
'original_attribute_name': None,
'value': 'manual_agent',
'value_type_id': 'metatype:String',
'value_url': None}]
just an update on this - NGD and refreshed XDTD builds are done and all pytests are passing in the kg2.10.0c
branch except for the XDTD Infer tests and the test_FET_ex1
test.
the XDTD Infer test failures appear to be due to an issue during edge decoration where the sqlite that is used to decorate XDTD's support graph edges doesn't yet have knowledge_level
and agent_type
columns. I've messaged @chunyuma in Slack about two options for addressing this, at least until the full XDTD rebuild is done.
Hi @amykglen, thank you for informing me about the test failures. I have fixed the XDTD Infer tests and the test_FET_ex1
test. Now, it seems like all tests are passing in the kg2.10.0c
branch.
awesome, thanks @chunyuma!
@sundareswarpullela - do you want to take care of turning off old Neo4j instances as appropriate? (per the checklist)
Sure! Will do now. Thanks for reminding :)
THE BRANCH FOR THIS ROLLOUT IS:
kg2.10.0c
THE ARAX-DATABASES.RTX.AI DIRECTORY FOR THIS ROLLOUT IS:/home/rtxconfig/KG2.10.0
**Sprint changelog link: (Changelog)
Prerequisites
ssh access
To complete this workflow, you will need
ssh
access to:arax-databases.rtx.ai
arax.ncats.io
(see example configuration information below)kg2cploverN.rtx.ai
kg2canoncalizedN.rtx.ai
cicd.rtx.ai
kg2webhost.rtx.ai
GitHub access
RTXteam/PloverDB
project areaRTXteam/RTX
project areancats/translator-lfs-artifacts
project area (not critical, but needed for some final archiving steps; Amy Glen and Sundar Pullela have access)AWS access
You will need:
s3://rtx-kg2/
(ask Stephen Ramsey for access)Slack workspaces
You will also need access to the following Slack workspaces:
Example ssh config for setting up login into
arax.ncats.io
:1. Build and load KG2c:
master
into the branch being used for this KG2 version (which would typically be named likeKG2.X.Yc
). Record this issue number in the merge message.code/UI/OpenAPI/python-flask-server/openapi_server/openapi/openapi.yaml
(github; local)code/UI/OpenAPI/python-flask-server/KG2/openapi_server/openapi/openapi.yaml
(github; local)buildkg2c.rtx.ai
from the branch (how-to is here)arax-databases.rtx.ai
(need at least 100G, ideally >120G). delete old KG2 database directories as needed (warn the team on Slack in advance).kg2c_config.json
, as described in the how-tonode_synonymizer.sqlite
should be around 8-15 GBnode_synonymizer.sqlite
's last modified date is today (or whatever day the build was run)kg2c_lite.json.gz
's last modified date is today (or whatever day the build was run)arax-databases.rtx.ai
(/home/rtxconfig/KG2.X.Y
)match (n {id:"RTX:KG2c"}) return n
RTX/code/config_dbs.json
in the branch:kg2c_lite_2.X.Y.json.gz
file (which you can get from the S3 buckets3://rtx-kg2/kg2c_lite.json.gz
(but CHECK THE DATE AND MD5 HASH TO BE SURE YOU ARE NOT GETTING AN OLD FILE) to the directory/home/ubuntu/nginx-document-root/
onkg2webhost.rtx.ai
kg2cploverN.rtx.ai
:ssh ubuntu@kg2cploverN.rtx.ai
cd PloverDB && git pull origin kg2.X.Yc
./run.sh ploverimage2.X.Y plovercontainer2.X.Y "sudo docker"
config_dbs.json
in the branch for this KG2 version in the RTX repo to point to the new Plover for the 'dev' maturity level2. Rebuild downstream databases:
The following databases should be rebuilt and copies of them should be put in
/home/rtxconfig/KG2.X.Y
onarax-databases.rtx.ai
. Please use this kind of naming format:mydatabase_v1.0_KG2.X.Y.sqlite
.NOTE: As databases are rebuilt,
RTX/code/config_dbs.json
will need to be updated to point to their new paths! Push these changes to the branch for this KG2 version, unless the rollout of this KG2 version has already occurred, in which case you should push tomaster
(but first follow the steps described here).3. Update the ARAX codebase:
All code changes should go in the branch for this KG2 version!
RTX/code/ARAX/KnowledgeSources/RTX_KG2c_test_triples.json
force_local = True
inARAX_expander.py
(to avoid using the old KG2 API)pytest -v
)RTX/code/UI/OpenAPI/python-flask-server/openapi_server/openapi/openapi.yaml
(github; local); the major and minor release numbers are kept synchronous with the TRAPI version; just bump the patch release version (least significant digit)RTX/code/UI/OpenAPI/python-flask-server/KG2/openapi_server/openapi/openapi.yaml
(github; local); the first three digits are kept synchronous with the KG2 release version4. Pre-upload databases:
Before rolling out, we need to pre-upload the new databases (referenced in
config_dbs.json
) toarax.ncats.io
and the ITRB SFTP server. These steps can be done well in advance of the rollout; it doesn't hurt anything to do them early.arax.ncats.io
has at least 100G of disk space free; delete old KG2 databases to free up space as needed (before doing this, warn the team on the#deployment
Slack channel on theARAXTeam
workspace)arax-databases.rtx.ai
toarax.ncats.io:/data/orangeboard/databases/KG2.X.Y
; example for KG2.8.0:ssh myuser@arax.ncats.io
cd /data/orangeboard/databases/
mkdir -m 777 KG2.8.0
scp rtxconfig@arax-databases.rtx.ai:/home/rtxconfig/KG2.8.0/*2.8.0* KG2.8.0/
5. Rollout new KG2c version to
arax.ncats.io
development endpoints#deployment
channel in theARAXTeam
Slack workspace that you are rolling out a new version of KG2c to the variousarax.ncats.io
development endpoints. Provide the KG2c version number in this notification.RTXteam/RTX
project, merge themaster
branch into the branch for this KG2 version. Record the RTX issue number (for the KG2c rollout checklist issue) in the merge message.RTXteam/RTX
project, merge this KG2 version's branch back into themaster
branch. Record this issue number in the merge message.master
out to a specific ARAX or KG2 endpoint named/EEE
, you would do the following steps:arax.ncats.io
)arax.ncats.io
:ssh arax.ncats.io
(you previously need to have set up your username, etc. in~/.ssh/config
; see the top of this issue template for an example)rtx1
container:sudo docker exec -it rtx1 bash
rt
:su - rt
EEE
endpoint:cd /mnt/data/orangeboard/EEE/RTX
git branch
(should show* master
)git stash
git pull origin master
git stash pop
requirements.txt
, make sure to dopip3 install -r code/requirements.txt
exit
(exiting out of your shell session as userrt
should return you to aroot
user session)service RTX_OpenAPI_EEE restart
tail -f /tmp/RTX_OpenAPI_EEE.elog
{"nodes": {"n00": {"ids": ["RTX:KG2c"]}}, "edges": {}}
(it should return 1 result and the name of that node gives the KG2c version that is installed in the PloverDB that is being queried by the endpoint)RTX:KG2
in the Synonyms tab in the UImaster
out to the variousarax.ncats.io
development endpoints. Usually in this order:devED
kg2beta
beta
kg2test
test
devLM
rtx1
container, run the pytest suite on the various ARAX development endpoints (that meansdevED
,devLM
,test
, andbeta
):cd /mnt/data/orangeboard/EEE/RTX/code/ARAX/test && pytest -v
ssh ubuntu@cicd.rtx.ai
cd RTX
git pull origin master
requirements.txt
, make sure to do~/venv3.9/bin/pip3 install -r requirements.txt
sudo bash
mkdir -m 777 /mnt/data/orangeboard/databases/KG2.X.Y
exit
~/venv3.9/bin/python3 code/ARAX/ARAXQuery/ARAX_database_manager.py --mnt --skip-if-exists --remove_unused
6. Final items/clean up:
kg2-X-Zc.rtx.ai
(whereZ
is one less than the new minor release version): runnslookup kg2-X-Zc.rtx.ai
(it will return eitherkg2canonicalized.rtx.ai
orkg2canonicalized2.rtx.ai
; we'll call itkg2canonicalizedN.rtx.ai
).#deployment
channel in theARAXTeam
Slack workspace that you will be stopping thekg2canonicalizedN.rtx.ai
Neo4j endpointssh ubuntu@kg2-X-Zc.rtx.ai
sudo service neo4j stop
kg2canonicalizedN.rtx.ai
kg2-X-Zcplover.rtx.ai
(whereZ
is one less than the new minor release version): runnslookup kg2-X-Zploverc.rtx.ai
(it will return eitherkg2cplover.rtx.ai
,kg2cplover2.rtx.ai
, orkg2cplover3.rtx.ai
; we'll call itkg2cploverN.rtx.ai
).#deployment
channel in theARAXTeam
Slack workspace that you will be stopping thekg2-X-Zcplover.rtx.ai
PloverDB servicekg2cploverN.rtx.ai
:ssh ubuntu@kg2cploverN.rtx.ai
sudo docker stop plovercontainer2.X.Z
(if you are not sure of the container name, usesudo docker container ls -a
to get the container name).main
branch intokg2.X.Yc
branch (ifmain
has any commits ahead ofkg2.X.Yc
). Reference this issue (via its full GitHub URL) in the merge message.kg2.X.Yc
branch intomain
branch. Reference this issue (via its full GitHub URL) in the merge message.kg_config.json
in themain
branch of the Plover repo to point to the newkg2c_lite_2.X.Y.json.gz
file (push this change)kg2cploverdb.ci.transltr.io
cd PloverDB && pytest -vs test/test.py -k test_version --endpoint https://kg2cploverdb.ci.transltr.io
cd PloverDB && pytest -v test/test.py --endpoint https://kg2cploverdb.ci.transltr.io
RTX/code/config_dbs.json
and setforce_local = True
in Expand)RTX/code/config_dbs.json
in themaster
branch to point to the ITRB Plover endpoints (all maturity levels): (dev
:kg2cploverdb.ci.transltr.io
;test
:kg2cploverdb.test.transltr.io
;prod
:kg2cploverdb.transltr.io
)master
branch code commit to the various endpoints onarax.ncats.io
that you previously updated (this is in order to get the changedconfig_dbs.json
file) and restart ARAX and KG2 services#deployment
channel to notify people what you are about to dossh ubuntu@kg2cploverM.rtx.ai
sudo docker container ls -a
(gives you the name of the container; assume it isplovercontainer2.X.Y
)sudo docker stop plovercontainer2.X.Y
kg2c_lite_2.X.Y.json.gz
file to the translator-lfs-artifacts repo (ask Amy Glen, who has permissions to do this)kg2_nodes_not_in_sri_nn.tsv
file to the translator-lfs-artifacts repo7. Roll-out to ITRB TEST
master
toitrb-test
. Record this issue number in the merge message.main
toitrb-test
. Record this issue number in the merge message.master
branch of RTXteam/RTX project.main
branch of RTXteam/PloverDB project.#devops-teamexpanderagent
channel in theNCATSTranslator
Slack workspace, put in a request to@Sarah Stemann
to open a ticket to re-deploy ARAX, RTX-KG2, and PloverDB to ITRB test#devops-teamexpanderagent
channel to follow the roll-out of the updated services in ITRB test (i.e., to see if there are any errors reported by ITRB)kg2cploverdb.test.transltr.io
RTXteam/PloverDB
project'smaster
branch, do :cd PloverDB && pytest -v test/test.py --endpoint https://kg2cploverdb.test.transltr.io
kg2.test.transltr.io
(look at messages logdebug
mesages to verify that it is indeed queryingkg2cploverdb.test.transltr.io
)arax.test.transltr.io
(look at messages logdebug
mesages to verify that ARAX-Expand is indeed queryingkg2.test.transltr.io
)8. Roll-out to ITRB PRODUCTION
master
toproduction
. Record this issue number in the merge message.main
toproduction
. Record this issue number in the merge message.#devops-teamexpanderagent
channel in theNCATSTranslator
Slack workspace, put in a request to@Sarah Stemann
to open a ticket to re-deploy ARAX, RTX-KG2, and PloverDB to ITRB production#devops-teamexpanderagent
channel to follow (i.e., to see if there are any errors reported by ITRB) the roll-out of the updated services in ITRB production (this could take several days, as there is a formal approval process for deployments to ITRB production)kg2cploverdb.transltr.io
kg2.transltr.io
(look at messages logdebug
mesages to verify that it is indeed queryingkg2cploverdb.transltr.io
)arax.transltr.io
(look at messages logdebug
mesages to verify that ARAX-Expand is indeed queryingkg2.transltr.io
)