search

microbiomedata / nmdc-runtime

Runtime system for NMDC data management and orchestration
https://microbiomedata.github.io/nmdc-runtime/
Other
4 stars 3 forks source link

Some Napa environment-specific URLs resolve to production environment (as of an 09/2023 commit) #442

Open aclum opened 5 months ago

aclum commented 5 months ago

See: https://dagit-napa.microbiomedata.org/runs/c3b9b292-7fc4-403d-a0bd-d628f61438a1?focusedTime=1705439489991&logFileKey=suimkuwe changesheets:submit endpoint works but json:submit endpoint does not. This endpoint is what we are using to add data objects and workflow execution metadata for proteomics.

This is blocking testing proteomics re-iding cc @picowatt @corilo @dwinston

env: napa

dwinston commented 5 months ago

It seems that https://api-napa.microbiomedata.org/objects/gfs0we84 redirects to https://api.microbiomedata.org/objects/gfs0we84. The first redirect in the chain is https://w3id.org/nmdc/gfs0we84 (inspecting network requests via Firefox's web developer tools), and I recalled that I had redirected to w3id.org for an early implementation of /objects/{object_id} routing in the codebase. Looking at the nmdc-napa namespace in NERSC Spin Rancher, I see the runtime-api workload is pinned to the microbiomedata/nmdc-runtime-fastapi:commit-481ab8e image, whereas the workload in the nmdc-dev namespace is pinned to microbiomedata/nmdc-runtime-fastapi:main. I'd recommend redeploying the nmdc-napa runtime-api workload to use the :main image, but I'll leave that to you, as I don't know if you have some dependency on that version of the runtime at the moment and need to cherry-pick the code changes.

dwinston commented 5 months ago

I went ahead and made a napa branch (https://github.com/microbiomedata/nmdc-runtime/compare/napa?expand=1) for which I reverted nmdc-schema to 7.8.0, and I pushed tagged images, so that redeploying the napa-namespace workloads using :napa images should work.

ssarrafan commented 5 months ago

Appears to be active. Moving to next sprint.

eecavanna commented 4 months ago

I will move this ticket to the backlog.

A workaround exists: The impacted Napa Compliance squad members were able to insert the data data directly into the Napa database, bypassing the Napa Runtime API altogether. For similar data insertion tasks in the future, they will be able to use the standalone Python script, insert_many_pymongo.py. That script has been updated to validate the data before inserting it into the database.

The issue is fixed in newer versions: Also, @dwinston pointed out on Slack that, in the latest version of the Runtime API (which is several months newer than the version in use by the Napa Compliance squad), the offending endpoint no longer redirects to w3id.org; so, ~I think~ the issue has been fixed in the main branch and is only occurring here because the squad is using a version from before that fix was implemented. More details are available on Slack. With that in mind, I will update the title of this issue to specify the nmdc-runtime version exhibiting the behavior.

eecavanna commented 4 months ago

I looked up the Git commit whose hash appears in the container image tag (i.e. commit-481ab8e), and found it here. The commit was made on September 22, 2023. I have renamed the issue accordingly.

In that commit, I can see the w3id URL in use: https://github.com/microbiomedata/nmdc-runtime/commit/481ab8e#diff-c9bdff18fbc6cff725b9941297b83ad46cfbbe37791743ebbb3aa47e5ed993b6R143

Once the Napa Compliance squad's environment is gone (date TBD), this issue can be closed.