search

ucd-library / aggie-experts

Publicly reported feedback and issues for Aggie Experts
https://ucd-library.github.io/aggie-experts/
MIT License
1 stars 2 forks source link

Verify one-time feed update results #223

Open qjhart opened 8 months ago

qjhart commented 8 months ago

It's apparent that we will most likely need to modify our one time archival data input feed for CDL elements. We have the following questions

No they are not. Once you add them in, there is no removing them from the system.

This is still an open question. If you include a visibility flag, then they are overwritten, but we are unclear if you don't include a visibility flag at all.

Don't know

To test, we need to create an updated feed that includes the above tests, and reload them over the current qa load, and answer the above questions.

Column Updates

Archival Grants

The archival version of grants (KFS), have the following properties:

property count disposition
#grantIndirectCosts "27133 ignore
#subAwardOf "1330 Move to flow_thru_funding
#assignedBy "28463 include
#dateTimeInterval "27133 include
#grantDirectCosts> "27133 ignore
#relates "49025 There are the relations
#sponsorAwardId "27081 include
#totalAwardAmount "27133 include
#type "28463 GrantType
#label "27133 title

Grant Types

Because we had an explicit set of grant types, we used URLs as their identifiers. However, there is already an existing Grant Type in the feed that is a text field, so I propose we use the names below in that field. Also, maintain Default, since that's how it is in KFS. Changing that to Research will be in the UI interface.

URL name count
<ark:/87287/d7gt0q/schema#GrantTypeAcademicSupport> AcademicSupport 5
<ark:/87287/d7gt0q/schema#GrantTypeDefault> Default 11959
<ark:/87287/d7gt0q/schema#GrantTypeInstruction> Instruction 719
<ark:/87287/d7gt0q/schema#GrantTypeResearch> Research 10821
<ark:/87287/d7gt0q/schema#GrantTypeService> Service 3629

flow_thru_funding

In the archival grants, we identified federal funding, by identifying sub-awards. For example:

<ark:/87287/d7gt0q/grant/105667#subAwardOf_b23f52202479e957b9bada847c1175d7>
        rdf:type         vivo:Grant;
        vivo:assignedBy  <ark:/87287/d7gt0q/funding_org/3099> .

<ark:/87287/d7gt0q/grant/105667>
        rdf:type                   vivo:Grant;
        rdfs:label                 "Nurse Education Initiative Phase II Continuation Project";
        ucdlib:grantIndirectCosts  0;
        ucdlib:grantType           <ark:/87287/d7gt0q/schema#GrantTypeService>;
        ucdlib:subAwardOf          <ark:/87287/d7gt0q/grant/105667#subAwardOf_b23f52202479e957b9bada847c1175d7>;
        vivo:assignedBy            <ark:/87287/d7gt0q/funding_org/6390>;
        vivo:dateTimeInterval      <ark:/87287/d7gt0q/grant/105667#duration>;
        vivo:grantDirectCosts      280332;
        vivo:relates               <ark:/87287/d7gt0q/admin_role/029992-105667> , <ark:/87287/d7gt0q/grant/105667#role-6b35e9d7e187995983d2f25d7f233252-COPI> , <ark:/87287/d7gt0q/grant/105667#role-76ef8d8b8edeb2311fdc0f0b2b032ce2-PI> , <ark:/87287/d7gt0q/grant/105667#role-211000addec50fbb93e1aee2ec508679-COPI> , <ark:/87287/d7gt0q/grant/105667#role-179993c04f059065fd1e920bd25439c6-COPI> , <ark:/87287/d7gt0q/grant/105667#role-65f12f8b3ed7ab6bf9a477b0721ca562-COPI> , <ark:/87287/d7gt0q/grant/105667#role-eb6192543a7c62d772f4347b5e795a0e-COPI> , <ark:/87287/d7gt0q/grant/105667#role-0f2be2d37e42d3dc48855f99978abc0c-COPI> , <ark:/87287/d7gt0q/grant/105667#role-bb1f157e61e75aa3d777a3fb63217b69-COPI>;
        vivo:sponsorAwardId        "K287150";
        vivo:totalAwardAmount      280332 .

This was consitant, but doesn't match well the way federal_grants are done in KFS or the new setup.

Therefore, I propose we make a new column flow_thru_funding that is also a URL and also the same as the ucop_sponsor_code. We will rewrite the grant so instead of being a sub-award we just include"

<ark:/87287/d7gt0q/grant/105667>
        rdf:type                   vivo:Grant;
        rdfs:label                 "Nurse Education Initiative Phase II Continuation Project";
        ucdlib:grantIndirectCosts  0;
        ucdlib:grantType           <ark:/87287/d7gt0q/schema#GrantTypeService>;
        vivo:assignedBy            <ark:/87287/d7gt0q/funding_org/6390>;
ucdlib:flow_thru_funding  <ark:/87287/d7gt0q/funding_org/3099> .
        vivo:dateTimeInterval      <ark:/87287/d7gt0q/grant/105667#duration>;
        vivo:grantDirectCosts      280332;
        vivo:sponsorAwardId        "K287150";
        vivo:totalAwardAmount      280332 .

New Grants.

The new grants look like this: 

   {
      "name": "Maternal Inflammation during Pregnancy and Neurodevelopmental Disorders K370D48",
      "number": "K370D48",
      "start_date": "2020/09/01",
      "end_date": "2024/05/31",
      "status": "Active",
      "purpose": "44-Research",
      "type": "30-Foundation - FFT - Contracts",
      "total_grant_amount": "96745",
      "direct_costs": "61621",
      "indirect_costs": "35124",
      "sponsor_award_number": "RNG210887-01",
      "description": "Maternal Inflammation during Pregnancy and Neurodevelopmental Disorders",
      "assistance_listing_numbers": {
        "assistance_listing_number": "93.865"
      },
      "principal_investigator": {
        "name": "Judith Van De Water",
        "number": "10223549",
        "email": "javandewater@ucdavis.edu"
      },
      "financial": {
        "funding_sources": {
          "funding_source": {
            "name": "KAISER PERMANENTE CENTER FOR HEALTH RESEARCH",
            "ucop_sponsor_code": "VAQJ"
          }
        },
        "organization_credits": {
          "organization_credit": {
            "name": "MINT Int Med Rheumatology and Allergy",
            "number": "4305413",
            "credit_percentage": "100"
          }
        },
        "flow_thru_funding": {
          "sponsor": "NATIONAL INSTITUTES OF HEALTH CENTER FOR SCIENTIFIC REVIEW",
          "ucop_sponsor_code": "5013"
        }
      },
      "participants": {
        "participant": {
          "person": {
            "number": "10223549",
            "name": "Judith Van De Water",
            "email": "javandewater@ucdavis.edu"
          },
          "percentage": "100",
          "role": "Principal Investigator"
        }
      }
    }

The two things to note, is that we will copy the flow_thu_funding naming. Also the types of grants that exist in the new setup are :

grant type
40-Instruction
44-Research
45-Research AES
62-Public Service/Other
68-Student Services
78-Student Financial Aid

Let's just use these directly as the Funding Type in the new grants as well.

rakunkel-ucd commented 8 months ago

I think that the missing grant fields are a result of our header filed names. "institution_reference","funding_type","start_date","end_date","funder_name","funder_reference","amount_value","amount_currencycode" should all use "-" instead of "" as word separators.

rakunkel-ucd commented 7 months ago

Confirmed that matching records are updated with revised feed file. Non-match (removed) records are not deleted. No bulk delete method available. Alainna suggests waiting for the next QA refresh.

qjhart commented 7 months ago

@rakunkel-ucd Can you update the top of this w.r.t. the emails and CDL issues we've received.
I believe you and VE decided to create a new issue about documenting this at #248

qjhart commented 6 months ago

@rakunkel-ucd / @Vensberg I have added flow-thru-funding to the UC Davis feed definition #280

rakunkel-ucd commented 6 months ago

I believe we need to request a reporting-db rebuild for our flow-thru-funding filed to appear there.