search

RENCI / ctmd

MIT License
2 stars 0 forks source link

Longer-term fix for API data pull #436

Open dcarsey opened 3 years ago

dcarsey commented 3 years ago

Current method is not a scalable solution for a large number of proposals (10s of 1000s) The solution is not generic enough / Not intuitive Limited to the no. of proposals that can be pulled thru API at a time

Vanderbilt's suggestion: making the query smaller (into 2 or 3 instead) Decision-making WRT re-structuring the API call on the TIN side would need to be sent up the chain at Vanderbilt With only 2 yrs left on the grant, it’s unclear what the decision would be.

Is the API built specifically for the pulls? Yes. Currently: No pagination feature on the API If you want us to pull smaller chunks, the API needs to allow us to do that and it currently does not

Possibilities: Pulling diffs would also be a way to pull a smaller amt of data Push of data from Vanderbilt to a landing zone at RENCI Callback that shows what was edited API that allows us to get the diffs Spare interface that allows us to pull all of the data

FWIW: About 100 proposals come in each year

krobasky commented 3 years ago

~100 proposals is the current prediction from Michelle and Jackie, but HEAL stewards have indicated there are many, MANY more studies, so I'm not sure how to reconcile that.

dcarsey commented 1 year ago

Smart words from Vladimir: need the ability to organize and query the data pull