curiouslearning / followthelearners

MIT License
2 stars 1 forks source link

Scripts for pulling one-time data from partner data sources needed in the Admin Dashboard UI #513

Closed burrage closed 3 years ago

burrage commented 3 years ago

User Story As a developer, I want to write scripts that pull one-time information from various partner data sources and display them when needed in the UI, So that I can populate the Tech Health and Deep Dive Tech sections of the Admin Dashboard with data from APIs of various services that help to run FTL.

Acceptance Criteria Given that I have completed the UI wireframes of the Tech Health and Deep Dive Tech sections and I have identified which data can be pulled one-time on-demand vs needing to be stored over time, When I view the Tech Health and Deep Dive Tech sections of the Admin Dashboard, Then I should see values returned from various API services displayed in the UI.

eric-gt commented 3 years ago

Notes on Integrations:

eric-gt commented 3 years ago

Unfortunate update on Google Cloud integration with the current project infrastructure. Google Cloud does not appear to be support being bundled through webpack: https://github.com/googleapis/google-cloud-node/issues/1821 It looks like there's ways around it, such as installing Babel and using it as an external bundler specifically for google cloud libraries. But it's also breaking our unit testing library somehow

C:\Users\Eric\Documents\bin\followthelearners\node_modules\protobufjs\src\root.js:1                                      
Error: not supported                                                                                                     
    at Root.loadSync (C:\Users\Eric\Documents\bin\followthelearners\node_modules\protobufjs\src\root.js:236:15)          
    at Object.loadSync (C:\Users\Eric\Documents\bin\followthelearners\node_modules\protobufjs\src\index-light.js:69:17)  
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\node_modules\@grpc\proto-loader\build\src\index.
js:230:37)                                                                                                               
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\node_modules\@google-cloud\logging\node_modules\
google-gax\src\grpc.ts:17:1)                                                                                             
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\node_modules\@google-cloud\logging\node_modules\
google-gax\src\index.ts:18:1)                                                                                            
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\node_modules\@google-cloud\logging\src\index.ts:
23:1)                                                                                                                    
    at Generator.next (<anonymous>)                                                                                      
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                            
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\static\js\admin\cloudLogReader.ts:1)            
    at Generator.next (<anonymous>)                                                                                      
    at Object.m._compile (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\src\index.ts:1056:23)       
    at Module._extensions..js (internal/modules/cjs/loader.js:1157:10)                                                   
    at Object.require.extensions.<computed> [as .ts] (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\
src\index.ts:1059:12)                                                                                                    
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\static\js\admin\adminStoplightChart.ts:1)       
    at Generator.next (<anonymous>)                                                                                      
    at Object.require.extensions.<computed> [as .ts] (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\
src\index.ts:1059:12)                                                                                                    
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\static\js\admin\admin.ts:1)                     
    at Generator.next (<anonymous>)                                                                                      
    at Object.require.extensions.<computed> [as .ts] (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\
src\index.ts:1059:12)                                                                                                    
    at Object.<anonymous> (C:\Users\Eric\Documents\bin\followthelearners\test\static\js\admin\admin.spec.ts:1)           
    at Generator.next (<anonymous>)                                                                                      
    at Object.require.extensions.<computed> [as .ts] (C:\Users\Eric\Documents\bin\followthelearners\node_modules\ts-node\
src\index.ts:1059:12)                                                                                                    
    at Object.exports.requireOrImport (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\esm-utils.js:
42:12)                                                                                                                   
    at Object.exports.loadFilesAsync (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\esm-utils.js:5
5:34)                                                                                                                    
    at Mocha.loadFilesAsync (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\mocha.js:431:19)       
    at singleRun (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\cli\run-helpers.js:125:15)        
    at exports.runMocha (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\cli\run-helpers.js:190:10) 
    at Object.exports.handler (C:\Users\Eric\Documents\bin\followthelearners\node_modules\mocha\lib\cli\run.js:362:11)   
    at C:\Users\Eric\Documents\bin\followthelearners\node_modules\yargs\build\index.cjs:443:71    at process.runNextTicks
 [as _tickCallback] (internal/process/task_queues.js:62:5)                                                               
    at internal/main/run_main_module.js:17:47                                                                            

It should be noted that Google exposes the RESTful API underlying its client libraries, so if we can figure out how to format our GET request to pass the necessary authentication parameters we can keep the current implementation that uses the fetch API, but Google really wants you to use client libraries and I haven't found any documentation on how to pass authentication in a simple request. I'm leaving this for now and including a link to the cloud functions builtin monitoring UI in the deep dive task. Capturing this though for when we have some cycles to actually track down what's happening