Run SSDF application package on U-SPS

unity-sds / unity-analytics

Unity Analytics Service (U-AS) project organization.

Apache License 2.0

0 stars 2 forks source link

Run SSDF application package on U-SPS #70

Open jjacob7734 opened 5 months ago

jjacob7734 commented 5 months ago

Run the SSDF application package on a supported U-SPS platform (e.g., Airflow, Kubernetes, HySDS). This ticket is dependent on a SSDF container being pushed to Docker Hub and SSDF workflow CWL being pushed to Dockstore in https://github.com/unity-sds/unity-analytics/issues/67.

Acceptance Criteria:

[ ] SSDF runs successfully on U-SPS
[ ] Verify output is the same as the output produced by the Jupyter notebook.

jjacob7734 commented 2 weeks ago

Tutorial on how to register and run an application on MDPS: https://docs.google.com/document/d/1WhYS764bWqjlpN1-5fWme703RmePYrg0/edit

jjacob7734 commented 1 week ago

Status:

Started trying to run the build application locally (not on U-SPS yet).
Papermill run worked fine both outside and inside the container.
Cwl-Runner gave some errors related to stage-in and stage-out parameters (being diagnosed now).

jjacob7734 commented 6 days ago

Unity Airflow venue-ops login URL: https://k8s-airflow-airflowi-4c3045c7a4-224403021.us-west-2.elb.amazonaws.com:5000/ The above could change, so refer to the updated URLs in this spreadsheet: https://docs.google.com/spreadsheets/d/1FU0sWr726gqtaM50BHaltY29uE2724szWRvtG1dmgXI/edit?gid=0#gid=0