M7 Pulling files from the Portal

elisabettai commented 2 years ago

Facilitation for users to have a one-click or few-click method to pull files from a dataset into o²S²PARC directly from the Portal. For example, to launch o²S²PARC from a dataset landing page, or from a file in the data-set file list, or based on a Shopping Cart (could be extended in Y7 to offer file-data-annotation-based intelligent selection of the most relevant services/workflows on o²S²PARC – under K-Core leadership).

Deliverable: Portal functionality to pull files from a dataset into o²S²PARC directly from the Portal with one / few clicks
Deadline: M7
Wrike M2.1.6

### Watermelon
- [x] Connect to portal @ignapas
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4342
- [x] db table in osparc.io that maps ext with services is up-to-date @elisabettai
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4392
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4390
- [ ] https://github.com/ITISFoundation/osparc-simcore/pull/4414
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4371
- [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4409
- [ ] https://github.com/nih-sparc/sparc-app/pull/683
- [ ] https://github.com/nih-sparc/sparc-app/pull/696

pcrespov commented 1 year ago

Goal for sprint Mithril

Proposal of implementation "Copy&Paste onto Lazy file-picker"

Select files in the UI
Click "copy for osparc"
Create download links and paste in the clipboard with special markers.
A message pops up and says ready to paste in osparc (with a link)
On top of osparc, in an open study, click paste
osparc recognizes the special mark and creates a FilePicker per link in the clipboard
- it downloads file when used the first time and sparc needs to change to s3: Lazy File Picker
Alternative: instead of download links, we could get special identifiers copied and the "Lazy File Picker" asks the backend to download onto S3 when needed

PROS:

The operation of copy&paste can be repeated at anytime (not only with "new studies").
Files are only downloaded when used

Portal @ignapas; osparc @odeimaiz and @pcrespov; Full UX and testing @elisabettai

pcrespov commented 1 year ago

Update for sprint Mithril

Discarded proposed designed (see Goal above) since nih-portal will take a new approach to select files (like a checkout kaufwagen)
Implementing /view approach in https://github.com/ITISFoundation/osparc-simcore/pull/4043
See PR references above for details

pcrespov commented 1 year ago

Goal for sprint Jelly Beans

Implement view_url approach to create study with a single file-picker that downloads selected file
- https://github.com/ITISFoundation/osparc-simcore/pull/4043
Fix random decoding error in download link
- https://github.com/ITISFoundation/osparc-simcore/issues/3951
NIH Portal side : compose view_url upon file selection and make request @ignapas ??

pcrespov commented 1 year ago

Update for sprint Jelly Beans

Implement view_url approach to create study with a single file-picker that downloads selected file
- https://github.com/ITISFoundation/osparc-simcore/pull/4043 (almost ready)
https://github.com/ITISFoundation/osparc-simcore/issues/3951
NIH Portal side : compose view_url upon file selection and make request @ignapas ??

esraneufeld commented 1 year ago

ideally to be finished in time for the next CCB meeting. ask manuel when that is.

pcrespov commented 1 year ago

Goal for sprint Pastel de Nata

@ignapas follows up from https://github.com/ITISFoundation/osparc-simcore/pull/4043 to implement portal-side functionality

pcrespov commented 1 year ago

Update for sprint Pastel de Nata

Done:

All backend implementation ready
- https://github.com/ITISFoundation/osparc-simcore/pull/4043
Extra backend details (e.g. messages, etc) completed.

Open:

Open: follow up from @ignapas in portal

esraneufeld commented 1 year ago

try to have it ready for the next CCB

elisabettai commented 1 year ago

@pcrespov, I have updated the deadline: we need this by this Friday (June 23rd)

elisabettai commented 1 year ago

@pcrespov, I've checked the box "db table in osparc.io that maps ext with services is up-to-date ". The table is up to date, only change I made was bumping the version of jupyterlab math to 2.0.9. I've noticed two small things while testing.

Testing

I've also tested a bit on staging.sparc.science (with file picker only, jupyter+IPYNB and raw graphs+CSV). All works as expected, with these 2 datasets:

https://staging.sparc.science/datasets/44?type=simulation&datasetDetailsTab=files&path=files%2Fcode (with the ipynb file or any other file)
https://staging.sparc.science/datasets/323?type=dataset&datasetDetailsTab=files&path=files%2Fprimary%2Fsub-FI-8 (with a csv). Only "problem" is that the file is very big and takes long to load in raw graph.

VERY minor "bugs"

I've only noticed two small things. 1) File picker option is called "No service". In the PR I think you proposed something different to @ignapas 2) When you open a ipynb from the Portal, the corresponding Study saved in your Dashboard should get the service thumbnail, while I have a default one.

fyi @mguidon

pcrespov commented 1 year ago

Update Watermelon

Some fixes, tuning and extra testing on the osparc stack (see details in tasks above)
Release to production in time fir the CCB. Synced with the nih-sparc portal release workflow
Finishing last issues from feedback

ITISFoundation / osparc-issues