Fix duplicate feature for large studies

odeimaiz commented 3 years ago

Duplicating large studies fails

KZzizzle commented 2 years ago

related to #466

pcrespov commented 2 years ago

Update on sprint Croissant

Planned

This issue aims to enhance cloning of large studies. We are

Redesigning the clone feature using the feedback gathered in related issues. For completeness, these are the most relevant:
- ITISFoundation/osparc-simcore#2152: SAN gives guidelines to develop
- ITISFoundation/osparc-issues#560: TN reports bug
- ITISFoundation/osparc-issues#259: copy nodes! including state data could be achieved as well
- ITISFoundation/osparc-issues#558: bug report
- ITISFoundation/osparc-simcore#2805: ops shows heavy use of resources with export
Redesign will be benchmarked against realistic use case (complexity and size)

Done

@Konohana0608 (MS) defined a use-case
ITISFoundation/osparc-simcore#3048

Ongoing

ITISFoundation/osparc-simcore#3082: improving API models
ITISFoundation/osparc-simcore#2974: clone handler and long polling operations

Open

clone in storage service (awaiting rclone assessment and 5GB upload feature ITISFoundation/osparc-issues#617)
benchmarks at least with MS use case

pcrespov commented 2 years ago

Update on sprint Diolkos

No progress on this issue during this sprint

sanderegg commented 2 years ago

Update on sprint Brutalism

Done

Summary

Thanks to https://github.com/ITISFoundation/osparc-simcore/pull/3077, https://github.com/ITISFoundation/osparc-simcore/issues/3100, https://github.com/ITISFoundation/osparc-simcore/issues/3144 and https://github.com/ITISFoundation/osparc-simcore/pull/3021 a.k.a. complete refactoring of the storage micro-service the problem with copying studies containing files >5Gb is possible and executed in place.

The webserver micro-service and the frontend were modified to allow to use the same mechanism as "creating a project from a template". The operation which can potentially be very long (depending on the amount of data) has been modified to leverage "long running operations in a REST API" which will allow to report a progress (still ongoing). A way to retrieve these long running operations of a specific user will allow the frontend to be F5-refreshed and recover its list of tasks.

1st version caveats

No progress bar to report study copy
If webbrowser is disconnected from osparc --> copy will be cancelled
If webbrowser is refreshed (F5) --> copy will be cancelled
If webserver is restarted --> copy will be cancelled
Currently limited to studies <= 30GB until tested with more data --> error message displayed

Ongoing/Next steps

Generate progress for copy operation from the backend
Improve UI (auto retries in case of network issue, show copy progress bar, recover copy tasks after refresh or reconnection)
Handle frontend disconnection/refresh
Possibly handle webserver restarts
e2e tests to increase current limit

sanderegg commented 2 years ago

Update on sprint Vaporwave

Done

Ongoing

https://github.com/ITISFoundation/osparc-simcore/pull/3168

Open

Not reproducible: need input from @elisabettai https://github.com/ITISFoundation/osparc-issues/issues/604
In case of web browser refresh or disconnection/later reconnection, currently no recovery of copy tasks (but they will still eventually complete) - will not be implemented for now.

elisabettai commented 2 years ago

Closing this, as discussed with @sanderegg. The only open part is an e2e test, that will be tackled on the side at another point. Note: as of today, this has not been released to production yet (but will be). fyi @esraneufeld

ITISFoundation / osparc-issues