5Gb limit

[esraneufeld]

Problem:

AWS S3 has a 5 GB limit for a sinlge put operation. The single pre-signed link strategy fails. This affects:

• Uploads through filepicker
• Uploads through nodeports
• Uploads from comp. backend

Remedy:

• rclone instad of nodeports?
• temporary S3 credentials for uplaods into specific subfolder of the bucket?
• multiple upload links?

  
  ### Baklava

• [ ] https://github.com/ITISFoundation/osparc-simcore/issues/4640
• [ ] ITISFoundation/osparc-simcore#3217

[newton1985]

Does "uploads from comp. backend" also cover the case of saving work.zip to S3 when a study is closed? Afaik, this is also affected.

[sanderegg]

Update on sprint Macarons

Done

• Creation of S3 links instead of single presigned links by the storage service: https://github.com/ITISFoundation/osparc-simcore/pull/3004
• Computational sidecar understands S3 links: https://github.com/ITISFoundation/osparc-simcore/pull/3006

Ongoing

• Computational sidecars in internal cluster use S3 links: https://github.com/ITISFoundation/osparc-simcore/pull/3006
• Modification of dynamic sidecar to use RClone library to synchronise state files and inputs/outputs using direct S3 access: https://github.com/ITISFoundation/osparc-simcore/pull/2993

Open

• Creation of temporary S3 credentials per user (limited to 1000 concurrent temp credentials by AWS)
• Investigate multipart upload with presigned links (for frontend, and potentially for computational sidecars in external clusters)
• Implement solution for frontend/external computational sidecars

[sanderegg]

Update on sprint Croissant

Done

• Computational sidecars in internal cluster use S3 links: https://github.com/ITISFoundation/osparc-simcore/pull/3006
• Modification of dynamic sidecar to use RClone library to synchronise state files and inputs/outputs using direct S3 access: https://github.com/ITISFoundation/osparc-simcore/pull/2993
• Creation of temporary S3 credentials per user (limited to 1000 concurrent temp credentials by AWS)
• Investigate multipart upload with presigned links (for frontend, and potentially for computational sidecars in external clusters)
• ITISFoundation/osparc-simcore/pull/3077

  Ongoing

  Implement multipart upload links to break 5Gb limit:

• https://github.com/ITISFoundation/osparc-simcore/pull/3021 (FilePicker)
  • First tests allowed to upload >10Gb file over the file picker
  • Backup solution for dynamic sidecar if RClone solution is too unstable

    Open

• Implement solution for frontend/external computational sidecars
• Implement solution for public API

[sanderegg]

Update on sprint Diolkos

Done

• https://github.com/ITISFoundation/osparc-simcore/issues/3100
• https://github.com/ITISFoundation/osparc-simcore/issues/3144

  Ongoing

• https://github.com/ITISFoundation/osparc-simcore/pull/3021

[sanderegg]

Update on sprint Meteora

Done

• ITISFoundation/osparc-simcore#3021
• ITISFoundation/osparc-simcore#3166

  Ongoing

• ITISFoundation/osparc-simcore#3224

  Open

• ITISFoundation/osparc-simcore#3217

[esraneufeld]

todo

• fix bug in API that limits to 2GB

[pcrespov]

Update on sprint Brutalism

No progress this sprint on this issue

Open

• fix bug in API that limits to 2GB
• ITISFoundation/osparc-simcore#3217

[sanderegg]

Update on sprint Vaporwave

Done

• https://github.com/ITISFoundation/osparc-issues/issues/705

Ongoing

• https://github.com/ITISFoundation/osparc-simcore/issues/3217
• fix bug that limits to 2GB (workaround exists)

Open

• https://github.com/ITISFoundation/osparc-simcore/pull/3224 Abandonned in favor of https://github.com/ITISFoundation/osparc-simcore/issues/3223, https://github.com/ITISFoundation/osparc-simcore/issues/4540

[esraneufeld]

@sanderegg dear sylvain. is there an update on this one?

[sanderegg]

@sanderegg dear sylvain. is there an update on this one?

@esraneufeld no there isn't we decided in PM1 to concentrate on other priorities, since this one has a workaround and that we concentrate on dynamic services at the moment.

[newton1985]

@esraneufeld - just to add some detail, the workaround is to use the "curl" command's "put" feature. It's a substitution of 1 or 2 lines of code, and seems to work just fine without any drawbacks. Though at some point it would be nice to fix it, since the average user won't know the trick.

[matusdrobuliak66]

@matusdrobuliak66 Investigate options AWS temporary credientials for S3

[sanderegg]

Current status is that presigned link are created with a very long valid time. this is not optimal. Best would be to use the osparc API directly, therefore we would not rely on a hard-coded time value. it would also be more secure.