octue / octue-sdk-python

The python SDK for @Octue services and digital twins.
https://octue.com
Other
9 stars 4 forks source link

Access to output datasets via output manifest expires after 7 days #675

Closed cortadocodes closed 2 months ago

cortadocodes commented 2 months ago

Bug report

What is the current behavior?

The signed URLs generated for datasets referenced in output manifests expires after 7 days, which denies long term access to datasets. This happens even when the requester has permission to access the output datasets (because unauthenticated signed URLs are used in output manifests instead of authenticated access to gs:// addresses).

What is the expected behavior?

Signed URLs were introduced to allow parents access to output datasets generated by a child which were then stored in the child's bucket where the parent didn't have permission to access that bucket directly (e.g. parents from outside an organisation to access outputs generated by a child inside that organisation).

Proposed Solution

cortadocodes commented 2 months ago

@thclark I'm interested to know if you think using gs:// paths should be the default? I think it makes sense for most use cases of the sdk