The signed URLs generated for datasets referenced in output manifests expires after 7 days, which denies long term access to datasets. This happens even when the requester has permission to access the output datasets (because unauthenticated signed URLs are used in output manifests instead of authenticated access to gs:// addresses).
What is the expected behavior?
Signed URLs were introduced to allow parents access to output datasets generated by a child which were then stored in the child's bucket where the parent didn't have permission to access that bucket directly (e.g. parents from outside an organisation to access outputs generated by a child inside that organisation).
Proposed Solution
[x] For parents and children with access to the same buckets (e.g. inside the same organisation), there should be an option to use gs:// addresses in the output manifest to allow long term authenticated access to output datasets. This could be switched on via an environment variable, the application configuration, or an argument to Service.ask.
[x] Consider if this should be the default
[x] Add conversion of a signed URL to a gs:// address in the event of failure to retrieve data from it
[x] Raise a better error when trying to access a dataset with an expired signed URL
[x] Update documentation on finalise and app configuration
Bug report
What is the current behavior?
The signed URLs generated for datasets referenced in output manifests expires after 7 days, which denies long term access to datasets. This happens even when the requester has permission to access the output datasets (because unauthenticated signed URLs are used in output manifests instead of authenticated access to
gs://
addresses).What is the expected behavior?
Signed URLs were introduced to allow parents access to output datasets generated by a child which were then stored in the child's bucket where the parent didn't have permission to access that bucket directly (e.g. parents from outside an organisation to access outputs generated by a child inside that organisation).
Proposed Solution
gs://
addresses in the output manifest to allow long term authenticated access to output datasets. This could be switched on via an environment variable, the application configuration, or an argument toService.ask
.Add conversion of a signed URL to ags://
address in the event of failure to retrieve data from itfinalise
and app configuration