Does downloading job results incur a fee?

Open-EO / openeo-api

The openEO API specification

http://api.openeo.org

Apache License 2.0

92 stars 11 forks source link

Does downloading job results incur a fee? #104

Closed m-mohr closed 6 years ago

m-mohr commented 6 years ago

Currently, people are billed by processing costs. Is there anybody who thinks downloading of processed results (and user files?) should be billed, too? If so, we would need to add it, too. This may be of special interest when sharing resources publicly becomes a thing with #85.

mkadunc commented 6 years ago

I think we need to add the option to include billing in download services as well, for at least two reasons:

in case of on-the-fly processing, request for download triggers processing, so the download and processing action are conflated into a single thing from the user's perspective
as you note, if sharing processed result is a use case we want to support (not sure we do), download requests could take up a large portion of the back-end resources; from our experience the cost of data transfer from cloud out into the internet is substantial, on a par with processing costs (AWS EC2 is ~$0.05 per CPU hour, $0.09 per 1GB output traffic)

m-mohr commented 6 years ago

@mkadunc Who would be paying for downloading? The user who has generated the results or the one who is downloading the results?

mkadunc commented 6 years ago

I'm not sure OpenEO will go into this much details, but I would imagine that both scenarios are possible.

"owner pays" - in this case the user that defined the on-the-fly job, or the one that generated the result, covers the cost of downloading. No extra API parameters or headers are required for this use case (the user who is downloading should not see the costs anywhere)
"requester pays" - similar to cloud providers', the user that downloads the result needs to have its own account with the backend provider and indicates her willingness to cover the costs of the download in the header or parameter of the download request (https://docs.aws.amazon.com/AmazonS3/latest/dev/RequesterPaysBuckets.html)

The difference between the two options will only be evident when the back-end supports enough security configuration to allow 3rd parties (or unauthenticated general public) to access results. Initial setup of all back-ends will probably only allow owners to download job results.

m-mohr commented 6 years ago

So, I'll propose this model for downloading:

Downloading synchronously processed data (POST /preview) is always included in the OpenEO-Costs header returned. Preview is only used by the owner and can't be shared so there is no problem with "requester pays".
The estimates for batch jobs (GET /jobs/:job_id/estimate) gets a new property downloads_included which specifies how many full downloads of the processed data are included in the estimate. (GET /jobs/:job_id/results) has a new header that indicates the costs for downloading. If a requester other than the owner is requesting the data of a shared job this header indicates the costs for the requester. When sharing jobs the owner can choose who needs to pay. If the requester is requested to pay, access is denied for unauthorized users.
Shared web services are usually paid by the owner, but this highly depends on the web service type and whether it supports to charge fees or not.