search

dlr-eoc / ukis-pysat

The ukis-pysat package provides generic classes and functions to query, access and process multi-spectral and SAR satellite images
Apache License 2.0
27 stars 8 forks source link

Expose fileobj to stream scenes #132

Closed fwfichtner closed 3 years ago

fwfichtner commented 3 years ago

It would be quite handy to not having to write to disk when we want to download files to S3. With the current solution we could for example do something like that using minio-py

s3client.put_object(bucket_name, object_name, response[0], length=-1, part_size=10 * 1024 * 1024, )

and then verify the checksum. response would be what's returned from self.get_download_bytes(). I guess we leave handling of the response out of this library because who knows what else you would want to do with this.

Let's start with Scihub and handlet the others later, once we are certain how to continue with them.

@j08lue you discussed something like this in https://github.com/sentinelsat/sentinelsat/issues/396, would you want a PR in sentinelsat? In that case we would not merge here.

j08lue commented 3 years ago

Thanks for proposing this, @fwfichtner. If I understand you correctly, you would add a method, e.g. api.get_stream(result) that takes the download link from a response and exposes a requests response object ready for streaming? This sounds like a handy function to have on a SentinelAPI instance and a PR would be more than welcome.

With Copernicus Open Access Hub as an endpoint, you might also need to handle the Long-Term Archive retrieval / trigger logic, which is a bit of a headache. Or you just raise an exception if the product is not available... We can discuss this further on the PR, if you go for that.

fwfichtner commented 3 years ago

Exactly, will do that in the coming days and we can discuss everything else there.