Allow export raster datasets and include them on the map file

[apercas]

To be included on https://github.com/splashblot/dronedb/pull/59/

We'll need a way to download the original .tif file the user uploads to Tileo. Right now this functionality does exist for .csvs and other files as such but not binary files. This will allow us to add them on the current .carto file we create when we export a whole map. Either download the original file or add a link where the file is stored so we can get it while importing that .carto object, but definitely bypassing the DB, I guess. Ideas on that last bit welcomed, @ivanprado @jjmata .

[jjmata]

I personally like the idea of storing the .tif in the filesystem beyond the temporary copy that Carto keeps for now. This would make it so rasters can be manipulated later for purposes beyond upload to Postgres and pgraster ... workloads like ML inference will always probably need the original raster anyway. So :+1: for this (which probably means that we need a "metadata" field that is associated with raster datasets and tells us where the file lives in the filesystem ...)

[ivanprado]

+1 to the idea of preserving original raster file. But I wouldn't maintain files at local filesystem, as it would break the failover/scalability of the architecture. Imagine the case where several servers are running behind a load balancer with sticky session (typical production deployment that we eventually should have). If for some reason user is changed to use a different server, then exporting won't work. So I would bet by some of the following options:

• Storing tif files at S3 and keep link at database
• Storing tif files at Postgres as blobs (https://wiki.postgresql.org/wiki/BinaryFilesInDB)

[apercas]

Yes, a link to the original file was the first thing I think of. Storing it in an AWS S3 bucket makes sense to me for what @ivanprado said. I think it's more reliable than a DB or the FS.

About what to include on the exported .carto file, the metadata with the link or the .tif itself? We'll check how to upload/import the file once we download it.

Thanks!

[ivanprado]

The problem of include just the metadata on the carto file is that then tif file at S3 should be public, which I think it is not generally acceptable.

[apercas]

It depends on how we develop it, if we are the ones that download it when it's importing the file, we can handle the security better than just giving a url to the user in order to download the file.

[ivanprado]

You are right. I was thinking on the possibility for the user to recover the tif file, but probably .carto file was not the right tool for that as it was thought mainly for importing/exporting capabilities. If recovering original tif file is a requested feature in the future, it could be implemented independently using the same S3 file (downloading it previously from S3 and then streaming it to the user).