Create a service to access AWS S3 Kargo storage

claustres commented 4 years ago

Will be used to access archived meteo data for instance or private GeoJson files.

claustres commented 4 years ago

Could be based on the core storage service with appropriate rights, e.g.

defineAbilities.registerHook(function defineUserAbilities (subject, can, cannot) {
  can(['read'], 'storage', { _id: { $regex: '^archive/' } })
})

However we face some issues on the underlying https://github.com/feathersjs-ecosystem/feathers-blob module:

claustres commented 4 years ago

We could also try to use the new API gateway in Kargo and create a dedicated plugin, e.g. the following code allows to proxy S3 objects:

var aws = require('aws-sdk')

const s3 = new aws.S3({
  accessKeyId: process.env.S3_ACCESS_KEY,
  secretAccessKey: process.env.S3_SECRET_ACCESS_KEY
})
app.get('/s3/:bucket/*', (req, res) => {
  s3.getObject({
    Bucket: req.params.bucket,
    Key: req.params[0]
  }).createReadStream().pipe(res)
})

We should then use URLs like https://api.subdomain/s3/kargo/archive/xxx.cog?jwt=xxx, see related issue https://github.com/kalisio/kano/issues/92.

claustres commented 4 years ago

Closing in favor of https://github.com/kalisio/kano/issues/92

kalisio / kano

Create a service to access AWS S3 Kargo storage #98