[Amplify Storage] What are we supposed to pass to authorization header?

delfme commented 1 year ago

Description

Can you please explain why I cannot pass auth token inside headers when using a standard http client?

...
        headers: {
          'authorization': 'Bearer $sessionToken',

For sessionToken I tried both: 1) final sessionToken = await auth.credentialsResult.value.sessionToken!; and 2)

  final auth = await Amplify.Auth.fetchAuthSession() as CognitoAuthSession;
  final sessionToken = (auth as CognitoAuthSession).userPoolTokensResult.value.idToken;

With 1) I get api auth error (so I guess I'm not supposed to pass that token)

And with 2) I get [ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: SignedOutException { "message": "No user is currently signed in" } which is thrown on userPoolTokensResult.value.idToken;

Note: I use cognito with unauthenticated user.

Steps to Reproduce

No response

Screenshots

No response

Platforms

[X] iOS
[X] Android
[ ] Web
[ ] macOS
[ ] Windows
[ ] Linux

Flutter Version

Channel master, 3.17.0-1.0.pre.37

Amplify Flutter Version

1.6.0

Deployment Method

Amplify CLI

Schema

No response

Jordan-Nelson commented 1 year ago

Hello @delfme - I am not quite sure if I follow what you are trying to do. If you use Amplify Auth and Amplify Storage the authorization header should be added to the request.

Are you trying to access S3 via an http client instead of using Amplify Storage?

delfme commented 1 year ago

Are you trying to access S3 via an http client instead of using Amplify Storage?

Yes. Im checking performance and trying with http client.

      Dio dio = new Dio();
      var response = await dio.put(
        url,
        data: file.openRead(),
        onSendProgress: (int sent, int total) {
          debugPrint('File upload progress $sent of total $total');
        },
        options: Options(
          contentType: "application/octet-stream", 
        headers: {
          'authorization': 'Bearer $sessionToken',
         // 'x-amz-acl' : 'public-read',
         // Headers.contentLengthHeader: '$len',
        },)
    );
    debugPrint('Time elapsed to upload '
        '${DateTime.timestamp().difference(startTime).inMilliseconds}ms');

If I make the bucket private, I need to authorize the http call. I'm passing token, taken from auth.credentialsResult.value.sessionToken! and userPoolTokensResult.value.idToken (not sure which one I should pass), but I get above errors.

Jordan-Nelson commented 1 year ago

Hi @delfme - If you would like to upload a file to S3 via an http request you will need to sign the request (for more info on signing requests see: https://docs.aws.amazon.com/AmazonS3/latest/API/sig-v4-authenticating-requests.html). You can use the Dart AWSSigV4Signer to sign requests. There is an example of that here - https://github.com/aws-amplify/amplify-flutter/blob/next/packages/aws_signature_v4/example/bin/example.dart.

Another option would be to upload the file with a pre-signed URL (see: https://docs.aws.amazon.com/AmazonS3/latest/userguide/PresignedUrlUploadObject.html). This would not be an accurate performance comparison though as it is a different request.

Let me know if you have any other questions.

delfme commented 1 year ago

Thx @Jordan-Nelson I try this. I tried presignedUrl but was not working, got a 403 error. Do you have a snipet where: 1) a presignedUrl is created from an unauthenticated user via flutter Amplify sdk or satellite plugins 2) file is upload via http request by passing the presigneUrl ?

delfme commented 1 year ago

https://github.com/aws-amplify/amplify-flutter/blob/next/packages/aws_signature_v4/example/bin/example.dart

This gives me error

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: InvalidCredentialsException: Could not load credentials for profile "default"
#0      ProfileCredentialsProvider.retrieve (package:aws_common/src/credentials/aws_credentials_provider.dart:199:7)
<asynchronous suspension>
#1      AWSSigV4Signer.sign.<anonymous closure> (package:aws_signature_v4/src/signer/aws_signer.dart:108:27)
<asynchronous suspension>

delfme commented 1 year ago

I tried also enviroment credential as instructed here but doesn't work.

@Jordan-Nelson would you be so kind as to provide a working sample for first method and also for the presigneUrl method?

Jordan-Nelson commented 1 year ago

Hello @delfme - See: https://docs.amplify.aws/lib/storage/download/q/platform/flutter/#generate-a-download-url

Let me know if you have any other questions.

delfme commented 12 months ago

Hello @Jordan-Nelson That is not addressing question. I guess you meant to type this to that other issue of mine about download.

This issue is about upload file using http request (not storage). That would resolve issue https://github.com/aws-amplify/amplify-flutter/issues/4105#issuecomment-1806566560

delfme commented 12 months ago

@Jordan-Nelson hello! Any news on this?

Jordan-Nelson commented 10 months ago

https://github.com/aws-amplify/amplify-flutter/blob/next/packages/aws_signature_v4/example/bin/example.dart

This gives me error

[ERROR:flutter/runtime/dart_vm_initializer.cc(41)] Unhandled Exception: InvalidCredentialsException: Could not load credentials for profile "default"
#0      ProfileCredentialsProvider.retrieve (package:aws_common/src/credentials/aws_credentials_provider.dart:199:7)
<asynchronous suspension>
#1      AWSSigV4Signer.sign.<anonymous closure> (package:aws_signature_v4/src/signer/aws_signer.dart:108:27)
<asynchronous suspension>

The example will automatically attempt to load credentials from the default profile on your machine. If you do not have these set you can pass them in manually.

  const signer = AWSSigV4Signer(
    credentialsProvider: AWSCredentialsProvider(
      AWSCredentials('accessKey', 'secretAccessKey'),
    ),
  );

Let me know if you have any other questions.

delfme commented 10 months ago

Thx @Jordan-Nelson , I think I tried this. Can we have working sample from you guys?

Jordan-Nelson commented 10 months ago

@delfme I just verified that the script worked for me when adding the credentials manually with the code below. If you are receiving an error please share the error here.

  const signer = AWSSigV4Signer(
    credentialsProvider: AWSCredentialsProvider(
      AWSCredentials(
        '<Your accessKeyId>',
        '<Your secretAccessKey>',
        '<Your sessionToken>',
      ),
    ),
  );

delfme commented 10 months ago

Can you please share the whole code?

Jordan-Nelson commented 10 months ago

// Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
// SPDX-License-Identifier: Apache-2.0

import 'dart:convert';
import 'dart:io';
import 'dart:math';

import 'package:args/args.dart';
import 'package:aws_common/aws_common.dart';
import 'package:aws_signature_v4/aws_signature_v4.dart';
import 'package:collection/collection.dart';
import 'package:path/path.dart' as p;

// This examples walks through the creation of an S3 bucket and the process of
// uploading a file to that bucket and retrieving a pre-signed URL for reading
// back its contents.

Future<void> main(List<String> args) async {
  final argParser = ArgParser();

  const bucketArg = 'bucket';
  const regionArg = 'region';

  argParser
    ..addOption(
      bucketArg,
      abbr: 'b',
      help: 'The name of the bucket to create',
      valueHelp: 'BUCKET',
      mandatory: false,
    )
    ..addOption(
      regionArg,
      abbr: 'r',
      help: 'The region of the bucket',
      valueHelp: 'REGION',
      mandatory: true,
    );

  final parsedArgs = argParser.parse(args);

  final bucket = parsedArgs[bucketArg] as String? ??
      'mybucket-${Random().nextInt(1 << 30)}';
  final region = parsedArgs[regionArg] as String;
  final filename = parsedArgs.rest.singleOrNull;

  if (filename == null) {
    exitWithError(
      'Usage: dart s3_example.dart --region=... <FILE_TO_UPLOAD>',
    );
  }

  const signer = AWSSigV4Signer(
    credentialsProvider: AWSCredentialsProvider(
      AWSCredentials(
        '<Your accessKeyId>',
        '<Your secretAccessKey>',
        '<Your sessionToken>',
      ),
    ),
  );

  // Set up S3 values
  final scope = AWSCredentialScope(
    region: region,
    service: AWSService.s3,
  );
  final host = '$bucket.s3.$region.amazonaws.com';
  final serviceConfiguration = S3ServiceConfiguration();

  // Create the bucket
  final createBody = utf8.encode(
    '''
<CreateBucketConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<LocationConstraint>$region</LocationConstraint>
</CreateBucketConfiguration>
''',
  );
  final createRequest = AWSHttpRequest.put(
    Uri.https(host, '/'),
    body: createBody,
    headers: {
      AWSHeaders.host: host,
      AWSHeaders.contentLength: createBody.length.toString(),
      AWSHeaders.contentType: 'application/xml',
    },
  );

  stdout.writeln('Creating bucket $bucket...');
  final signedCreateRequest = await signer.sign(
    createRequest,
    credentialScope: scope,
    serviceConfiguration: serviceConfiguration,
  );
  final createResponse = await signedCreateRequest.send().response;
  final createStatus = createResponse.statusCode;
  stdout.writeln('Create Bucket Response: $createStatus');
  if (createStatus == 409) {
    exitWithError('Bucket name already exists!');
  }
  if (createStatus != 200) {
    exitWithError('Bucket creation failed');
  }
  stdout.writeln('Bucket creation succeeded!');

  // Upload the file
  final file = File(filename).openRead();
  final path = '/${p.basename(filename)}';
  final uploadRequest = AWSStreamedHttpRequest.put(
    Uri.https(host, path),
    body: file,
    headers: {
      AWSHeaders.host: host,
      AWSHeaders.contentType: 'text/plain',
    },
  );

  stdout.writeln('Uploading file $filename to $path...');
  final signedUploadRequest = await signer.sign(
    uploadRequest,
    credentialScope: scope,
    serviceConfiguration: serviceConfiguration,
  );
  final uploadResponse = await signedUploadRequest.send().response;
  final uploadStatus = uploadResponse.statusCode;
  stdout.writeln('Upload File Response: $uploadStatus');
  if (uploadStatus != 200) {
    exitWithError('Could not upload file');
  }
  stdout.writeln('File uploaded successfully!');

  // Create a pre-signed URL for downloading the file
  final urlRequest = AWSHttpRequest.get(
    Uri.https(host, path),
    headers: {
      AWSHeaders.host: host,
    },
  );
  final signedUrl = await signer.presign(
    urlRequest,
    credentialScope: scope,
    serviceConfiguration: serviceConfiguration,
    expiresIn: const Duration(minutes: 10),
  );
  stdout.writeln('Download URL: $signedUrl');
}

/// Exits the script with an [error].
Never exitWithError(String error) {
  stderr.writeln(error);
  exit(1);
}

delfme commented 10 months ago

Thx @Jordan-Nelson

It is some time for me not playing this code, can you LMK how to fetch that params with flutter Amplify ?

AWSCredentials(
        '<Your accessKeyId>',
        '<Your secretAccessKey>',
        '<Your sessionToken>',
      ),

Jordan-Nelson commented 10 months ago

Hi @delfme - Please see the docs here: https://docs.amplify.aws/flutter/build-a-backend/auth/accessing-credentials/#retrieving-aws-credentials

Please note that the example creates a new S3 bucket, which requires S3 access that an amplify end user would not have. If you want to fetch the credentials using Amplify you could modify the script to use an S3 bucket that was added via Amplify Storage.

delfme commented 10 months ago

Hello @Jordan-Nelson

I’m a bit confused here and I think I also didn't explain my need correctly.

I'm uploading files via Amplify storage 1)

final uploadOperation = await Amplify.Storage.uploadFile(
        localFile: AWSFile.fromPath(file.path),
        key: key,
        onProgress: (progress) {
          debugPrint(
            '$_debugPrefix $key upload fraction completed: ${progress.fractionCompleted}',
          );
          if (eventOnProgress != null) {
            eventOnProgress(progress.fractionCompleted);
          }
        },
      ).result;

I'm somehow hit by slow upload performance when using Storage package and would like to test the upload of the same file via dio or native plugins.

2)

final getPresignedUrl = await Amplify.Storage.getUrl(key: key).result;
final presignedUrl = getPresignedUrl.url.toString();

Dio dio = new Dio();
      final response = await dio.put(
        presignedUrl,
        data: file.openRead(),
        onSendProgress: (int sent, int total) {
          debugPrint('File upload progress $sent of total $total');
        },
        options: Options(
          contentType: "application/octet-stream", 
        headers: {
          'authorization': 'Bearer $sessionToken',
        },)
    );

My rationale is that if I get the presignedUrl bound to a key, I can then use whatever http client to upload the file. Unfortunately code (2) doesn't work due to auth issues. Could you please share a solution to this problem that only requires to know key + AWS Amplify + maybe additional AWS flutter plugins?

Jordan-Nelson commented 9 months ago

Hi @delfme

Amplify.Storage.getUrl() will generate a presigned URL that can be used for downloads only, not uploads. Apologies for pointing you in that direction. I think I had gotten mixed up and thought you wanted a download URL.

Preseigned URLs are specific to one HTTP method. So you will need to generate a presigned URL for PUT requests. Amplify doesn't have an API for that as files can be uploaded the uploadFile/uploadData APIs, but you can use the Dart AWS Signer to generate one. Below is a code example of how you could use the AWS Signer to generate a presigned URL that can be used for uploads.

  Future<AWSSigV4Signer> getAwsCredentialsSigner() async {
    final plugin = Amplify.Auth.getPlugin(AmplifyAuthCognito.pluginKey);
    final session = await plugin.fetchAuthSession();
    final provider = AWSCredentialsProvider(session.credentialsResult.value);
    return AWSSigV4Signer(credentialsProvider: provider);
  }

  Future<Uri> getPresignedUri({required String key}) async {
    final jsonConfig = jsonDecode(amplifyconfig) as Map<String, Object?>;
    final config = AmplifyConfig.fromJson(jsonConfig);
    final bucket = config.storage!.awsPlugin!.bucket;
    final region = config.storage!.awsPlugin!.region;
    final host = '$bucket.s3.$region.amazonaws.com';
    final path = 'public/${p.basename(key)}'; // Note that this will upload files to the PUBLIC folder which is equivalent to using StorageAccessLevel.guest
    final urlRequest = AWSHttpRequest.put(
      Uri.https(host, path),
      headers: {
        AWSHeaders.host: host,
        AWSHeaders.contentType: 'application/json; charset=utf-8',
      },
    );
    final signer = await getAwsCredentialsSigner();
    final signedUrl = await signer.presign(
      urlRequest,
      credentialScope: AWSCredentialScope(
        region: region,
        service: AWSService.s3,
      ),
      serviceConfiguration: S3ServiceConfiguration(),
      expiresIn: const Duration(minutes: 10),
    );
    return signedUrl;
  }

  Future<void> uploadWithPresignedUrl({required String key}) async {
    try {
      final presignedUrl = await getPresignedUri(key: key);
      await dio.put<Object>(
        presignedUrl.toString(),
        data: {'uploaded_method': 'presigned URL'},
        options: Options(
          contentType: 'application/json; charset=utf-8',
        ),
      );
    } on Exception catch (e) {
      _logger.debug('Could not upload file: $e');
    }
  }

I created a sample app to test this out and also tested the timing a little bit on a few different devices. The results vary a bit by device, but I see similar request times for the Amplify Lib and for presigned URLs.

Full App

```dart // Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved. // SPDX-License-Identifier: Apache-2.0 import 'dart:convert'; import 'package:amplify_auth_cognito/amplify_auth_cognito.dart'; import 'package:amplify_authenticator/amplify_authenticator.dart'; import 'package:amplify_core/amplify_core.dart'; import 'package:amplify_storage_s3/amplify_storage_s3.dart'; import 'package:amplify_storage_s3_example/amplifyconfiguration.dart'; import 'package:aws_signature_v4/aws_signature_v4.dart'; import 'package:dio/dio.dart'; import 'package:flutter/material.dart'; import 'package:go_router/go_router.dart'; import 'package:path/path.dart' as p; final AmplifyLogger _logger = AmplifyLogger('MyStorageApp'); final dio = Dio(); void main() { AmplifyLogger().logLevel = LogLevel.debug; runApp(const MyApp()); } class MyApp extends StatefulWidget { const MyApp({super.key}); @override State createState() => _MyAppState(); } class _MyAppState extends State { static final _router = GoRouter( routes: [ GoRoute( path: '/', builder: (BuildContext _, GoRouterState __) => const HomeScreen(), ), ], ); @override void initState() { super.initState(); configureAmplify(); } Future configureAmplify() async { final auth = AmplifyAuthCognito(); final storage = AmplifyStorageS3(); try { await Amplify.addPlugins([auth, storage]); await Amplify.configure(amplifyconfig); _logger.debug('Successfully configured Amplify'); } on Exception catch (error) { _logger.error('Something went wrong configuring Amplify: $error'); } } @override Widget build(BuildContext context) { return Authenticator( preferPrivateSession: true, child: MaterialApp.router( title: 'Flutter Demo', builder: Authenticator.builder(), theme: ThemeData.light(useMaterial3: true), darkTheme: ThemeData.dark(useMaterial3: true), routeInformationParser: _router.routeInformationParser, routerDelegate: _router.routerDelegate, debugShowCheckedModeBanner: false, ), ); } } class HomeScreen extends StatefulWidget { const HomeScreen({super.key}); @override State createState() => _HomeScreenState(); } class _HomeScreenState extends State { String fileData = 'NONE'; @override void initState() { super.initState(); } Future getAwsCredentialsSigner() async { final plugin = Amplify.Auth.getPlugin(AmplifyAuthCognito.pluginKey); final session = await plugin.fetchAuthSession(); final provider = AWSCredentialsProvider(session.credentialsResult.value); return AWSSigV4Signer(credentialsProvider: provider); } Future getPresignedUri({required String key}) async { final jsonConfig = jsonDecode(amplifyconfig) as Map; final config = AmplifyConfig.fromJson(jsonConfig); final bucket = config.storage!.awsPlugin!.bucket; final region = config.storage!.awsPlugin!.region; final host = '$bucket.s3.$region.amazonaws.com'; final path = 'public/${p.basename(key)}'; final urlRequest = AWSHttpRequest.put( Uri.https(host, path), headers: { AWSHeaders.host: host, AWSHeaders.contentType: 'application/json; charset=utf-8', }, ); final signer = await getAwsCredentialsSigner(); final signedUrl = await signer.presign( urlRequest, credentialScope: AWSCredentialScope( region: region, service: AWSService.s3, ), serviceConfiguration: S3ServiceConfiguration(), expiresIn: const Duration(minutes: 10), ); return signedUrl; } Future uploadWithPresignedUrl({required String key}) async { try { final timer = Timer(name: 'Upload File with presigned URL'); final presignedUrl = await getPresignedUri(key: key); await dio.put

aws-amplify / amplify-flutter