Issues with File Upload and Retrieval using GCP

[alexdess]

Describe the bug I am encountering several issues when using file upload and retrieval through Google Cloud Platform (GCP). After reading the existing issues and setting up my bucket and configuration files, I am facing the following errors:

1. Issue with direct ByteData upload: When I try to upload a ByteData object to a custom endpoint, I get the following error: Connection closed before full header was received, uri = http://localhost:8080/[endpointName] The test was performed with a png of 1.6MBytes.

2. Issue with file permissions after upload: When using the FileUpload component on the client side, the file uploads successfully but is made public, even though I have configured the bucket for private files (public: false).

3. Issue with server-side file retrieval: When I try to retrieve a file server-side, it only works if the file is at the root of the bucket. For example:

   • succursale/logo.png does not work.
   • logo.png works.

This happens with the retrieveFile and getPublicURL methods.

To Reproduce Steps to reproduce the behavior:

1. Configure a GCP bucket with private permissions (public: false).
2. Try uploading a ByteData object to an endpoint.
3. Use the FileUpload component to upload a file.
4. Attempt to retrieve a file in a subdirectory of the bucket.

Expected behavior

1. Upload ByteData without errors: Uploading a ByteData object to a custom endpoint should proceed without any errors, and the connection should not be interrupted.

2.. Respect file privacy settings during upload: When I upload a file using the FileUpload method, the file should be stored as private in GCP, as configured (public: false).

3. File retrieval, including those in subdirectories: Files should be retrievable using the retrieveFile and getPublicURL methods, whether they are at the root of the bucket or in a subpath.

Serverpod versions

• CLI Version: 2.1.1
• Library version: 2.1.1

Platform information Db and redis on Docker. Server and client on localhost using vscode debugger

Additional context

client code

```dart Future _updateLogoDirect(BuildContext context, String id) async { var succId = int.tryParse(id); if (succId == null) { return; } // ask user to upload image FilePickerResult? result = await FilePicker.platform.pickFiles( allowMultiple: false, type: FileType.image, ); if (result == null) { return; } var file = result.files.single; var bytes = await file.xFile.readAsBytes(); if (bytes == null) { return; } var byteData = bytes.buffer.asByteData(); await client.succursale.setLogo(succId, byteData); } Future _updateLogoFileUploader(BuildContext context, String id) async { var succId = int.tryParse(id); if (succId == null) { return; } // ask user to upload image FilePickerResult? result = await FilePicker.platform.pickFiles( allowMultiple: false, type: FileType.image, ); if (result == null) { return; } var file = result.files.single; var bytes = await file.xFile.readAsBytes(); if (bytes == null) { return; } var logoStoragePath = await client.succursale.getLogoStoragePath(succId); if (logoStoragePath == null) { return; } FileUploader uploader = FileUploader(logoStoragePath); var byteData = bytes.buffer.asByteData(); uploader.uploadByteData(byteData); } ```

server code

```dart Future getLogo(Session session, int id) async { var succursale = await Succursale.db.findById(session, id); var logoPath = succursale?.logoPath; logoPath = _getLogoPath(succursale!); if (logoPath == null) { return null; } var logo = await session.storage .retrieveFile(path: logoPath, storageId: 'private'); var publicUrl = await session.storage .getPublicUrl(storageId: 'private', path: logoPath); var object = session.storage.retrieveFile(storageId: 'private', path: logoPath); return logo; } // set logo Future setLogo(Session session, int id, ByteData logo) async { var succursale = await Succursale.db.findById(session, id); if (succursale == null) { throw Exception('Succursale not found'); } var logoPath = _getLogoPath(succursale); await session.storage .storeFile(storageId: "private", path: logoPath, byteData: logo); succursale.logoPath = logoPath; return await Succursale.db.updateRow(session, succursale); } //getLogoPath Future getLogoStoragePath(Session session, int id) async { var succursale = await Succursale.db.findById(session, id); if (succursale == null) { return null; } var logoPath = _getLogoPath(succursale); var storagePath = await session.storage.createDirectFileUploadDescription( storageId: 'private', path: logoPath); return storagePath; } String _getLogoPath(Succursale succursale) { return '/toto/logo_1_asdf.png'; } ```

[Isakdl]

Thank you for the detailed bug report!

We need to take a pass on the bucket implementation. There is already a PR for a complete new implementation of the GCP bucket using the real google endpoints rather than AWS compatibility layer. You can find the PR here: https://github.com/serverpod/serverpod/pull/2745.

The final implementation may differ from what is in there right now, but that code does work and should solve most if not all the above mentioned issues.

[alexdess]

Thanks for your reply. PR looks promising What about uploading ByteData files from the client to the server. Is this the right approach, or is it not planned to be able to transit files in this way?

[Isakdl]

This is possible today already. You can use the file upload API, there is only one serverside integration for this and it writes the file to the db.

If you want to manage this in memory you can set the maxRequestSize in the config to increase the byte limit in the endpoints. See: https://docs.serverpod.dev/concepts/configuration#configuration-options

[iampopal]

Thank you for opening this issue. I have been also facing this using GCP Storage.

1) I get the file description and the file get uploaded. But it returns success: false.

2) The verify File exists and getPublicUrl is not working correctly.

3) The privacy concerns that every person shall only see their own files is very important.

Happy seeing there will be progress. When we can get the new implementation because it's required for our app. Thank you.