[firebase_storage]: Out of memory error when uploading large files (1GB) on newer versions of firebase_storage and firebase_core

[Fudal]

Is there an existing issue for this?

• [X] I have searched the existing issues.

Which plugins are affected?

Storage

Which platforms are affected?

Android, iOS

Description

I am experiencing an issue with the firebase_storage library in my Flutter app. The problem occurs when I try to upload large files (around 1GB) to Firebase Storage.

When using the following versions of the libraries:

firebase_storage: 11.7.7 firebase_core: 2.32.0 I consistently encounter an "out of memory" error and my app crashes when trying to upload the 12th recording (around 1GB).

However, with the following older versions of the libraries:

firebase_storage: 11.3.1 firebase_core: 2.19.0 I am able to upload up to 30 recordings (1GB each) without any issues or crashes.

This problem seems to be specific to the newer versions of the firebase_storage and firebase_core libraries. Every version above 11.3.1 of firebase_storage and 2.19.0 of firebase_core appears to trigger the same "out of memory" error and app crashes.

I would appreciate if the maintainers of the firebase_storage library could look into this issue and provide a fix or workaround, as the newer versions of the library are crucial for keeping my app up-to-date.

Reproducing the issue

import 'dart:async';
import 'dart:io';
import 'package:firebase_core/firebase_core.dart';
import 'package:firebase_storage/firebase_storage.dart';
import 'package:flutter/material.dart';
import 'package:path_provider/path_provider.dart';

void main() async {
  WidgetsFlutterBinding.ensureInitialized();
  await Firebase.initializeApp();
  runApp(MyApp());
}

class MyApp extends StatelessWidget {
  @override
  Widget build(BuildContext context) {
    return MaterialApp(
      home: UploadScreen(),
    );
  }
}

class UploadScreen extends StatefulWidget {
  @override
  _UploadScreenState createState() => _UploadScreenState();
}

class _UploadScreenState extends State<UploadScreen> {
  List<File> _filesToUpload = [];
  int _uploadedFiles = 0;
  double _progress = 0.0;

  @override
  void initState() {
    super.initState();
    _prepareLargeFiles();
  }

  Future<void> _prepareLargeFiles() async {
    final directory = await getTemporaryDirectory();
    for (int i = 0; i < 15; i++) {
      final file = File('${directory.path}/large_file_$i.bin');
      await file.writeAsBytes(List.generate(1024 * 1024 * 1024, (index) => 0)); // 1GB file
      _filesToUpload.add(file);
    }
    setState(() {});
  }

  Future<void> _uploadFiles() async {
    for (var file in _filesToUpload) {
      if (!mounted) return;
      await _uploadFile(file);
      setState(() {
        _uploadedFiles++;
        _progress = _uploadedFiles / _filesToUpload.length;
      });
    }
  }

  Future<void> _uploadFile(File file) async {
    final ref = FirebaseStorage.instance.ref().child('large_files/${file.path.split('/').last}');
    final uploadTask = ref.putFile(file);

    uploadTask.snapshotEvents.listen((TaskSnapshot snapshot) {
      if (snapshot.state == TaskState.success) {
        print('File uploaded successfully: ${file.path}');
      }
    }, onError: (error) {
      print('Error uploading file: $error');
    });

    await uploadTask;
  }

  @override
  Widget build(BuildContext context) {
    return Scaffold(
      appBar: AppBar(title: Text('Firebase Storage Upload Test')),
      body: Center(
        child: Column(
          mainAxisAlignment: MainAxisAlignment.center,
          children: [
            Text('Files prepared: ${_filesToUpload.length}'),
            Text('Files uploaded: $_uploadedFiles'),
            LinearProgressIndicator(value: _progress),
            ElevatedButton(
              onPressed: _uploadFiles,
              child: Text('Start Upload'),
            ),
          ],
        ),
      ),
    );
  }
}

dependencies:
  flutter:
    sdk: flutter
  firebase_core: ^2.32.0
  firebase_storage: ^11.7.7
  path_provider: ^2.1.2

Firebase Core version

11.7.7

Flutter Version

3.24.3

Relevant Log Output

I/labs.appname(10128): Starting a blocking GC Alloc
I/labs.appname(10128): Starting a blocking GC Alloc
I/labs.appname(10128): Alloc young concurrent copying GC freed 8954KB AllocSpace bytes, 3(60KB) LOS objects, 2% free, 375MB/384MB, paused 108us,41us total 19.569ms
I/labs.appname(10128): WaitForGcToComplete blocked Background on Alloc for 19.519ms
W/AdaptiveStreamBuffer(10128): Turning off adaptive buffer resizing to conserve memory.
D/UploadTask(10128): Waiting 0 milliseconds
W/AdaptiveStreamBuffer(10128): Turning off adaptive buffer resizing to conserve memory.
D/UploadTask(10128): Waiting 0 milliseconds
I/labs.appname(10128): Starting a blocking GC Alloc
I/labs.appname(10128): Starting a blocking GC Alloc
I/labs.appname(10128): Clamp target GC heap from 395MB to 384MB
I/labs.appname(10128): Alloc concurrent copying GC freed 12MB AllocSpace bytes, 2(40KB) LOS objects, 3% free, 371MB/384MB, paused 70us,28us total 32.573ms
I/labs.appname(10128): WaitForGcToComplete blocked Background on Alloc for 32.736ms

Flutter dependencies

Expand Flutter dependencies snippet

```yaml - audio_session 0.1.21 [flutter flutter_web_plugins rxdart meta] - audioplayers 6.1.0 [audioplayers_android audioplayers_darwin audioplayers_linux audioplayers_platform_interface audioplayers_web audioplayers_windows file flutter http meta path_provider synchronized uuid] - auto_route 9.2.0 [flutter path web collection meta] - camera 0.11.0+2 [camera_android_camerax camera_avfoundation camera_platform_interface camera_web flutter flutter_plugin_android_lifecycle] - cloud_firestore 4.11.0 [cloud_firestore_platform_interface cloud_firestore_web collection firebase_core firebase_core_platform_interface flutter meta] - cross_local_storage 3.0.0 [flutter shared_preferences] - device_info_plus 10.1.2 [device_info_plus_platform_interface ffi file flutter flutter_web_plugins meta web win32 win32_registry] - disk_space 0.2.1 [flutter] - email_validator 3.0.0 - equatable 2.0.5 [collection meta] - firebase_app_check 0.2.1 [firebase_app_check_platform_interface firebase_app_check_web firebase_core firebase_core_platform_interface flutter] - firebase_auth 4.11.1 [firebase_auth_platform_interface firebase_auth_web firebase_core firebase_core_platform_interface flutter meta] - firebase_core 2.32.0 [firebase_core_platform_interface firebase_core_web flutter meta] - firebase_crashlytics 3.4.1 [firebase_core firebase_core_platform_interface firebase_crashlytics_platform_interface flutter stack_trace] - firebase_storage 11.7.7 [firebase_core firebase_core_platform_interface firebase_storage_platform_interface firebase_storage_web flutter] - flutter 0.0.0 [characters collection material_color_utilities meta vector_math sky_engine] - flutter_bloc 8.1.6 [bloc flutter provider] - flutter_blue_plus 1.32.12 [flutter] - flutter_ffmpeg 0.4.2 [flutter] - flutter_markdown 0.7.3+2 [flutter markdown meta path] - flutter_serial_communication 0.2.7 [flutter plugin_platform_interface] - flutter_sound 9.2.13 [path_provider recase uuid provider path synchronized logger flutter flutter_sound_platform_interface flutter_sound_web] - flutter_svg 2.0.9 [flutter vector_graphics vector_graphics_codec vector_graphics_compiler] - flutter_tagging_plus 4.0.2 [flutter flutter_typeahead collection meta] - fluttertoast 8.2.8 [flutter flutter_web_plugins web] - geolocator 12.0.0 [flutter geolocator_platform_interface geolocator_android geolocator_apple geolocator_web geolocator_windows] - get_it 7.7.0 [async collection meta] - google_mlkit_face_detection 0.11.1 [flutter google_mlkit_commons] - graphql 5.2.0-beta.4 [meta path gql gql_exec gql_link gql_http_link gql_transform_link gql_error_link gql_dedupe_link hive normalize http collection web_socket_channel stream_channel rxdart uuid] - injectable 2.4.4 [get_it meta] - internet_connection_checker_plus 1.0.1 [flutter http] - light 3.0.1 [flutter] - package_info_plus 8.0.2 [ffi flutter flutter_web_plugins http meta path package_info_plus_platform_interface web win32 clock] - path 1.9.0 - path_provider 2.1.4 [flutter path_provider_android path_provider_foundation path_provider_linux path_provider_platform_interface path_provider_windows] - permission_handler 11.3.1 [flutter meta permission_handler_android permission_handler_apple permission_handler_html permission_handler_windows permission_handler_platform_interface] - polar 7.5.0 [flutter flutter_plugin_android_lifecycle permission_handler device_info_plus recase json_annotation] - provider 6.1.2 [collection flutter nested] - rxdart 0.27.7 - screen_brightness 1.0.1 [flutter screen_brightness_platform_interface screen_brightness_android screen_brightness_ios screen_brightness_macos screen_brightness_windows] - sensors 0.4.2+8 [flutter] - wakelock_plus 1.2.8 [flutter flutter_web_plugins meta wakelock_plus_platform_interface win32 dbus package_info_plus web] dev dependencies: - auto_route_generator 9.0.0 [build source_gen analyzer path build_runner code_builder dart_style xml args glob package_config auto_route] - build_runner 2.4.12 [analyzer args async build build_config build_daemon build_resolvers build_runner_core code_builder collection crypto dart_style frontend_server_client glob graphs http_multi_server io js logging meta mime package_config path pool pub_semver pubspec_parse shelf shelf_web_socket stack_trace stream_transform timing watcher web_socket_channel yaml] - flutter_lints 5.0.0 [lints] - flutter_test 0.0.0 [flutter test_api matcher path fake_async clock stack_trace vector_math leak_tracker_flutter_testing async boolean_selector characters collection leak_tracker leak_tracker_testing material_color_utilities meta source_span stream_channel string_scanner term_glyph vm_service] - injectable_generator 2.6.2 [build source_gen path glob analyzer code_builder dart_style injectable collection recase meta] ```

Additional context and comments

Any assistance in resolving this issue would be greatly appreciated, as it's critical for the functionality of my app. Thank you for your time and effort.

[SelaseKay]

Hi @Fudal , thanks for submitting this issue. We are currently looking into it.

[SelaseKay]

Hi @Fudal , Thank you for sharing the details of the issue. I tried reproducing the error using the code you provided, but I encountered an "out of memory" error during the _prepareLargeFiles() method while generating the 1GB files, rather than during the upload process itself.

Could you clarify whether the "out of memory" error you're seeing happens during file preparation or upload?

[Fudal]

@SelaseKay My production code is different than what is shown here, in my case the large files are video recordings created earlier, I do not create them as shown here in the example code. I created this sample code just for this issue. In my app, the "out of memory" error definitely occurs when uploading files to Firebase and only on versions higher than firebase_storage: 11.3.1

[SelaseKay]

Thanks for the clarification

[Fudal]

Please let me know if there's anything else I can do or add. I'm very keen to use the latest versions of the libraries, as other dependencies are also adapted to the Firebase version.

[SelaseKay]

I'm currently working on reproducing the issue on my end. I assumed you'd tested the reproducible code, which is why I raised my earlier question.