search

OneDrive / onedrive-sdk-android

OneDrive SDK for Android!
https://dev.onedrive.com
Other
148 stars 52 forks source link

OutOfMemory when uploading a huge file. #108

Closed isabsent closed 1 month ago

isabsent commented 8 years ago

I am trying to upload media file with size of 141 MB to OneDrive by means of your code snippet:

byte[] fileInMemory = IOUtils.toByteArray(context.getContextResolver().openInputStream(Uri.fromFile(fileToUpload)));
ONE_DRIVE.getOneDrive().getService()
        .getDrive()
        .getItems(((OneDriveFile) parent).getId())
        .getChildren()
        .byId(getName())
        .getContent()
        .buildRequest()
        .put(fileInMemory, callback);

(IOUtils is from org.apache.commons.io version 2.4) and get OOM error immediately:

java.lang.OutOfMemoryError: Failed to allocate a 147687689 byte allocation with 16777216 free bytes and 83MB until OOM
at org.apache.commons.io.output.ByteArrayOutputStream.toByteArray(ByteArrayOutputStream.java:322)
at org.apache.commons.io.IOUtils.toByteArray(IOUtils.java:463)

How about a promised wrapper on chunked upload? :)

FaustoMoon commented 8 years ago

Hi all, I've experienced the same issue. Is it possible to implement any workaround? In your opinion (and experience) Is it possible to implement in android an "upload session" for huge file, can anyone post an example?

Thanks in advance!

iambmelt commented 8 years ago

@isabsent @FaustoMoon Sorry to see that nobody has gotten back to you yet regarding these questions. There's some documentation regarding the limitations of this endpoint that's worth calling out here: mainly, that it supports payloads up to 4MB in size.

From graph.microsoft.io

The simple upload API allows you to provide the contents of a new file or update the contents of an existing file in a single API call. This method only supports files up to 4MB in size. To upload large files see Upload large files with an upload session.

I'm not aware of any support for large uploads yet in this SDK, perhaps the maintainer can chime in with any insight here...

/cc @daboxu

daboxu commented 8 years ago

hey guys, sorry for late reply, here is a PR for supporting large file upload: https://github.com/OneDrive/onedrive-sdk-android/pull/111

isabsent commented 8 years ago

Thanks a lot! Would you be so kind to provide code snippets for your chunked upload in synchronous and asynchronous cases?

FaustoMoon commented 8 years ago

Nice news, thanks!

daboxu commented 8 years ago

@isabsent here you go, I only applied callback pattern with synchronous case, curious how to apply async one? The operation is tightly combined with File IO and network so haven't found a good way to snitch into async way in the code. Any good ideas? The code I tested so far looks like:

            final AsyncTask<Void, Void, Void> uploadFile = new AsyncTask<Void, Void, Void>() {
                @Override
                protected Void doInBackground(final Void... params) {
                    try {
                        final ContentResolver contentResolver = getActivity().getContentResolver();
                        final ContentProviderClient contentProvider = contentResolver
                                .acquireContentProviderClient(data.getData());
                        contentProvider.release();

                        final String filename = FileContent.getValidFileName(contentResolver, data.getData());
                        final int fileSize = FileContent.getFileSize(contentProvider, data.getData());
                        final InputStream fileStream = contentResolver.openInputStream(data.getData());
                        final Option option = new QueryOption("@name.conflictBehavior", "fail");

                        oneDriveClient.getDrive()
                                .getRoot()
                                .getItemWithPath(<"Path to file">)
                                .getCreateSession(new ChunkedUploadSessionDescriptor())
                                .buildRequest()
                                .post()
                                .createUploadProvider(oneDriveClient, fileStream, fileSize, Item.class)
                                .upload(Collections.singletonList(option),
                                        new IProgressCallback<Item>() {
                                            @Override
                                            public void progress(long current, long max) {
                                                dialog.setProgress((int) current);
                                                dialog.setMax((int) max);

                                            }

                                            @Override
                                            public void success(Item item) {dialog.dismiss();
                                                Toast.makeText(getActivity(),
                                                        application
                                                                .getString(R.string.upload_complete,
                                                                        item.name),
                                                        Toast.LENGTH_LONG).show();
                                                refresh();
                                            }

                                            @Override
                                            public void failure(ClientException error) {
                                                dialog.dismiss();
                                                if (error.isError(OneDriveErrorCodes.NameAlreadyExists)) {
                                                    Toast.makeText(getActivity(),
                                                            R.string.upload_failed_name_conflict,
                                                            Toast.LENGTH_LONG).show();
                                                } else {
                                                    Toast.makeText(getActivity(),
                                                            application
                                                                    .getString(R.string.upload_failed,
                                                                            filename),
                                                            Toast.LENGTH_LONG).show();
                                                }
                                            }
                                        });
                    } catch (final Exception e) {
                        Log.e(getClass().getSimpleName(), e.getMessage());
                        Log.e(getClass().getSimpleName(), e.toString());
                    }
                    return null;
                }
            };
            uploadFile.execute();
daboxu commented 8 years ago

@isabsent @FaustoMoon it should be out with version 1.3.1, feel free to try it out and let me know if you have any questions.

isabsent commented 8 years ago

Thanks a lot! I will try right now.

isabsent commented 8 years ago

Works fine. I have used synchronous variant with Android IntentService. Does

public void progress(long current, long max)

fire after each 4 MB uploaded? If so, it seems too much because progress bar looks very intermittent . Other clouds use smaller portion.

By the way, is there an option to interrupt the upload?

daboxu commented 8 years ago

@isabsent we do provide the the customization on the chunk you want to upload: https://github.com/OneDrive/onedrive-sdk-android/blob/master/onedrivesdk/src/main/java/com/onedrive/sdk/concurrency/ChunkedUploadProvider.java#L140

so you can invoke like:

                        oneDriveClient.getDrive()
                                .getRoot()
                                .getItemWithPath(<"Path to file">)
                                .getCreateSession(new ChunkedUploadSessionDescriptor())
                                .buildRequest()
                                .post()
                                .createUploadProvider(oneDriveClient, fileStream, fileSize, Item.class)
                                .upload(Collections.singletonList(option),
                                        new IProgressCallback<Item>() {
                                            @Override
                                            public void progress(long current, long max) {
                                                dialog.setProgress((int) current);
                                                dialog.setMax((int) max);

                                            }

                                            @Override
                                            public void success(Item item) {dialog.dismiss();
                                                Toast.makeText(getActivity(),
                                                        application
                                                                .getString(R.string.upload_complete,
                                                                        item.name),
                                                        Toast.LENGTH_LONG).show();
                                                refresh();
                                            }

                                            @Override
                                            public void failure(ClientException error) {
                                                dialog.dismiss();
                                                if (error.isError(OneDriveErrorCodes.NameAlreadyExists)) {
                                                    Toast.makeText(getActivity(),
                                                            R.string.upload_failed_name_conflict,
                                                            Toast.LENGTH_LONG).show();
                                                } else {
                                                    Toast.makeText(getActivity(),
                                                            application
                                                                    .getString(R.string.upload_failed,
                                                                            filename),
                                                            Toast.LENGTH_LONG).show();
                                                }
                                            }
                                        }, <Your Size In Bytes>);

and due to service requirement, the number should be a multiple of 320KB.

isabsent commented 8 years ago

320 KB is good for smooth progress, thanks. Is there an option to cancel the upload?

daboxu commented 8 years ago

do you mean cancel during uploading? or cancel if the result returned is an exception?

isabsent commented 8 years ago

I mean cancel during uploading.

daboxu commented 8 years ago

that's a good point, I will add it to backlog, probably a Cancelalbe Interface is needed.

FaustoMoon commented 7 years ago

hey guys, I've just tested the chunk upload and I find out an unexpected behaviour, after one hour of uploading I receive this message: Response code 401, Unauthorized

hereafter the logcat in starting phase:

`12-13 07:58:21.504 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 71: Intercepting request, https://api.onedrive.com/up/....

12-13 07:58:21.505 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 82: Found account information

12-13 07:58:21.505 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 219: Starting to send request, URL https://api.onedrive.com/up/....

12-13 07:58:21.505 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 223: Request Method PUT

12-13 07:58:21.505 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 229: Sending byte[] as request body

12-13 07:58:40.481 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 265: Response code 202, Accepted

12-13 07:58:40.481 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 270: StatefulResponse is handling the HTTP response.

12-13 07:58:40.486 2652-8018/com.onedfau.apiexplorer D/ChunkedUploadResponseHandler[generateResult] - 90: Chunk bytes has been accepted by the server.

12-13 07:58:40.489 2652-8018/com.onedfau.apiexplorer D/DefaultSerializer[deserializeObject] - 67: Deserializing type UploadSession

12-13 07:58:40.489 2652-8018/com.onedfau.apiexplorer D/PingMe: BigFile progress: 327680`

then the logcat after one hour:

`12-13 08:56:29.876 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 71: Intercepting request, https://api.onedrive.com/up/...

12-13 08:56:29.876 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 76: Found an existing authorization header!

12-13 08:56:29.877 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 219: Starting to send request, URL https://api.onedrive.com/up/...

12-13 08:56:29.877 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 223: Request Method PUT

12-13 08:56:29.878 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 229: Sending byte[] as request body

12-13 08:58:13.416 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 265: Response code 401, Unauthorized

12-13 08:58:13.417 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 270: StatefulResponse is handling the HTTP response.

12-13 08:58:13.417 2652-8018/com.onedfau.apiexplorer D/ChunkedUploadResponseHandler[generateResult] - 107: Receiving error during upload, see detail on result error

12-13 08:58:21.425 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 71: Intercepting request, https://api.onedrive.com/up/....

12-13 08:58:21.425 2652-8018/com.onedfau.apiexplorer D/AuthorizationInterceptor[intercept] - 76: Found an existing authorization header!

12-13 08:58:21.426 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 219: Starting to send request, URL https://api.onedrive.com/up/...

12-13 08:58:21.427 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 223: Request Method PUT

12-13 08:58:21.427 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 229: Sending byte[] as request body

12-13 08:58:59.558 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 265: Response code 401, Unauthorized

12-13 08:58:59.558 2652-8018/com.onedfau.apiexplorer D/DefaultHttpProvider[sendRequestInternal] - 270: StatefulResponse is handling the HTTP response.

12-13 08:58:59.559 2652-8018/com.onedfau.apiexplorer D/ChunkedUploadResponseHandler[generateResult] - 107: Receiving error during upload, see detail on result error

12-13 08:58:59.560 2652-8018/com.onedfau.apiexplorer D/PingMe: BigFile: upload error`

Do you think that I've made a mistake in my implementation? thanks in advance for any suggestions.

daboxu commented 7 years ago

hi @FaustoMoon , did your uploading job exists for more than one hour? The oauth token is valid for 1 hour only so that's why you got unauthorized. Try with larger chunk size and you can see my example code above.

KevinCWKevin commented 4 years ago

Hi @daboxu, I follow your sample code, used "createUploadProvider" to upload large files. But the fileSize is limited to 2GB because the type of variable is Integer. Could you please provide a way to upload file with over 2GB?

baywet commented 1 month ago

Thank you for reaching out and for your patience. This SDK is being officially deprecated. See #172 for more information