PnP/Samples/Core.RestFileUpload/ - Incorrect and/or misleading information in ReadMe

adambu commented 7 years ago

Thank you for reporting an issue or suggesting an enhancement. We appreciate your feedback - to help the team to understand your needs, please complete the below template to ensure we have the necessary details to assist you. If you have a actual question, we would ask you to use PnP Yammer group at http://aka.ms/OfficeDevPnPYammer. Thanks!

Which PnP repository to report the issue?

PnP Samples - https://github.com/OfficeDev/PnP

Environment

[X] Office 365 / SharePoint Online [ ] SharePoint 2016 [ ] SharePoint 2013

If SharePoint on-premises, what's exact CU version:

Expected or Desired Behavior

DOC BUG: In the Readme these statements are false or misleading:

Using the REST approach, you do not need to slice your file into pieces and can send a file up to 2 GB, but the same security time-out restrictions mentioned in the large upload sample apply here. If you really want to upload large files and you're on SharePoint Online the sliced upload is the advised approach. For on-premises or for files that can be uploaded within the security timeout window the REST approach is good one.

I have confirmed there is a hard coded size limited of 250 MB in SPO. So for file larger than that you MUST use the chunked file approach. Given that, there is a rather glaring absence of a "pure" rest example. I came up with one using WebClient, but it was pretty hard, and involved some experimentation. This example definitely left my customer with the impression that they could avoid the chunking if they used REST. I have seen that repeated in numerous blogs on the internet.

I'm happy to provide my example, but not going to put it here because the bug report is about the ReadMe file. If you want my complete example, let me know.

Observed Behavior

Any file larger than 250 megabytes uploaded with this example will result in: Microsoft.SharePoint.Client.InvalidClientQueryException: The request message is too big. The server does not allow messages larger than 262144000 bytes. at Microsoft.SharePoint.Client.Rest.RestService.ProcessQuery(ClientServiceHost serviceHost, Stream inputStream, IList`1 pendingDisposableContainer)

adambu commented 7 years ago

Just realized that I didn't make it clear that I'm talking about SPO when I say there's a hard-coded limit on the uploaded file size using this all-at-once approach. In general we need to be more clear about the differences in SPO versus SharePoint on-prem, but this particular issue has led to some real confusion. I think it would be good if the PnP samples led by example in clarifying the differences in the APIs (if there is such a difference).

adambu commented 7 years ago

Sorry, didn't mean to close the issue.

VesaJuvonen commented 7 years ago

thx Adam, I've just updated the readme based on your input. Your sample on doing large file uploads using REST would be absolutely also valuable for the community, if you are willing to share this. Readme has though now updated, so closing this one. Thx for the input.

adambu commented 7 years ago

Thanks for the note Vesa. I think I did add the code on our SPDev blog, but it was so long ago I’ll have to check. We haven’t had a similar case since then.

I see the change in the Introduction, but IMHO, it’s still not clear enough:

If you really want to upload large files and you're on SharePoint Online the sliced upload is the advised approach. For on-premises or for files that can be uploaded within the security timeout window the REST approach is good one.

I think it should say:

If you want to upload file over 250 MB and you’re on SharePoint Online, you MUST use the sliced upload approach. The issue here is not the security timeout, but rather a hard file size upload limit set in the Web Application properties to which the tenant has no access. This setting is immutable, so you will not be able to upload file larger than 250 MB unless you use the chunked file approach.

Because that is the actual fact.

I’ll just include my example here and you can do with it what you will.

Best regards,

Adam Burns [Description: MSFT_logo_Gray DE sized SIG1.png] Sr. Support Escalation Engineer - SharePoint development Email: adambu@microsoft.commailto:adambu@microsoft.com Phone: 425-707-2428 Hours: 10:00am-7:00pm Pacific M - F

From: Vesa Juvonen [mailto:notifications@github.com] Sent: Monday, January 30, 2017 11:13 AM To: SharePoint/PnP PnP@noreply.github.com Cc: Adam Burns adambu@microsoft.com; State change state_change@noreply.github.com Subject: Re: [SharePoint/PnP] PnP/Samples/Core.RestFileUpload/ - Incorrect and/or misleading information in ReadMe (#1558)

thx Adam, I've just updated the readme based on your input. Your sample on doing large file uploads using REST would be absolutely also valuable for the community, if you are willing to share this. Readme has though now updated, so closing this one. Thx for the input.

— You are receiving this because you modified the open/close state. Reply to this email directly, view it on GitHubhttps://github.com/SharePoint/PnP/issues/1558#issuecomment-276159632, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AMKa6b7w9r_-9zA-bFIYDwuxIszEhsi-ks5rXjYogaJpZM4Km3Ob.

using System; using System.IO; using System.Net; using System.Text; using System.Threading; using System.Web.Script.Serialization;

namespace Core.RestFileUpload { / Sample Code is provided for the purpose of illustration only and is not intended to be used in a production environment. THIS SAMPLE CODE AND ANY RELATED INFORMATION ARE PROVIDED "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE IMPLIED WARRANTIES OF MERCHANTABILITY AND/OR FITNESS FOR A PARTICULAR PURPOSE. We grant You a nonexclusive, royalty-free right to use and modify the Sample Code and to reproduce and distribute the object code form of the Sample Code, provided that. You agree: (i) to not use Our name, logo, or trademarks to market Your software product in which the Sample Code is embedded; (ii) to include a valid copyright notice on Your software product in which the Sample Code is embedded; and (iii) to indemnify, hold harmless, and defend Us and Our suppliers from and against any claims or lawsuits, including attorneys’ fees, that arise or result from the use or distribution of the Sample Code. -This is intended as a sample of how code might be written for a similar purpose and you will need to make changes to fit to your requirements. -This code has not been tested. This code is also not to be considered best practices or prescriptive guidance. -No debugging or error handling has been implemented. -It is highly recommended that you FULLY understand what this code is doing and use this code at your own risk. / class Program { ///

/// Change this contant if you are having trouble and /// want to experiment with different chunk sizes. /// Currently set to the recommended chunk size. ///

const int chunkSize = 10 1024 1024; static string formDigest; static string token;

    static void Main(string[] args)
    {
        string url = "http://adambu-rrvmas16";

        /// SharePoint site Relative Url
        string folderUrl = "/Shared%20Documents";

        ChunkFile(url, folderUrl, @"F:\temp\BigBig\test500.txt");
        //UploadSaveBinaryStream(url, folderUrl, @"F:\temp\BigBig\test500_2.txt");
        //UploadDocumentContent(url, "Documents", @"F:\temp\BigBig\test500_2.txt");

    }

    /// <summary>
    /// Worker menthod.  Does all the chunking and uploading.
    /// </summary>
    /// <param name="baseUrl">Site collection url</param>
    /// <param name="folderUrl">site relative url of the folder where
    /// the file is to be uploaded</param>
    /// <param name="filePath">Path to the file on the local file system.</param>
    static void ChunkFile(string baseUrl, string folderUrl, string filePath)
    {
        using (var client = new WebClient())
        {
            try
            {
                //token = GetAccessToken(baseUrl);
                formDigest = GetFormDigest(baseUrl);

                client.BaseAddress = baseUrl;
                client.Headers.Add("X-RequestDigest", formDigest);
                //client.Headers.Add("Authorization", "Bearer " + token);
                client.Credentials = CredentialCache.DefaultCredentials;
                client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36");

                var fileUrl = "/Shared%20Documents/test500.txt";  //File's url after initial upload. 
                var fileName = System.IO.Path.GetFileName(filePath);

                //Special handling for the start of file upload
                var firstChunk = true;
                //Special id for the timeout lifetime
                var uploadId = Guid.NewGuid();
                var offset = 0L;

                using (var inputStream = System.IO.File.OpenRead(filePath))
                {
                    var buffer = new byte[chunkSize];
                    int bytesRead;
                    while ((bytesRead = inputStream.Read(buffer, 0, buffer.Length)) > 0)
                    {
                        if (firstChunk)
                        {
                            //Upload the initial empty file
                            var endpointUrl = string.Format(
                                "{0}/_api/web/GetFolderByServerRelativeUrl('{1}')/Files/Add(url='{2}', overwrite=true)",
                                baseUrl, folderUrl, fileName);
                            try
                            {
                                client.UploadData(endpointUrl, new byte[0]);
                            }
                            catch (WebException ex)
                            {
                                HandleWebException(ex);
                            }

                            // chunking to start
                            endpointUrl = string.Format(
                                "{0}/_api/web/getfilebyserverrelativeurl('{1}')/startupload(uploadId=guid'{2}')",
                                baseUrl, fileUrl, uploadId);
                            try
                            {
                                client.UploadData(endpointUrl, buffer);
                            }
                            catch (WebException ex)
                            {
                                HandleWebException(ex);
                            }

                            firstChunk = false;
                        }
                        //This is for the final chunk.
                        else if (inputStream.Position == inputStream.Length)
                        {
                            var endpointUrl = string.Format("{0}/_api/web/getfilebyserverrelativeurl('{1}')/finishupload(uploadId=guid'{2}',fileOffset={3})", baseUrl, fileUrl, uploadId, offset);
                            var finalBuffer = new byte[bytesRead];
                            Array.Copy(buffer, finalBuffer, finalBuffer.Length);
                            try
                            {
                                client.UploadData(endpointUrl, finalBuffer);
                            }
                            catch (WebException ex)
                            {
                                HandleWebException(ex);
                                return;
                            }
                        }
                        else  //As we're continuing on each chunk...
                        {
                            var endpointUrl = string.Format("{0}/_api/web/getfilebyserverrelativeurl('{1}')/continueupload(uploadId=guid'{2}',fileOffset={3})", baseUrl, fileUrl, uploadId, offset);
                            try
                            {
                                client.UploadData(endpointUrl, buffer);
                            }
                            catch (WebException ex)
                            {
                                HandleWebException(ex);
                                return;
                            }
                        }
                        offset += bytesRead;
                        Console.Clear();
                        Console.WriteLine("%{0:P} completed", (((float)offset / (float)inputStream.Length)));
                    }
                }
            }
            catch (Exception ex)
            { }
            Console.ReadKey();
        }
    }

    /// <summary>
    /// This approach uses the SaveBinaryStream method.  It will NOT work in SPO and in on-prem you must up
    /// the Maximum File Upload Size under Web Application Settingings (General Settings).
    /// </summary>
    /// <param name="baseUrl"></param>
    /// <param name="folderUrl"></param>
    /// <param name="filePath"></param>
    static void UploadSaveBinaryStream(string baseUrl, string folderUrl, string filePath)
    {
        using (var client = new WebClient())
        {
            try
            {
                //token = GetAccessToken(baseUrl);
                formDigest = GetFormDigest(baseUrl);

                client.BaseAddress = baseUrl;
                client.Headers.Add("X-RequestDigest", formDigest);
                //client.Headers.Add("Authorization", "Bearer " + token);
                client.Headers.Add("User-Agent", "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36");

                client.UseDefaultCredentials = true;

                var fileUrl = "/Shared%20Documents/BigNew.txt";  //File's url after initial upload. 
                var fileName = System.IO.Path.GetFileName(filePath);

                //Special id for the timeout lifetime
                var uploadId = Guid.NewGuid();
                //var offset = 0L;

                var buffer = new byte[chunkSize];

                //Upload the initial empty file
                var endpointUrl = string.Format(
                    "{0}/_api/web/GetFolderByServerRelativeUrl('{1}')/Files/Add(url='{2}', overwrite=true)",
                    baseUrl, folderUrl, fileUrl);
                try
                {
                    client.UploadData(endpointUrl, new byte[0]);
                }
                catch (WebException ex)
                {
                    HandleWebException(ex);
                }
                using (var docStream = System.IO.File.Open(filePath, FileMode.Open))
                {
                    docStream.Seek(0, SeekOrigin.Begin);

                    byte[] fileBytes;
                    using (var memoryStream = new MemoryStream())
                    {
                        docStream.CopyTo(memoryStream);
                        fileBytes = memoryStream.ToArray();
                    }

                    endpointUrl = string.Format("{0}/_api/web/getfilebyserverrelativeurl('{1}')/savebinarystream?@target='{2}'", baseUrl, fileUrl, baseUrl);
                    try
                    {
                        client.UploadData(endpointUrl, fileBytes);
                    }
                    catch (WebException ex)
                    {
                        HandleWebException(ex);
                        return;
                    }
                }
            }
            catch (Exception ex)
            { }
            Console.ReadKey();
        }
    }

    static void UploadDocumentContent(string baseUrl, string libraryName, string filePath)
    {

        try
        {
            ClientContext ctx = new ClientContext(baseUrl);
            ctx.Credentials = CredentialCache.DefaultCredentials;
            Web web = ctx.Web;
            List docs = web.Lists.GetByTitle(libraryName);

            byte[] fileData = System.IO.File.ReadAllBytes(filePath);
            using (System.IO.Stream stream = new System.IO.MemoryStream(fileData))
            {
                var fci = new FileCreationInformation
                {
                    Url = System.IO.Path.GetFileName(filePath),
                    ContentStream = stream,
                    Overwrite = true
                };

                Folder folder = docs.RootFolder;
                FileCollection files = folder.Files;
                Microsoft.SharePoint.Client.File file = files.Add(fci);

                ctx.Load(files);
                ctx.Load(file);
                ctx.ExecuteQuery();

            }
        }
        catch (WebException wex)
        {
            Console.Write(wex.ToString());
        }
        catch (Exception ex)
        {
            Console.Write(ex.ToString());
        }

    }

    #region Utililty Methods

    /// <summary>
    /// This is I can call the same code in every catch block.
    /// </summary>
    /// <param name="ex"></param>
    private static void HandleWebException(WebException ex)
    {
        var error = ex.Response.GetResponseStream();
        StreamReader sr = new StreamReader(error);

        Console.WriteLine("\nResponse received was {0}", sr.ReadToEnd());
        throw ex;
    }

    /// <summary>
    /// Demonstrates how to get the FormDigest.  It may not always be necessary.
    /// </summary>
    /// <param name="baseUrl">The site collection url.</param>
    /// <returns>The form digest string.  Just attach to the Headers.</returns>
    static string GetFormDigest(string baseUrl)
    {
        HttpWebRequest restRqst = (HttpWebRequest)HttpWebRequest.Create(baseUrl + "/_api/ContextInfo");
        //restRqst.Headers.Add("Authorization", "Bearer " + token);
        restRqst.UserAgent = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.71 Safari/537.36";
        restRqst.Method = "POST";
        restRqst.Accept = "application/json;odata=verbose";
        //restRqst.Credentials = spoCreds;
        restRqst.UseDefaultCredentials = true;

        restRqst.ContentLength = 0;

        string xHeader = string.Empty;

        using (HttpWebResponse restResponse = (HttpWebResponse)restRqst.GetResponse())
        {
            Stream postStream = restResponse.GetResponseStream();
            StreamReader postReader = new StreamReader(postStream);
            string results = postReader.ReadToEnd();

            JavaScriptSerializer jss = new JavaScriptSerializer();
            var d = jss.Deserialize<dynamic>(results);
            xHeader = d["d"]["GetContextWebInformation"]["FormDigestValue"];
        }
        return xHeader;
    }

    /// <summary>
    /// Gets the access token using Token Helper.
    /// </summary>
    /// <param name="url">Site Collection url.</param>
    /// <returns>The access token as a string.</returns>
    internal static string GetAccessToken(string url)
    {
        Uri uri = new Uri(url);

        string realm = TokenHelper.GetRealmFromTargetUrl(uri);

        //Get the access token for the URL.  
        //   Requires this app to be registered with the tenant
        string accessToken = TokenHelper.GetAppOnlyAccessToken(
            TokenHelper.SharePointPrincipal,
            uri.Authority, realm).AccessToken;

        return accessToken;
    }

    #endregion Utility Methods
}

}

VesaJuvonen commented 7 years ago

Hi Adam, thx for the editorial suggestion, have updated that now in. We absolutely due appreciate the shared code as well, but we can't really take code suggestions using email / issue list submissions. We would appreciate these kind of things to be shared Pull Requests, so that the contributors would get officially listed in the repository and would get also called out in our monthly contributions. Secondary reason is also the fact that since PnP is community open source initiative, we really don't have time to take care of the edits manually either. Anyway - if nothing else, it's now stored in this issue list, unless someone submits a Pull Request with the needed changes.

pnp / PnP