Downloading from iCloud, onProgress after a few of iterations is not called

[psquizzle]

I tried calling multiple downloads asynchronously it however freezes the whole app. So instead my solution is to call each download recursively. It only manages to achieve 5-10 iterations before stopping. On the successful iteration, the onProgress callback is called twice once with 0 and once with 100. On the iteration that it gets stuck on neither 0 nor 100 are called and no errors are thrown. I have tried adding a timeout to call the function again and then I get the same problem as when calling a batch of downloads that it freezes up.

import * as CloudStore from "react-native-cloud-store";

let cancelDownload = false;
let actualIndex = 0;
let itemsCalled = [];
function sleep(ms) {
  return new Promise((resolve) => setTimeout(resolve, ms));
}
async function commenceDownloadChain({
  arrayOfFiles,
  settingsContext,
  cb,
  index,
}) {
    if(settingsContext.numberOfFiles !==null){
       return false
    }

  let arrayOfUris = arrayOfFiles;
  if (index === 0) {
    settingsContext.setNumberOfFiles(arrayOfFiles.length);
    settingsContext.numberOfFilesFinished.value = 0;
    cancelDownload = false;
    actualIndex = 0;
    itemsCalled = [];
    arrayOfUris = arrayOfFiles.map((file) => file.uri);
  }

  if (arrayOfUris[actualIndex] && !cancelDownload) {
    if (!itemsCalled.includes(actualIndex)) {
      itemsCalled.push(actualIndex);

      const item = arrayOfUris[actualIndex];
      arrayOfUris[actualIndex] = "";

      actualIndex = actualIndex + 1;
      settingsContext.numberOfFilesFinished.value = actualIndex;

      try {
        console.log("called awaiting onProgress");
        console.log(item);

        await CloudStore.download(item, {
          onProgress: async (data) => {
            if (data.progress === 100) {

              if (!cancelDownload) {
                commenceDownloadChain({
                  arrayOfFiles: arrayOfUris,
                  settingsContext,
                  cb,
                  index: index + 1,
                });
              } else {
                cb();
              }
            }
          },
        });
      } catch (e) {
        //  clearTimeout(timer)
        console.error(e);
        if (!cancelDownload) {
          commenceDownloadChain({
            arrayOfFiles: arrayOfUris,
            settingsContext,
            cb,
            index: index + 1,
          });
        } else {
          cb();
        }
      }
    }
  } else {
    await sleep(100);
    cb();
  }
  return true
}

module.exports = {
  cancelDownload: () => (cancelDownload = true),
  commenceDownloadChain,
};

[XHMM]

I tried calling multiple downloads asynchronously and everything works well, can you provide a sample repo?

[psquizzle]

Okay it seems to work well with not many files but as soon as you get above 1000 files nothing downloads.

Here is a repository, reproducible example using expo. https://github.com/psquizzle/icloud-playground-expo.git

Copy "COPY_TO_ICLOUD" to the iCloud container and in the app press Read directory and then once complete Download.

[YaoHuiJi]

in my case, onProgress for download seems never be called, but onProgress for upload works well.（iOS 15&16）

[psquizzle]

I think it is somehow an issue relating to the placeholder files ".fileName.ext.icloud", not matching the query that is generated. In iOS 17 the iCloud container has been revamp and now uses a dataless file structure so no placeholder file no ".iCloud" extensions. In iOS17 the onProgress callback seems to be called consistently.

[YaoHuiJi]

I think it is somehow an issue relating to the placeholder files ".fileName.ext.icloud", not matching the query that is generated. In iOS 17 the iCloud container has been revamp and now uses a dataless file structure so no placeholder file no ".iCloud" extensions. In iOS17 the onProgress callback seems to be called consistently.

hi, @psquizzle, I am not familiar with native development, can you show me the related url or tell me the keywords that I can google, I tried but can not find related posts. Some data sync logic in my app depends on on ".icloud" file, so if it is true, I think I must rewrite it.

[psquizzle]

Don't use the ".iCloud" files for logic within the app as they are being deprecated.

TN3150: Getting ready for dataless files

To determine if a file is dataless, you can use the NSURLUbiquitousItemIsDatalessKey attribute provided by NSFileManager's attributesOfItemAtPath method or URLResourceKey.ubiquitousItemIsDatalessKey with the resourceValues(forKeys:fromBookmarkData:) method. If the value of this attribute is true, then the file is dataless. This attribute indicates that the file exists in iCloud Drive but does not have its data downloaded to the local device.

[YaoHuiJi]

Don't use the ".iCloud" files for logic within the app as they are being deprecated.

TN3150: Getting ready for dataless files

To determine if a file is dataless, you can use the NSURLUbiquitousItemIsDatalessKey attribute provided by NSFileManager's attributesOfItemAtPath method or URLResourceKey.ubiquitousItemIsDatalessKey with the resourceValues(forKeys:fromBookmarkData:) method. If the value of this attribute is true, then the file is dataless. This attribute indicates that the file exists in iCloud Drive but does not have its data downloaded to the local device.

Thank you, @psquizzle. So, in the future(iOS 17), a file in icloud container, let's say, "xxx.backup", we can't tell if it is a dataless file or a 'normal' file with real data via its file name, we must check file's stat(st_flags) to find out the answer, did I get it right?

and if it's right, does it mean that this library(v0.10.3) can no longer be used to correctly determine the file that needs to be downloaded, Or on the contrary, the library will not only still work, but it will become easier to use, because we can just ignore '.cloud' files and just check file's downloadStatus to check if it's NSURLUbiquitousItemDownloadingStatusCurrent

[XHMM]

Callback not triggered may be caused by this predicate, I think I should expose this as a config to RN-side so users can customize how to match the file path. I will do it when I'm free

[psquizzle]

Don't use the ".iCloud" files for logic within the app as they are being deprecated. TN3150: Getting ready for dataless files To determine if a file is dataless, you can use the NSURLUbiquitousItemIsDatalessKey attribute provided by NSFileManager's attributesOfItemAtPath method or URLResourceKey.ubiquitousItemIsDatalessKey with the resourceValues(forKeys:fromBookmarkData:) method. If the value of this attribute is true, then the file is dataless. This attribute indicates that the file exists in iCloud Drive but does not have its data downloaded to the local device.

Thank you, @psquizzle. So, in the future(iOS 17), a file in icloud container, let's say, "xxx.backup", we can't tell if it is a dataless file or a 'normal' file with real data via its file name, we must check file's stat(st_flags) to find out the answer, did I get it right?

and if it's right, does it mean that this library(v0.10.3) can no longer be used to correctly determine the file that needs to be downloaded, Or on the contrary, the library will not only still work, but it will become easier to use, because we can just ignore '.cloud' files and just check file's downloadStatus to check if it's NSURLUbiquitousItemDownloadingStatusCurrent

1. Yes from iOS17 onward you must use .stat(uri) function. and logic based on downloadStatus. If you try to just use the file it does download it, but in my use case leads to a bad user experience with indeterminate delays.
2. This library works and becomes easier to use because you no longer deal with transforming filenames. Between .icloud and normalised forms.

[YaoHuiJi]

Don't use the ".iCloud" files for logic within the app as they are being deprecated. TN3150: Getting ready for dataless files To determine if a file is dataless, you can use the NSURLUbiquitousItemIsDatalessKey attribute provided by NSFileManager's attributesOfItemAtPath method or URLResourceKey.ubiquitousItemIsDatalessKey with the resourceValues(forKeys:fromBookmarkData:) method. If the value of this attribute is true, then the file is dataless. This attribute indicates that the file exists in iCloud Drive but does not have its data downloaded to the local device.

Thank you, @psquizzle. So, in the future(iOS 17), a file in icloud container, let's say, "xxx.backup", we can't tell if it is a dataless file or a 'normal' file with real data via its file name, we must check file's stat(st_flags) to find out the answer, did I get it right? and if it's right, does it mean that this library(v0.10.3) can no longer be used to correctly determine the file that needs to be downloaded, Or on the contrary, the library will not only still work, but it will become easier to use, because we can just ignore '.cloud' files and just check file's downloadStatus to check if it's NSURLUbiquitousItemDownloadingStatusCurrent

1. Yes from iOS17 onward you must use .stat(uri) function. and logic based on downloadStatus. If you try to just use the file it does download it, but in my use case leads to a bad user experience with indeterminate delays.
2. This library works and becomes easier to use because you no longer deal with transforming filenames. Between .icloud and normalised forms.

Thank you @psquizzle for the explanation, and thank you @XHMM for your great library, love you guys❤️

[XHMM]

Okay it seems to work well with not many files but as soon as you get above 1000 files nothing downloads.

Here is a repository, reproducible example using expo. https://github.com/psquizzle/icloud-playground-expo.git

Copy "COPY_TO_ICLOUD" to the iCloud container and in the app press Read directory and then once complete Download.

Hi, @psquizzle , I just tried your demo repository and released 0.10.4, can you give it a try to see if the problems solved?

[psquizzle]

Nice work! Tested on iOS 17 and iOS 16 seems to be working on both. I would say you should hit "npm publish". :)