Store the loaded model in cache and reuse again

[Damu1234]

Hi,Thanks for sharing your code , its really useful for me , But i am facing issue while trying to load the model which is already downloaded ,Every time i assign the url of model placed in server its loading model from net , its taking almost 20 secs to download the model, Can we store the loaded model in cache and reuse again and make the model to load in less time ?

[Damu1234]

Hi ,

Any suggestion for how storing the model in cache or external drive and reusing it when ever needed .

Regards, S.DamodharaReddy

[gpvigano]

For now there is only a reuseLoaded flag in ImportOptions that allow to reuse models already in memory (see Loader.cs, line 160). I think this could be the entry point for implementing a cache, adding a cacheModels flag to ImportOptions. Do you already have an idea about a possible implementation?

[Damu1234]

I dont know how exactly loading model from cache will work , but there is an option in unity LoadFromCacheOrDownload , i used this one for asset bundle , if asset bundle if downloaded once it will be placed in cache , so from next time the bundle wont be loaded from server , it will load from cache only. I need similar type with .obj file ,But Loadfromcacheordownload is not supourting it.
Currently i am doing r & d on it .I think there should be a option of loading model to external drive first and we need to give acess to unity to load from external drive instead of server

[Damu1234]

By the way can u please explain how reuseloaded flag will be used .

[gpvigano]

I prepared another example, just for testing the importer with URLs, you can find it in the latest version, example 003_Import_URLTest. I just used twice the test model stored here, just to test reuseLoaded flag.

[guizmo01]

Why can't you download the file with UnityWebrequest DownloadhandlerFile and store it in persistent path then load it the simple way?

[gpvigano]

Thank you very much for your suggestion, this seems to be a working solution. I will have a look to it (as soon as I have enough time...).

[guizmo01]

Hi, I would recoment to user the downloadHandlerScript for such files:

` class FileDownloadHandler : DownloadHandlerScript { private int expected = -1; private int received = 0; private string filepath; private FileStream fileStream; private bool canceled = false;

    public FileDownloadHandler(byte[] buffer, string filepath) : base(buffer)
    {
        this.filepath = filepath;
        fileStream = new FileStream(filepath, FileMode.Create, FileAccess.Write);
    }

    protected override byte[] GetData()
    {
        return null;
    }

    protected override bool ReceiveData(byte[] data, int dataLength)
    {
        if (data == null || data.Length < 1)
        {
            return false;
        }

        received += dataLength;

        if (!canceled)
        {
            fileStream.Write(data, 0, dataLength);
        }

        return true;
    }

    protected override float GetProgress()
    {
        if (expected < 0)
        {
            return 0;
        }

        return (float)received / expected;
    }

    protected override void CompleteContent()
    {
        fileStream.Close();
    }

    protected override void ReceiveContentLength(int contentLength)
    {
        expected = contentLength;
    }

    public void Cancel()
    {
        canceled = true;
        fileStream.Close();
        File.Delete(filepath);
    }
}

// Method that downloads a file public IEnumerator DownloadFiles(string url, string file, string folderStructure = "") { using (UnityWebRequest www = new UnityWebRequest(url)) { byte[] buffer = new byte[256 * 1024];

            www.SetRequestHeader("Cache-Control", "max-age=0, no-cache, no-store");
            www.SetRequestHeader("Pragma", "no-cache");

            //www.chunkedTransfer = false;

            string path;

            if (folderStructure == "")
            {
                path = Path.Combine(DeviceDatabasePath, file);
            }
            else
            {
                path = Path.Combine(DeviceDatabasePath + folderStructure, file);
            }

            if (!Directory.Exists(Path.GetDirectoryName(path)))
            {
                Directory.CreateDirectory(Path.GetDirectoryName(path));
            }

            // Download file if it does not exist
            if (!File.Exists(path))
            {
                yield return FireWebRequest(buffer, path, url, www);
            }
            else
            {
                yield return CheckFileSize(url, path);

                if (needDeleting == true)
                {
                    yield return FireWebRequest(buffer, path, url, www);
                    needDeleting = false;
                }
            }
        }
    }`

`private IEnumerator FireWebRequest(byte[] buffer, string path, string url, UnityWebRequest www) { www.downloadHandler = new FileDownloadHandler(buffer, path);

        www.timeout = 1000;

        www.SendWebRequest();

        yield return DownloadProgress(www);

        // Check for errors
        if (www.isNetworkError || www.isHttpError)
        {
            long responseCode = www.responseCode;

            Debug.Log(string.Format("request {0} error {1}", url, www.error + " " + responseCode.ToString()));

            if (www.timeout >= 1000)
            {
                response.text = www.error + " Error Code: " + "504";
            }
            else
            {
                response.text = www.error + " Error Code: " + responseCode.ToString();
            }

            www.Abort();

            if (OnError != null)
            {
                OnError(responseCode);
            }
        }
        else
        {
            // File have been downloaded
            Debug.Log("File successfully downloaded and saved to " + path);

            if (OnComplete != null)
            {
                OnComplete();
            }

            www.downloadHandler.Dispose();

            buffer = null;
        }
    }`

`// Download progress bar and text public IEnumerator DownloadProgress(UnityWebRequest uwr) { while (!uwr.isDone) { progress = uwr.downloadProgress;

            if (Application.internetReachability == NetworkReachability.NotReachable)
            {
                Debug.Log("Internet connection lost " + uwr.responseCode);

                response.text = "Internet connection has been lost" + " Error Code: " + "503";
            }
            else
            {
                response.text = string.Empty;
                bytesText.text = string.Format("{0} MB", (uwr.downloadedBytes / 1000000)).ToString();
            }

            if (OnProgress != null)
            {
                 Debug.Log("OnProgess event not null");
                OnProgress(progress);
            }

            yield return null;
        }
    }`

` public IEnumerator CheckFileSize(string url, string filePath) { using (UnityWebRequest webRequest = UnityWebRequest.Head(url)) { yield return webRequest.SendWebRequest();

            if (webRequest.GetResponseHeaders() == null)
            {
                Debug.LogError("response header is null");
            }
            else
            {
                //webRequest.GetResponseHeader("Content-Length");
                Debug.Log(webRequest.GetResponseHeader("Content-Length"));

                var fileInfo = new FileInfo(filePath);

                Debug.Log("The size of the file is = " + fileInfo.Length);

                yield return new WaitForEndOfFrame();

                if (fileInfo.Length.ToString() != webRequest.GetResponseHeader("Content-Length"))
                {
                    Debug.Log("File isn't the same size so delete it and start download again");
                    File.Delete(filePath);
                    yield return new WaitForEndOfFrame();

                    needDeleting = true;
                }
            }
        }
    }`

[guizmo01]

If you using a cached value system for checking for updated file from your server you can use thise method

`// For Checking against cached values // a = file already in persistant path // b = files in json (DownloadedData) public void findMissing(string[] a, string[] b, int n, int m) { for (int i = 0; i < n; i++) { int j;

            for (j = 0; j < m; j++)
                if (a[i] == b[j])
                    break;

            if (j == m)
            {
                File.Delete(a[i]);
                Debug.Log(a[i] + " is the file to delete ");
            }
        }
    }

`

usage:

`string[] a = { "cache12343554", "cache44364545" }; string[] b = { "cache129834", "cache44364545" }; int n = a.Length; int m = b.Length;

        findMissing(a, b, n, m);

`

[gpvigano]

I see you have a better knowledge on this topic than me, I'm sorry, but I'm not yet able to integrate the code you wrote in AsImpL, I'd like to see a simple working example with this implementation. What about creating a new branch (feat/cache) with an example scene where you can implement and test a basic cache, even if not integrated in AsImpL? If you prefer you can create your own repository (please choose an open source license, e.g. MIT). Finally I can try to adopt your implementation within AsImpL merging or importing your work. What do you think about it? Anyway thank you very much for this contribution.

[guizmo01]

Hi,

Yes this implementation is very generic and wouldn’t be hard to implement in any project however there a few elements in my code that does not need to be included. But yes when I get the time maybe over the weekend I will set up sample project and put it on GitHub. I use sourcetree for source control which seams to always have been working fine for me.

Kind regards.

JC Mazza

Sent from Mailhttps://go.microsoft.com/fwlink/?LinkId=550986 for Windows 10

---

From: Giovanni Paolo Viganò notifications@github.com Sent: Tuesday, October 9, 2018 1:18:02 PM To: gpvigano/AsImpL Cc: guizmo01; Comment Subject: Re: [gpvigano/AsImpL] Store the loaded model in cache and reuse again (#6)

I see you have a better knowledge on this topic than me, I'm sorry, but I'm not yet able to integrate the code you wrote in AsImpL, I'd like to see a simple working example with this implementation. What about creating a new branch (feat/cache) with an example scene where you can implement and test a basic cache, even if not integrated in AsImpL? If you prefer you can create your own repository (please choose an open source license, e.g. MIT). Finally I can try to adopt your implementation within AsImpL merging or importing your work. What do you think about it? Anyway thank you very much for this contribution.

— You are receiving this because you commented. Reply to this email directly, view it on GitHubhttps://github.com/gpvigano/AsImpL/issues/6#issuecomment-428169827, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AEJVkyIJE53FAz7y2mGTAQlJlGNEM4Euks5ujJP6gaJpZM4Qm6QJ.