Downloading the file in background

sojharo commented 7 years ago

part of https://github.com/Cloudkibo/Android/issues/162

We should write our logic in such a way that uploading of file should not block the UI and movement of user to other screens till file is fully uploaded. This will happen in background using service.

dependent on UI issue #290 and server side issue https://github.com/Cloudkibo/CloudKibo/issues/460

Similar Android issue: https://github.com/Cloudkibo/Android/issues/167

jekram commented 7 years ago

@sumairasaeed

Please describe exactly what will be done

sumairasaeed commented 7 years ago

Sir, in this issue, we will write logic to download file from server in background

sumairasaeed commented 7 years ago

Mistakenly added it in todays deck. this is dependent on #290 so will do that first

sumairasaeed commented 7 years ago

This is blocked until #292 is done

sumairasaeed commented 7 years ago

Error in #292 is corrected so it is unblocked now.

jekram commented 7 years ago

Fantastic. Good Job

sumairasaeed commented 7 years ago

Worked on this. Currently it is downloading file and server returns status as "success". We first check if there is any pending file which needs to be downloaded. If file is pending to be downloaded, we request for download.

Issue is we are not able to locate downloaded file.

Remaining work on this is:

Save downloaded file into correct path Send download confirmation to server

jekram commented 7 years ago

I do not understand your comment: "Issue is we are not able to locate downloaded file"?

sumairasaeed commented 7 years ago

Sir, I meant that file downloaded is saved in application container but i was not able to locate it. Later i found that it is not saving file with filename i was searching. It saves file with fixed filename called "download" . Also i found that it is only downloading one single same file everytime. It is not downloading second file shared.

jekram commented 7 years ago

so what is the resolution and next step?

sumairasaeed commented 7 years ago

Worked more on this. Now we are able to save file with correct filename with which it was sent. We are able to correctly download file and send confirmation to server without any error. Also worked on showing image on UI. Images are shown on UI but are not correctly alligned.

It needs some more work to allign it properly. Currently images are rendered correctly. But documents are not. Also, documents and images are rendered differently on UI. We need some identifier that file shared is "image" or "document" so we can render correctly.

sumairasaeed commented 7 years ago

Worked on this. Images are now rendered and shown correctly. Files correctly identified as "images" or "documents" with help of file extensions. Correct metadata is stored in database. There is minor error of alignment of documents UI. working on correcting it. will update more on this within an hour.

jekram commented 7 years ago

Thanks

sumairasaeed commented 7 years ago

This has been completed. Pushing app now. Will mark it as resolved after some more testing.

jekram commented 7 years ago

Thanks. Great Job

sumairasaeed commented 7 years ago

Reopening it to look into speed issue when downloading file.

sumairasaeed commented 7 years ago

I have worked on speed issue of file download. I investigated and found that our networking library Alamofire allows 2 types of file download. (1) in memory-download using request data API and (2) on-disk using download API. Alamofire library recommends to use on-disk-download for larger files and for video streaming.

We were previously using on-disk download API for all files so it was slow. Now, I have changed it to use in-memory download for files less than a threshhold size. Currently i have set threshhold size as 10MB.

I have noticed that download speed has significantly increased after this change. Some work is remaining on this task to perform this download in background.

jekram commented 7 years ago

Thanks for the update.

What is the improvement ? Significantly cannot be quantified.

As recommended before send a large file on Whatsapp and KiboChat and let me know what is the difference.

Our current logic was designed for background?

sumairasaeed commented 7 years ago

I have now fixed the error. Now it is happening on background thread and not freezing the UI. Needs to do some more testing to quantify speed. Will mark it as resolved after testing today.

jekram commented 7 years ago

How long does it take? why are we pushing this next day?

sumairasaeed commented 7 years ago

I further worked on this for improving speed for upload and download. I found that watsapp downsizes images and then compresses images and files before uploading them. We were previously sending images without any compression. I have worked and implemented this for images.We are now compressing images and then upload it on server. This change has increased the speed for upload/download of images and also reduced the file size so lesser storage is consumed. Speed will further be increase if we downsize images in a way that quality will not be affected. Here is a link which discusses how Watsapp is sharing images: https://www.quora.com/What-is-the-technique-algorithm-used-by-messengers-like-WhatsApp-and-BBM-to-compress-images Working to downsize image size to bring further improvement.

Also, I have investigated on compressing documents. Here is an article on compressing documents libraries: https://www.infoq.com/news/2016/07/apple-lzfse-lossless-opensource

I have looked into Apples Documentation about Compressing Data. It discusses 4,5 algorithms and their pros and cons. Here is the summary of compression algorithms:

Choose an algorithm according to the following guidelines:

-

Use LZ4 if speed is critical, and you are willing to sacrifice compression ratio to achieve it.

Use LZMA if compression ratio is critical, and you are willing to sacrifice speed to achieve it. Note that LZMA is an order of magnitude slower for both compression and decompression than other choices.

Otherwise, if speed and compression are more or less equally important, use LZFSE unless you require interoperability with non-Apple devices.

If you do require interoperability with non-Apple devices, use ZLIB.

After reading Apples documentation, conclusion was that ZLIB is well suited for us as we need interoperability with Android. Also, speed and quality both are equally important for us.

iOS and Android has data compressing libraries. "Zlib" is a common algorithm available for both Android and iOS. Here is the library in Swift for Zlib compression: https://cocoapods.org/pods/SwiftCompressor Here is library of Zlib compression for Android: https://developer.android. com/reference/java/util/zip/Deflater.html (. @sojharo FYI ) . Working to use Zlib compression into our App.

Also, I came across a sample demo application for Android which claims to use WatsApp type image compression. It might be helpful for @sojharo : http://voidcanvas.com/whatsapp-like-image-compression-in-android/

On Sat, Nov 26, 2016 at 9:56 AM, Cloudkibo notifications@github.com wrote:

How long does it take? why are we pushing this next day?

— You are receiving this because you were assigned. Reply to this email directly, view it on GitHub https://github.com/Cloudkibo/iOS/issues/293#issuecomment-263044281, or mute the thread https://github.com/notifications/unsubscribe-auth/AKbhp0sB29NROVsr-DqosuZrw9h08jmcks5rB7v_gaJpZM4JUhel .

jekram commented 7 years ago

@sumairasaeed

When will task will get done?

Has the current code pushed to test flight? What is the current time vs. Whatsapp? (I have asked this question before and have never been given the answer).

We should be doing design and investigation first. Not after the coding is complete. This is the third version. This is not an academic project. What you have pointed is is correct. However, the bigger question is why we do not investigate and design to begin with?

@sojharo FYI

jekram commented 7 years ago

Please look at two dumps that were sent today.

sumairasaeed commented 7 years ago

After changes and implementation of Compressing images, Image transfer speed for upload/download is now taking same time on watsapp and KiboChat (tested 1.8MB image took 3-4 seconds for upload and 3-4 seconds for download)

Sir, speed of uploading a document is same as watsapp and in some cases performed better than watsapp (e.g. it took 23 sec on KiboChat and 27sec on WatsApp).

For downloading documents, there are now 2 ways. On-disk download and in-memory download. We need to define a threshold file size. If file size is less than threshold, we use in-memory, else we use on-disk. Threshold value is upto us to set. There is no default. I tested on both methods.

File Size: I tested files of size 1.1MB and 11MB 1- On-disk (recommended for larger files and videos) With this method, it took around twice as much time as WatsApp

2- In-memory: It took same time as watsapp.

File Size: I tested files of size 24MB (large file) 1- On-disk (recommended for larger files and videos) With this method, it took around same time as WatsApp (WatsApp took 7minutes and KiboChat took 8minutes)

In-memory is not recommended for large files (e.g.24.1MB).

The time difference in larger file download can be improved further if we do document compression.

On Sun, Nov 27, 2016 at 1:34 AM, Cloudkibo notifications@github.com wrote:

Please look at two dumps that were sent today.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cloudkibo/iOS/issues/293#issuecomment-263084198, or mute the thread https://github.com/notifications/unsubscribe-auth/AKbhp41Y5A60P83z_Tno5-GOQspAyXx2ks5rCJfcgaJpZM4JUhel .

sumairasaeed commented 7 years ago

Okay sir. Looking into the dump

On Sun, Nov 27, 2016 at 3:42 AM, Sumaira Saeed sumaira.syedsaeed@gmail.com wrote:

After changes and implementation of Compressing images, Image transfer speed for upload/download is now taking same time on watsapp and KiboChat (tested 1.8MB image took 3-4 seconds for upload and 3-4 seconds for download)

Sir, speed of uploading a document is same as watsapp and in some cases performed better than watsapp (e.g. it took 23 sec on KiboChat and 27sec on WatsApp).

For downloading documents, there are now 2 ways. On-disk download and in-memory download. We need to define a threshold file size. If file size is less than threshold, we use in-memory, else we use on-disk. Threshold value is upto us to set. There is no default. I tested on both methods.

File Size: I tested files of size 1.1MB and 11MB 1- On-disk (recommended for larger files and videos) With this method, it took around twice as much time as WatsApp

2- In-memory: It took same time as watsapp.

File Size: I tested files of size 24MB (large file) 1- On-disk (recommended for larger files and videos) With this method, it took around same time as WatsApp (WatsApp took 7minutes and KiboChat took 8minutes)

In-memory is not recommended for large files (e.g.24.1MB).

The time difference in larger file download can be improved further if we do document compression.

On Sun, Nov 27, 2016 at 1:34 AM, Cloudkibo notifications@github.com wrote:

Please look at two dumps that were sent today.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cloudkibo/iOS/issues/293#issuecomment-263084198, or mute the thread https://github.com/notifications/unsubscribe-auth/AKbhp41Y5A60P83z_Tno5-GOQspAyXx2ks5rCJfcgaJpZM4JUhel .

jekram commented 7 years ago

@sumairasaeed Please ignore the comment about dumps. Wrong thread

jekram commented 7 years ago

Let's use 1 MB as the threshold.

Also, coordinate with @sojharo when implementing the compression. We need to have this logic on all end devices. Publish the current code.

sumairasaeed commented 7 years ago

Ok sir. I have already pushed current code. It is in progress. Will change value of threshhold to 1MB in next push.

On Sun, Nov 27, 2016 at 3:57 AM, Cloudkibo notifications@github.com wrote:

Let's use 1 MB as the threshold.

Also, coordinate with @sojharo https://github.com/sojharo when implementing the compression. We need to have this logic on all end devices. Publish the current code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cloudkibo/iOS/issues/293#issuecomment-263091030, or mute the thread https://github.com/notifications/unsubscribe-auth/AKbhp3rGV1grNBXY8djJxVjwutA2ZPpAks5rCLlygaJpZM4JUhel .

sumairasaeed commented 7 years ago

I have worked on compressing documents. There were some error in linking compression library which were solved. I integrated library and documents are successfully getting compressed and uncompressed. I have just carried a small test on 24.1MB file and speed was same as Watsapp. I will push the app after meeting. We can close this today after some testing.

On Sun, Nov 27, 2016 at 4:01 AM, Sumaira Saeed sumaira.syedsaeed@gmail.com wrote:

Ok sir. I have already pushed current code. It is in progress. Will change value of threshhold to 1MB in next push.

On Sun, Nov 27, 2016 at 3:57 AM, Cloudkibo notifications@github.com wrote:

Let's use 1 MB as the threshold.

Also, coordinate with @sojharo https://github.com/sojharo when implementing the compression. We need to have this logic on all end devices. Publish the current code.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cloudkibo/iOS/issues/293#issuecomment-263091030, or mute the thread https://github.com/notifications/unsubscribe-auth/AKbhp3rGV1grNBXY8djJxVjwutA2ZPpAks5rCLlygaJpZM4JUhel .

sumairasaeed commented 7 years ago

Tested and speed issue has been resolved now.

jekram commented 7 years ago

Fantastic :-)

Have we implemented compression?

sumairasaeed commented 7 years ago

Yes sir. We are compressing both images and documents.

On Tue, Nov 29, 2016 at 9:47 AM, Cloudkibo notifications@github.com wrote:

Fantastic :-)

Have we implemented compression?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/Cloudkibo/iOS/issues/293#issuecomment-263475193, or mute the thread https://github.com/notifications/unsubscribe-auth/AKbhpwiyr-uE_4dVdsxu7dJQjfsI0C3pks5rC65WgaJpZM4JUhel .

jekram commented 7 years ago

@sojharo I am assuming that you are also, implementing compression or already have?

Cloudkibo / iOS

Downloading the file in background #293

Use LZ4 if speed is critical, and you are willing to sacrifice compression ratio to achieve it.

Use LZMA if compression ratio is critical, and you are willing to sacrifice speed to achieve it. Note that LZMA is an order of magnitude slower for both compression and decompression than other choices.