App Stuck on Downloading and Parsing Adlists

Doodle-Aks commented 2 years ago

Is there an existing issue for this?

[X] I have searched the existing issues

Current Behavior

Hello, I've been a huge fan of Adaway (Root) and I'm glad there's an option to use it without root. I've been using it for a couple of months and I'd like to report a few issues I've had with it. Hope these could be addressed.

I've added a few preferred adlists as URLs under "Hosts Source". On some occasions, the app downloads and parses through all of them and adds the respective entries to its block list.

However, on most occasions, the app checks for adlist updates, reports there are updates & where the download icon is clicked, it gets stuck on parsing "*****"adlist. I would like to reiterate that this happens on most occasions and when the UI enters the app switcher, the download/parsing gets reset and the app restarts the process once again when the download button is clicked.

The parsing issue I've reported occurs mostly when the adlist is massive and contains tens/thousands of entries.

Improvement/Suggestion : A clear update progress bar (with completion percentages) instead of an infinite progress bar would be really helpful to gauge whether the application is stuck parsing this/it's working its way through voluminous adlists.

Device Config & Relevant Info : Samsung Galaxy Note 20U (Exynos) Android 10 - One UI 2.5 Adaway v5.12.x (Occurs On 5.12 & 5.12.1) App Protocol Used : Non-Roor - Local VPN API

Expected Behavior

Click the Check for Host Source Adlist Updates button - Reports Source Updates available or Sources Up-to-Date.
Click the download button to Download Source Updates and parses the adlists
Update the Blocked Sources DB or a progress bar with percentages indicating the update progress.

Steps To Reproduce

Add these adlists to Hosts Source : a. Developer Dan - AD Blocking List : .https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt b. . StevenBlack Unified Block list : https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts c. Firebog's Adlists, Tracking & Malware Hosts (Top 3 Approved/Recommended/updated Regularly - Marked with a Green Tick).
Click the Check Host Source Updates Button .
Click the Download Updates Button to fetch new lists from these sources/download updates for existing sources.
Gets stuck on parsing "" adlist. Keep the App open for 5 min or so and still no progress - Continues parsing the "" adlist - doesn't update the block list DB or proceed to the next adlisr/source.

Device Configuration

No response

Additional context and screenshots

No response

github-actions[bot] commented 2 years ago

Hello @Doodle-Aks

It looks like you report a bug without providing all the context details like the AdAway version you use. Thanks to update your issue with the most details you can, and use the bug report template instead of free form issue.

Regards

Doodle-Aks commented 2 years ago

Hello @Doodle-Aks

It looks like you report a bug without providing all the context details like the AdAway version you use. Thanks to update your issue with the most details you can, and use the bug report template instead of free form issue.

Regards

I have provided all the relevant details under my issues and skipped the "Device" Section. Hope that's okay.

PerfectSlayer commented 2 years ago

Hi,

Don't worry about the bot comment, it was looking for AdAway version from the bug report template.

About the behavior you describe, I tried the hosts you list and I successfully parsed them on my dev device (Pixel 2).

2022-05-15 10:32:57.530 7217-7405/org.adaway I/SourceModel: Parsed https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt in 67s

But keep in mind the block list has nearly half a million entries... It creates a lot of pressure on your device memory during the parsing (the whole file is not loaded in memory but streaming a huge number of elements to parse and insert to the block list database create a certain amount of memory throughput).

About having a progress indicator, I already thought about it but it have several inconvenients:

I can't know for sure how many hosts there will be to parse. So I don't know what will be the 100% completion to reach,
Progress will jump at several steps when cache is hit. There is multiple caches in place that will make the progress instantly moves forward,
It might reduce the update performance. Adding a progress counter will make data flow from several workers to the UI while the device is already under pressure,
And mainly, most of the time, users let the update work in background and never come back to the homescreen to watch the UI progress. Making this feature not as worth it as it looks.

Doodle-Aks commented 2 years ago

Hi,

Don't worry about the bot comment, it was looking for AdAway version from the bug report template.

About the behavior you describe, I tried the hosts you list and I successfully parsed them on my dev device (Pixel 2).

2022-05-15 10:32:57.530 7217-7405/org.adaway I/SourceModel: Parsed https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt in 67s

But keep in mind the block list has nearly half a million entries... It creates a lot of pressure on your device memory during the parsing (the whole file is not loaded in memory but streaming a huge number of elements to parse and insert to the block list database create a certain amount of memory throughput).

About having a progress indicator, I already thought about it but it have several inconvenients:

I can't know for sure how many hosts there will be to parse. So I don't know what will be the 100% completion to reach,

Progress will jump at several steps when cache is hit. There is multiple caches in place that will make the progress instantly moves forward,

It might reduce the update performance. Adding a progress counter will make data flow from several workers to the UI while the device is already under pressure,

And mainly, most of the time, users let the update work in background and never come back to the homescreen to watch the UI progress. Making this feature not as worth it as it looks.

Hi,

Thanks for the instant response. I understand that it puts a lot of pressure on the device's memory but for newer devices, I'm not sure that'd be an issue considering most devices these days have 6-8gb.

Furthermore, as I've mentioned the issue happens on most occasions but in some instances, it parses them successfully. I'm not sure if this is OEM specific as different OEMs have different memory management algorithms. Could you help me figure out if this is the issue ?

Regarding the progress bar, I understand the limitations you've listed and tbh it makes sense to favor performance over some UI Updates that most people do not use.

Suggestions :

For the Progress Bar, if it doesn't exceed say 5MB, count the exact lines but if the file size is > 5MB, estimate the number of lines using file size.
On my device, the parsing gets reset/cancelled when the app leaves the foreground. A persistent Notification indicating host sources are being updated - (hopefully) ensure the app doesn't get killed
Optional UI Section: A summary report containing the Time it took to parse each adlist for those keen on being aware of such stats. Similar to how the below event report is reported.

2022-05-15 10:32:57.530 7217-7405/org.adaway I/SourceModel: Parsed https://www.github.developerdan.com/hosts/lists/ads-and-tracking-extended.txt in 67s

AdAway / AdAway