search

plateaukao / einkbro

A small, fast web browser based on Android WebView. It's tailored for E-Ink devices but also works great on normal android devices.
Other
1.15k stars 80 forks source link

EinkBro crashes if `Auto update adblock` is checked and custom AdBlock filter is used #226

Closed winstonma closed 1 year ago

winstonma commented 1 year ago

Please write necessary information as described below. If information is not enough, I may not have time to look into it.

What device and app version are you using

Describe the bug I added OISD adblock filter on EinkBro and after several restart the EinkBro crash on start.

To Reproduce Steps to reproduce the behavior.

Example description:

  1. Open EinkBro and then go to Settings→Start Control→Adblock list url
  2. Add https://big.oisd.nl/ (OISD blocklist big) and check Auto update adblock
  3. Close EinkBro
  4. After a while EinkBro would crash

Expected behavior A clear and concise description of what you expected to happen.

Screenshots If applicable, add screenshots to help explain your problem.

Additional context Add any other context about the problem here.

winstonma commented 1 year ago

Just tried Step 2 using https://small.oisd.nl/ (OISD blocklist small) and EinkBro behave normally. Not sure if adblock list size matters.

Just wonder if I could put multiple adblock lists onto this entry

plateaukao commented 1 year ago

@winstonma No, currently EinkBro does not support multiple adblock lists. I tried the url and it works fine on Android Emulator. Could you give more information about what device you're using, and what Android OS is it?

Also, after modifying the list, please click on Update Adblock content button and wait until it's successfully updated.

winstonma commented 1 year ago
  1. I am using Redmi Note 11 which have 6GB of RAM
  2. I just tested 1Host Pro list (url: https://o0.pages.dev/Pro/adblock.txt) on my phone and it worked. But EInkbro on my Kindle Fire HD (2GB of RAM) crashed 3 minutes after adding the adblock list.
  3. After adding the OISD blocklist big list and pressed Update Adblock content it would not crash right away. I have to restart EinkBro for several times (maybe after 5 minutes) then EinkBro would start crashing on start.
plateaukao commented 1 year ago

Did you see the toast showing update adblock succeeds after pressed Update Adblock content (maybe after several seconds when file is downloaded)?

plateaukao commented 1 year ago

Please try adblock list url which has the same format as: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

The adblock lists you used are all more sophiscated and support kind of matching pattern. For now, EinkBro does not support those patterns now.

winstonma commented 1 year ago

Did you see the toast showing update adblock succeeds after pressed Update Adblock content (maybe after several seconds when file is downloaded)?

Yes it said Adblock is updated and EInkbro works normally until I close and then open EInkBro again after 5 minutes.

Please try adblock list url which has the same format as: https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

The adblock lists you used are all more sophiscated and support kind of matching pattern. For now, EinkBro does not support those patterns now.

Yes I would use StevenBlack list

winstonma commented 1 year ago

Here is part of the logcat for your reference

03-25 23:11:52.906 23652  4593 E AndroidRuntime: FATAL EXCEPTION: DefaultDispatcher-worker-1
03-25 23:11:52.906 23652  4593 E AndroidRuntime: Process: info.plateaukao.einkbro, PID: 23652
03-25 23:11:52.906 23652  4593 E AndroidRuntime: java.lang.NullPointerException: it must not be null
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at z5.b.x(Unknown Source:30)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at z5.b.t(Unknown Source:0)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at z5.b$e.k(Unknown Source:123)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at a7.a.q(Unknown Source:11)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at r7.y0.run(Unknown Source:88)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at kotlinx.coroutines.scheduling.a.s(Unknown Source:0)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at kotlinx.coroutines.scheduling.a$c.d(Unknown Source:14)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at kotlinx.coroutines.scheduling.a$c.n(Unknown Source:28)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    at kotlinx.coroutines.scheduling.a$c.run(Unknown Source:0)
03-25 23:11:52.906 23652  4593 E AndroidRuntime:    Suppressed: r7.w0: [r2{Cancelling}@eed43dd, Dispatchers.Default]
plateaukao commented 1 year ago

This may be the crash logs uploaded to Google by similar devices.

image

Looks like these adblock list does not have date information in it. # Date: 21 March 2023 20:03:11 (UTC)

winstonma commented 1 year ago

Hmm... I checked both Adblock list and it is possible that those Adblock filters don't have the Date entry

OISD Big

[Adblock Plus]
! Version: 202303251205
! Description: Block. Don't break.
! Title: oisd big
! Last modified: 2023-03-25T12:05:29+0000
! Expires: 1 days (update frequency)
! Homepage: https://oisd.nl
! Syntax: Adblock Plus Filter List

1Host Pro

! Title: 1Hosts (Pro)
! Last modified: 2023-03-20T15:40:03.437Z
! ..............
!  |URL: https://github.com/badmojr/1Hosts
!  |Count: 125,081 rules!
!  |Size: 2.65 MB
!  |
!  |Credits & ©: https://o0.pages.dev/-data/lists/assets.txt
!  |Licence: MPLv2,
!  | https://www.mozilla.org/en-US/MPL/2.0/
!  /
! ( Please report any issues related to this file.
!  \......................................
!  | GH: github.com/badmojr/1Hosts/issues
!  | e-mail: badmojr@gmail.com
!  | Telegram: t.me/badmojr
!  \____________________
winstonma commented 1 year ago

@plateaukao I am trying to understand how Adaway source obtain the Date entry (but may need your help)?

Github Hosted File I just try to get how they obtain the Date of the host file using Github (e.g. StevenBlack)

$ curl -s "https://api.github.com/repos/StevenBlack/hosts/commits?per_page=1&path=hosts" | jq '.[0].commit.committer.date'
"2023-03-25T17:09:26Z"

Other Seems they are using the last-modified entry:

$ curl -I https://small.oisd.nl/hosts | grep `last-modified`
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
last-modified: Sat, 25 Mar 2023 12:05:34 GMT

And for other file without last-modified (e.g. 1Hosts) it seems to me that they just update on a weekly basis

winstonma commented 1 year ago

@plateaukao Do you think it would be a good idea if EinkBro force update if the Date check fail?

plateaukao commented 1 year ago

@winstonma

Although the crash is caused by not having Date in adblock list, the bigger problem is that the content format is different. Even if you fixed Date issue, it does not help because EinkBro does not read format from https://big.oisd.nl/ or other url you provided.

You can see formats below: StevenBlack only list IPs and their domain names, but OISD and 1HOSTs are more complicated, and need more logic to handle the content and do the comparison in codes, which are not yet implemented in EinkBro, and in short time, I don't have intention to do so because current implementation already blocked most of the ads I've seen.

Before EinkBro support this kind of format, the best way you can do is fetch content from OISD and 1HOST, and convert their content to be of the same format as StevenBlack site, and serve the url by yourselves (for example, put it on your github, and use the file link to get it).

stevenblack

0.0.0.0 ck.getcookiestxt.com
0.0.0.0 eu1.clevertap-prod.com
0.0.0.0 wizhumpgyros.com
0.0.0.0 coccyxwickimp.com
0.0.0.0 webmail-who-int.000webhostapp.com
0.0.0.0 010sec.com
0.0.0.0 01mspmd5yalky8.com
0.0.0.0 0byv9mgbn0.com

https://big.oisd.nl/

||0--foodwarez.da.ru^
||0-000.store^
||0-24bpautomentes.hu^
||0-29.com^
||0-800-email.com^
||0-day.us^

1hosts

||0--foodwarez.da.ru^
||0-000.store^
||0-24bpautomentes.hu^
||0-29.com^
||0-800-email.com^
||0-day.us^
plateaukao commented 1 year ago

@winstonma

I tried to add basic support for format exactly as ||domain_name^ for sites you provided, and fixed the issue of Date check in commit https://github.com/plateaukao/einkbro/commit/50339e5621cdf11b60e995a7e321e5c6fd8c90e3

Please have a try with snapshot_zip button on readme. :)

In current implementation, if you check Auto Update Adblock option in Settings>Start Control, it will try to update every week.

winstonma commented 1 year ago

Oh I think I missed the AdBlock format. I didn't mean to ask EinkBro to support non-host format adblocker file (I genuinely think that EinkBro Adblocker uses the same adblock format as uBlock running on Chrome). I just want to make sure that EInkBro would not crash.

Actually both adblock providers offer host file format too:

Before the new release I will uncheck Auto Update Adblock and use OISD Big and see if any crash.

Thanks for supporting other adblock list that doesn't contain Date:

plateaukao commented 1 year ago

I see now. I didnt know both site support host only content too.

As for the original crash, it's fixed with one line that I missed to add before. 😃 Thank you for finding this issue.

winstonma commented 1 year ago

First of all thanks for fixing the crash.

By the way I just replaced the einkbro adblock filter using Android Private DNS mode + NextDNS. Now it work not only inside EInkbro but also it's system-wide blocking. Seems it makes EinkBro run faster too.

It's just share of alternative adblocking method.

plateaukao commented 1 year ago

@winstonma Thanks for sharing the info. I will give a try too.