Implement Filtering Log v1

ameshkov commented 9 years ago

View 1 - Filtering Log is stopped

Placeholder Start button Warning about performance

Filtering Log Notification

When filtering is started we should show an ongoing notification about this.

View 2 - Filtering Log is started

List of log records.

Show for every log record: URL App name Source domain Third-Party icon

Log record color indicates if request was blocked or whitelisted. Red for blocked, green for whitelisted, white bg - for processed.

View 3 - Log Record Details

Tapping on log record opens log record details view.

URL Type Source domain Filter Rule and Filter Subscription name for blocked and whitelisted requests

Buttons: Open in browser Block for common requests Unblock for blocked requests (Unblock creates whitelist rule) Nothing for requests whitelisted by some filter subsription

View 4 - Filter Rule Creation

Example of such dialog in browser add-on: http://screencast.com/t/kbttytH1SWj

New rules should be added to user filter.

Nick-White commented 9 years ago

+1 This is the feature I'm waiting the most because there are many ads that go through all the filter subscriptions...

ameshkov commented 9 years ago

For now you can report these to us: https://github.com/AdguardTeam/ExperimentalFilter/issues

Or you can read AG logs and find out what is not blocked: https://github.com/AdguardTeam/ExperimentalFilter/wiki/Android

ameshkov commented 7 years ago

@Revertron here is a list of comments for the first filtering log implementation

Filtering log icon

Use this icon for the navigation menu item: filtering_log_icons.zip

Please let me know what sizes do you need.

Action bar

We need an option to clear the filtering log Put it in the action bar. Use the same icon we have in the user filter/whitelist.

"Enabled" should the same size/weigth as action bar title has.

Empty filtering log

Use the same icons and layout as in the case of user filter/whitelist. Text: Filtering log is empty

Log item

Time and URL color should be of our regular gray color.

Revertron commented 7 years ago

For small icons (as 24dp in UI) we use these sizes: 96, 72, 48, 24, 36 pixels.

ameshkov commented 7 years ago

@Revertron got it, what color do we need for the navigation menu?

Revertron commented 7 years ago

Other icons have $757575 color.

ameshkov commented 7 years ago

Color indication

Change the line background color to indicate request status.

Regular requests: no indication. Blocked requests: #fff4f2 (pressed state: #ffb6a9) Whitelisted requests: #defae5 (pressed state: #98ffb2)

ameshkov commented 7 years ago

Log item details fields should depend on the event type. However, there are some common requirements.

Common requirements

Tap on a field should automatically copy it's value to the clipboard. We should also show a toast about it.
The Status field possible values: Processed, Blocked, Threat, Whitelisted, Redirect. The latter makes sense for Web request type only.
Event type field possible values are Web request, DNS request, HTTPS tunnel
Application field name depends on the app type and can be App, Browser, System app
Filtering rule field is visible in two cases only: if the status is either Blocked or Whitelisted.
Possible buttons in the item details footer: BLOCK, UNBLOCK (raised buttons), OPEN (flat button). OPEN button opens request URL in the default browser. OPEN is available for regular web requests with GET method only.
Remote Address field format is 127.0.0.1:443 for IPv4 and [2001:4860:4860::8888]:443 for IPv6. Use IpAddressUtils.toString() method to get the proper format for it. Do not show this field if value is empty.

Regular web request fields

Event type
Application
Domain
Remote Address
Status (You should use Redirect status whenever redirectUrl field is not empty.)
Filtering rule
Request URL
Source URL (do not show if empty or null)
Redirect URL (show it if status is Redirect)
Content type (no need to translate, just use the enum value)
Start time (format: 18:45.123) (we don't really need date here)
Elapsed time (format: 123 ms)
Size (format: ↓123KB ↑5KB)

Please note, that Event type in the case of regular web request also contains HTTP method value. It should look like:

Value: Web request (GET)
Field name: Event type

DNS request fields

Event type
Domain
Status
Filtering rule
Start time
Elapsed time
Size

Tunnel request fields

Event type
Application
Domain
Remote Address
Status
Filtering rule
Start time
Elapsed time
Size

ameshkov commented 7 years ago

Filtering log menu item icon: filtering_log_icons.zip

Fields icons: filtering_log_fields_icons.zip

ameshkov commented 7 years ago

@Revertron just caught a crash:

22:52:00.037 [main] ERROR c.a.a.c.CustomExceptionHandler - Thread: Thread[main,5,main]; Uncaught exception:

java.lang.UnsupportedOperationException: null
    at java.util.AbstractList.add(AbstractList.java:148) ~[na:0.0]
    at android.widget.ArrayAdapter.insert(ArrayAdapter.java:255) ~[na:0.0]
    at com.adguard.android.ui.FilteringLogActivity$3.run(FilteringLogActivity.java:119) ~[na:0.0]
    at android.os.Handler.handleCallback(Handler.java:751) ~[na:0.0]
    at android.os.Handler.dispatchMessage(Handler.java:95) ~[na:0.0]
    at android.os.Looper.loop(Looper.java:154) ~[na:0.0]
    at android.app.ActivityThread.main(ActivityThread.java:6119) ~[na:0.0]
    at java.lang.reflect.Method.invoke(Native Method) ~[na:0.0]
    at com.android.internal.os.ZygoteInit$MethodAndArgsCaller.run(ZygoteInit.java:886) ~[na:0.0]
    at com.android.internal.os.ZygoteInit.main(ZygoteInit.java:776) ~[na:0.0]

ameshkov commented 7 years ago

Rules constructor issues:

Does not work properly for DNS requests
Patterns should start with ||
Add one more option: [x] Apply to the APPNAME requests only. In case if this box is checked, add $app modifier to the rule text.

ameshkov commented 7 years ago

I see that startappexchange.com SDK sometimes sends a GET request with URL length more than 20000. Actually, every time I start ru.sports app, I see this request. We'd better handle long URLs specifically.

ameshkov commented 7 years ago

Add $important modifier support to the rules constructor dialog.

Add a checkbox: Give a higher priority to the rule
By default, this checkbox should be disabled.
However, if you try to unblock a URL blocked with an $important rule, or block a whitelisted URL, make it checked.

ameshkov commented 7 years ago

Handle third-party flag in a smart way

If web request is third-party, "Apply to third party requests only" option should be checked.
Otherwise, let it be unchecked.

Revertron commented 7 years ago

Done.

Alex-302 commented 7 years ago

Issues:

Incorrect content type for Web-Socker connections wss://ws.hghit.com/ws Content type = Document Add @@||hdrezka.me^$content and go to hdrezka.me;
~After clearing, filtering log again contains the records;~
~Request sorting is strange - requests from the page are located before request to the page~ [Moved to #1074]

~Elements, which was removed from HTML are not displayed(like in Adguard for Windows)~ [Moved to #1075]

ameshkov commented 7 years ago

Just fixed point 1.

Regarding point 3, this is by design (at least for now). We should fix it eventually, but we'd better move it to a separate issue.

Alex-302 commented 7 years ago

Regarding point 3

done https://github.com/AdguardTeam/AdguardForAndroid/issues/1074

Alex-302 commented 7 years ago

~Added

Elements, which was removed from HTML are not displayed(like in Adguard for Windows)~

Revertron commented 7 years ago

Fixed point 2.

AdguardTeam / AdguardForAndroid