Urlscan.io, a Virustotal alternative

Martinligabue commented 1 year ago

Describe a related problem (optional)

No response

Describe your suggested feature

I'd like urlchecker to support the urlscan.io api, it works similar to virustotal but specifically with urls and can return if the site is safe, scam, virus and lots of other threat information.

Describe alternatives you've considered for your suggested feature

No response

Other details

No response

Acknowledgements

[X] I have searched the existing issues and this is a new ticket, NOT a duplicate or related to another open issue.
[X] I have written a short but informative title.
[X] I will fill out all of the requested information in this form.

TrianguloY commented 1 year ago

I wasn't aware of that api, but it looks promising. Thanks for the suggestion!

I wonder if a duplicated virus total module can be easily adapted to support it...

Martinligabue commented 1 year ago

I have almost no experience with this, but I'm trying things in a fork, I'll let you know if I get something usable

pressRtowin commented 1 year ago

I wasn't aware of that api, but it looks promising. Thanks for the suggestion!

I wonder if a duplicated virus total module can be easily adapted to support it...

Would it make sense to add another module just for this? I'm thinking where that leads long-term if other scanning services that are also deemed useful/noteworthy continue to be added. Maybe it'd be better to make one module that supports sending the URL to various scanning services that can be enabled/disabled in the settings?

I'm not necessarily arguing for one way or the other. I haven't thought enough about the pros and cons of either to have a definitive opinion. Just putting out a thought that occurred to me.

TrianguloY commented 1 year ago

I though about that. Maybe it's better to include a single module for scanning, but all the logic of enabling/disabling options is already "for free" in the module logic.

But it would be easy to implement, so in the end the decision is just of the one doing the code. Do you think from a user perspective it's better to have a single module? It will need to have a dedicated button for each service anyway...

pressRtowin commented 1 year ago

It will need to have a dedicated button for each service anyway...

How big of a button are we talking? Could it be handled a different way instead?

Just some quick brainstorming:

have one button that sends to all enabled services.
handle it like some of the other modules that allow you to select from multiple choices in a list, where each item and the way to interact with it only takes up one small line. Would have to figure out a way to present the results that are returned though.

TrianguloY commented 1 year ago

A single button to scan in all services is a bit dangerous. Some scanning services may publish all scanned urls, others maybe don't. If such a button is implemented it will need to be alongside the individual single-service buttons, or as a disabled-by-default option.

My first idea was to just add a row for each service (like literally copy paste the current module layout) but now that I think about it maybe a single line with buttons is the better option. This way it will take the same space, and only when you scan something it will "expand" to show the results below.

Martinligabue commented 1 year ago

Just to let you know, I also found https://www.abuseipdb.com/ which can scan urls, this might help to test how a single module for scanning with multiple services might look like.

I'm not sure I get all that's been proposed, but I think a single button that scans all the services that have been provided with an API key, and return to the user just the result: None of the services found anything/This engine found something/This, this and this services found something. (Also, since you need to provide the API key, you know what you're enabling or not)

pressRtowin commented 1 year ago

A single button to scan in all services is a bit dangerous. Some scanning services may publish all scanned urls, others maybe don't. If such a button is implemented it will need to be alongside the individual single-service buttons, or as a disabled-by-default option.

Could have some kind of "multi scan" button (rather than a scan in all button) that allows you to send the url to multiple services that you choose in settings. That way you could still list services A-F for quick manual scanning if wanted but have one button that sends to A, B, D, and F with one tap for instance.

PabloOQ commented 1 year ago

I think it should be one module only, after all some time ago the name of the modules was changed to represent better what they do (ClearURLs -> URL Cleaner), having 2 URL scanners would be confusing.

These are the options I think could be useful in the layout, in no specific order:

A spinner with the service the URL will be sent to (current service). With a preference in the settings to have the default value be random or a specific service. It hides ifself if there is one service only.
A button to send the URL to the service of the spinner.
A button to send it to all services. It hides ifself if there is one service only.

The unshort module could benefit from this change too. Both modules do different things in the same way, request something to a service. Also because unshorten.me hasn't been able to resolve t.co URLs for more than a month now, it would be wise to have an alternative, in case something else breaks. For this reason a whitelist-blacklist would be a helpful too.

Finally a way to disable-enable services (maybe using a "*" regex in the whitelist-blacklist).

On another note, it seems to me that a way to streamline automation and whitelist-blacklist would be a great addition to many modules.

Martinligabue commented 1 year ago

Perhaps all the external services could be merged somewhere else, to clean up the main screen?

Murilogs1910 commented 1 year ago

Perhaps all the external services could be merged somewhere else, to clean up the main screen?

I don't think that will be a big problem after the drawer module is added to the app.

Antsy4904 commented 9 months ago

Meanwhile you can use the pattern checker to check urlscan with your browser, no API key needed:

"Check with urlscan.io": {
    "regex1": "^(https?):\/\/(\\S*)$",
    "regex": "^(?!.*(urlscan.))(.*)",
    "replacement": "https:\/\/urlscan.io\/#$2",
    "enabled": "true"

based on this https://github.com/Raj9039852537/urlChecker-patternchecker-config/blob/a129b7b9ebb94a6d4cb3a858c24b6f997dbe46e1/miscellaneous.json#L2C27-L2C27

TrianguloY / UrlChecker