search

autopkg / nstrauss-recipes

Public AutoPkg recipes
23 stars 27 forks source link

VirusTotalReporter Rate Limiting #79

Closed mihelm closed 1 month ago

mihelm commented 1 month ago

Hi @nstrauss

Thanks a lot for VirusTotalReporter! I've tested it a few times and would like to use it on a large set of recipes. However, I'm hesitant to do so without rate limiting, which you list as a to do.

Would it make sense to adopt the manner in which VirusTotalAnalyzer does that? Or did you have a different approach in mind?

Cheers, michael.

nstrauss commented 1 month ago

VirusTotalAnalyzer doesn’t handle rate limiting at all. It only attempts to catch JSON load errors and assumes rate limiting is to blame. I’d like to eventually implement exponential backoff, but haven’t found cycles or a reason to yet.

https://github.com/hjuutilainen/autopkg-virustotalanalyzer/blob/e113937171b1ae58a0f17bfae3f05f3b538d6d78/VirusTotalAnalyzer/VirusTotalAnalyzer.py#L139-L145

I use the community key with dozens of recipes on a daily basis without being rate limited. Anecdotally, I’ve only hit rate limiting issues when submitting the same package multiple times in quick succession when testing. My recommendation is to try the community key, see if you hit rate limits, and if so register your own VirusTotal account to generate a personal key. https://virustotal.readme.io/docs/please-give-me-an-api-key

I’m also happy to review and accept PRs if you’d like to contribute the feature. Glad you’ve found the processor useful and I hope it does get more adoption.

mihelm commented 1 month ago

Hi @nstrauss

I've submitted two PRs. The first #80 fixes a small bug that occurs when submission_timeout is supplied via the CLI to VirusTotalReporter as a PostProcessor.

The second #81 adds an implementation of an exponential backoff.

I've just run the version with both changes against a large set of recipes, and it seems to be working quite nicely.

Cheers, michael.