feat: Added my own checkers - count is 19K

[bakasur89]

I have added 19k+ checkers into the tool, but somehow it is not working it gives error as TypeError:'module' object is not callable.

1. I want to use this tool on windows.
2. my aim is to create each and every checker, for which the product is present in DB.
3. 19k already created. more to go.
4. there is NO issue in the checker, as when I try to run these checkers but not all 19k together, but as 2-3k checkers at a time it works.
5. but when it comes to 19k together it fails.

Also, I want to make sure when I achieve 19k checkers for now, it should run quickly.

Please Please help me, I need to achieve this at any cost.

Best regards, bakasur89

[terriko]

It looks like maybe you have a typo in __init__.py

Please try getting 1 or 2 checkers merged before you start a project of this scale. Most people have trouble with the first couple.

That said 19k, checkers is awfully ambitious:

• you're probably going to rethink how checkers are loaded so you don't have to add them all to __init__.py. Currently it's set up to make importing all of them easier, but that may not be a thing anyone wants to do if you have 19k of them.
• As far as performance goes: 19k checkers means running more than 19k regexes against every single string encountered by the tool. That's going to be slow no matter how you slice it. I think some of the commercial tools I've used do this and we're talking 4hr+ execution times on beefy servers rather than the maybe 30 minutes that we run in cheap docker containers.
• You may need to rethink some architecture to make this viable, such as how to mass-disable things based on various criteria (like "is this likely to run on the platform I'm scanning?"), or different ways to batch things.
• You likely will need to re-implement parts of our scanning engine in a more performant language such as rust. (I love Python but the best way to make things faster in python remains to not use python for the parts that are slowest.)
• I also have serious doubts as to whether all of those 19k have viable signatures (sometimes things don't with our current scanning methodology).

I don't know why you're saying you need to do this at any cost, but I just want to be clear that you're talking about a development process that would probably require a full team of engineers and potentially years of work. Which isn't to say it wouldn't be awesome, or even necessarily that it'd be infeasible, but maybe start a bit slower and see if you can get 1-2 checkers merged before trying to make a checker for everything?

[chenjianquan7]

what is the checker with 19k? thanks