search

designsecurity / progpilot

A static analysis tool for security
MIT License
332 stars 61 forks source link

Excluded file still processed if explicitly passed #28

Closed wpenton-itc closed 2 years ago

wpenton-itc commented 3 years ago

If you add a file to the "exclude_files" section of a configuration file but then pass that file to be processed the file is still processed instead of being excluded.

My case: I have a VERY large project and for some reason there are a few files within the project that causes Progpilot to crash. For now I want to exclude those files to be able to still process everything else. My issue is that I am doing a diff between my current working branch to the master branch to get a list of files that have been changed, then pass those filenames to progpilot to be processed. Mind you this way saves processing time but it still takes over 30 minutes to run, as opposed to 3-4 hours (I told you it was a large project).

eric-therond commented 3 years ago

Hi @wpenton-itc about performances problems, it's a lot of work, I plan to do something at the end of the year.

Normally excluded files/folders should not be analyzed. Can you share your exclude_files configuration? and the project structure?

Eric

wpenton-itc commented 3 years ago

Well I am not allowed to post the actual list of directories (company policies and such) but there are 11074 directories in this project. This is the exclusions section:

  exclude_files:
    - "/usr/www/trunk/.blackfire.yml"
    - "/usr/www/trunk/.dockerignore"
    - "/usr/www/trunk/.editorconfig"
    - "/usr/www/trunk/.env"
    - "/usr/www/trunk/.env.example"
    - "/usr/www/trunk/.env.testing"
    - "/usr/www/trunk/.env.trunk"
    - "/usr/www/trunk/.php_cs.dist"
    - "/usr/www/trunk/.psysh.php"
    - "/usr/www/trunk/.artifactignore"
    - "/usr/www/trunk/.phpstorm.meta.php"
    - "/usr/www/trunk/c3.php"
    - "/usr/www/trunk/app/Providers/FakerServiceProvider.php"
    - "/usr/www/trunk/customerPost.php"
    - "/usr/www/trunk/Classes/DropdownClass.php"
  exclude_folders:
    - "/usr/www/trunk/.git"
    - "/usr/www/trunk/.internal"
    - "/usr/www/trunk/vendor"
    - "/usr/www/trunk/node_modules"
    - "/usr/www/trunk/.sass-cache"
    - "/usr/www/trunk/.idea"

But when working on DropdownClass.php that file now shows up in the git diff of files that have been modified and "should be scanned" but because it is listed in the excludes it should be, well, excluded. :)

eric-therond commented 3 years ago

How to you run progpilot?

progpilot --configuration configuration.yml /usr/www/trunck/

I forgot to ask, in the configuration file what are the values for:

Eric

wpenton-itc commented 3 years ago

full configuration.yml file

inputs:
  setDev: false
  setLanguages:
    - php
  setFrameworks:
    - laravel
  setSources: ~
  setSinks: ~
  setValidators: ~
  setSanitizers: ~
  setCustomRules: ~
  setIncludes: ~
  setExcludes:
  exclude_files:
    - "/usr/www/trunk/.blackfire.yml"
    - "/usr/www/trunk/.dockerignore"
    - "/usr/www/trunk/.editorconfig"
    - "/usr/www/trunk/.env"
    - "/usr/www/trunk/.env.example"
    - "/usr/www/trunk/.env.testing"
    - "/usr/www/trunk/.env.trunk"
    - "/usr/www/trunk/.php_cs.dist"
    - "/usr/www/trunk/.psysh.php"
    - "/usr/www/trunk/.artifactignore"
    - "/usr/www/trunk/.phpstorm.meta.php"
    - "/usr/www/trunk/c3.php"
    - "/usr/www/trunk/app/Providers/FakerServiceProvider.php"
    - "/usr/www/trunk/customerPost.php"
    - "/usr/www/trunk/Classes/DropdownClass.php"
  exclude_folders:
    - "/usr/www/trunk/.git"
    - "/usr/www/trunk/.internal"
    - "/usr/www/trunk/vendor"
    - "/usr/www/trunk/node_modules"
    - "/usr/www/trunk/.sass-cache"
    - "/usr/www/trunk/.idea"
  setFile: ~
  setCode: ~
  setResolvedIncludes: ~
  setFalsePositives: ~

outputs:
  taintedFlow: false
  resolveIncludes: false
  resolveIncludesFile: ~

options:
  setAnalyzeIncludes: true
  setAnalyzeFunctions: true
  setAnalyzeHardRules: true
  setPrintFile: true
  setPrintWarning: false
  setPrettyPrint: true
  setLimitTime: 10
  setLimitDefs: 3000
  setLimitSize: 500000
wpenton-itc commented 3 years ago

Also it is run via Azure Pipelines:

docker run --rm --user="$(id -u):$(id -g)" -v $(pwd):/usr/www/trunk -w /usr/www/trunk $IMAGES_URL/progpilot $(cat .internal/phpstan/files.txt | tr '\n' ' ')
eric-therond commented 3 years ago

You have to pass --configuration option in the command line with the path to your configuration file:

docker run --rm --user="$(id -u):$(id -g)" -v $(pwd):/usr/www/trunk -w /usr/www/trunk $IMAGES_URL/progpilot --configuration /path/to/configuration.yml $(cat .internal/phpstan/files.txt | tr '\n' ' ')
wpenton-itc commented 3 years ago

Wow I feel stupid.... 🤦 Thank you! I will not be able to check this until next week.

sarfrazeteam commented 3 years ago

BTW, i don't see laravel supportted and you have added that. See:

https://github.com/designsecurity/progpilot/tree/master/package/src/uptodate_data/php/frameworks

Not sure if i am missing something. I also want to use this for laravel but not sure if it is supportted.