search

coala / coala-bears

Bears for coala
https://coala.io/
GNU Affero General Public License v3.0
295 stars 580 forks source link

BanditBear: Tests to execute option missing #2386

Open sangamcse opened 6 years ago

sangamcse commented 6 years ago

bandit has an option -t comma-separated list of test IDs to run which is missing in BanditBear.

difficulty/low

gitmate-bot commented 6 years ago

GitMate.io thinks possibly related issues are https://github.com/coala/coala-bears/issues/599 (test), https://github.com/coala/coala-bears/issues/1885 (KeywordBearTest: Missing language setting in test), https://github.com/coala/coala-bears/issues/2071 (test issue), https://github.com/coala/coala-bears/issues/2070 (test issue), and https://github.com/coala/coala-bears/issues/95 (Testing docs).

sangamcse commented 6 years ago

Please assign this to me 😃

jayvdb commented 6 years ago

Why is this option useful?

sangamcse commented 6 years ago

Sometime we want only a few tests to run. We can do that in our current setting of BanditBear by giving a long list of tests to skip in bandit_skipped_tests or we can add an option(in this issue) to run some specific tests.

Note: I found it odd that we don't have an option to run some specific tests. There is a lot more options https://github.com/openstack/bandit#usage

sangamcse commented 6 years ago

For example: what if I wants to run only B112 try_except_continue test? Then I have to give a long list of tests ID to bandit_skipped_tests list and that is not a good option.

Here is the list of tests with test IDs.

  B101  assert_used
  B102  exec_used
  B103  set_bad_file_permissions
  B104  hardcoded_bind_all_interfaces
  B105  hardcoded_password_string
  B106  hardcoded_password_funcarg
  B107  hardcoded_password_default
  B108  hardcoded_tmp_directory
  B109  password_config_option_not_marked_secret
  B110  try_except_pass
  B111  execute_with_run_as_root_equals_true
  B112  try_except_continue
  B201  flask_debug_true
  B301  pickle
  B302  marshal
  B303  md5
  B304  ciphers
  B305  cipher_modes
  B306  mktemp_q
  B307  eval
  B308  mark_safe
  B309  httpsconnection
  B310  urllib_urlopen
  B311  random
  B312  telnetlib
  B313  xml_bad_cElementTree
  B314  xml_bad_ElementTree
  B315  xml_bad_expatreader
  B316  xml_bad_expatbuilder
  B317  xml_bad_sax
  B318  xml_bad_minidom
  B319  xml_bad_pulldom
  B320  xml_bad_etree
  B321  ftplib
  B322  input
  B323  unverified_context
  B324  hashlib_new_insecure_functions
  B401  import_telnetlib
  B402  import_ftplib
  B403  import_pickle
  B404  import_subprocess
  B405  import_xml_etree
  B406  import_xml_sax
  B407  import_xml_expat
  B408  import_xml_minidom
  B409  import_xml_pulldom
  B410  import_lxml
  B411  import_xmlrpclib
  B412  import_httpoxy
  B501  request_with_no_cert_validation
  B502  ssl_with_bad_version
  B503  ssl_with_bad_defaults
  B504  ssl_with_no_version
  B505  weak_cryptographic_key
  B506  yaml_load
  B601  paramiko_calls
  B602  subprocess_popen_with_shell_equals_true
  B603  subprocess_without_shell_equals_true
  B604  any_other_function_with_shell_equals_true
  B605  start_process_with_a_shell
  B606  start_process_with_no_shell
  B607  start_process_with_partial_path
  B608  hardcoded_sql_expressions
  B609  linux_commands_wildcard_injection
  B701  jinja2_autoescape_false
  B702  use_of_mako_templates
jayvdb commented 6 years ago

ok, we need to look at other bears which have similar lists of checks, and work towards a consistent settings system for these. pycodestyle and pydocstyle do have similar lists of checks and command line options to include/select or ignore various checks.

jayvdb commented 6 years ago

@sangamcse , I dont see any reply to my query at https://github.com/coala/coala-bears/issues/2386#issuecomment-377512889

sangamcse commented 6 years ago

@jayvdb, bandit, pydocstyle and pycodestyle all have a different list of checks. However, pydocstyle and pycodestyle are using --ignore= and --select= as command for all their checks, while bandit uses --skip/-s to skip and --tests/-t to select, as command for all it's checks. Here is the link for pydocstyle checks and pycodestyle checks.

There are another 2 bears (PEP8NotebookBear, PEP8Bear) of autopep8 which uses --ignore and --select, as a command for it's checks.

jayvdb commented 6 years ago

@sangamcse , we dont care what the linter API is. coala wraps linters to hide their differences. We provide setting names which are more consistent, to make it easier for users to manage many different tools.

sangamcse commented 6 years ago

@jayvdb, So I looked at other bears which have similar lists of checks and I have noted their command type also. Here it is: GoErrCheckBear --> -ignore ignore_list and -ignorepkg ignorepkg_list ArtisticStyleBear --> bool type list of checks CPPCheckBear --> --enable list_of_issues_type CPPLintBear --> --filter=ignore_list,include_list HTMLLintBear --> --disable=htmllint_ignore_list HTTPoliceLintBear --> -s=httpolice_silence_ids_list PycodestyleBear --> --ignore=ignore_checks_list and --select=select_checks_list PydocstyleBear --> similar to PycodestyleBear PyLintBear --> --disable=pylint_disable_ids_list and --enable=pylint_enable_ids_list RSTcheckBear --> --ignore-language=code_block_language_ignore_list, --ignore-directives=directive_ignore_list and --ignore-roles=role_ignore_list ShellCheckBear --> -e list_of_ignore_rules

Note: I have tried to write a CheckSelector. Please suggest me and correct me if I'm doing anything wrong. 😃

jayvdb commented 6 years ago

Note there is also https://github.com/coala/coala-bears/pull/2257 , by @manankalra

jayvdb commented 6 years ago

RSTcheckBear --> --ignore-language=code_block_language_ignore_list, --ignore-directives=directive_ignore_list and --ignore-roles=role_ignore_list

I think these are a slightly different concept, especially directive & roles, as these are ignoring syntax in the rst which rstcheck doesnt know about.

sangamcse commented 6 years ago

Oh. I tried to find #2257 PR based on your suggestion earlier on gitter but was unable to find it. So, what I wrote in ChecksSelector can work with RSTcheckBear. But real difficulty is with bool type.

jayvdb commented 6 years ago

Right. So the remaining design question is whether we should be trying to merge the two varieties:

  1. lots of bool args with (sometimes descriptive) names, where the arg names are hard-coded per linter release, and the bear needs updating for each new linter release

  2. select/unselect args with opaque codes, with codes added in each release but the bear doesnt need modification

  3. is beyond the scope of this BanditBear bug, but clearly it is the bigger maintenance/usability problem for us, and it is the true abstraction of our settings from the linters cmdline. Ideally we solve both at the same time, but that will definitely require a few enhancements to coalib. Lets defer that for the moment.

What is in the scope of this issue is select/unselect are two mutually exclusive lists, and at least that is a coalib core enhancement to Settings. But you have a decent working version which can get merged into BanditBear before being migrated to coalib.

Also in the scope of this issue is the settings names should be rather consistent, and have prefixes for each bear, so that the user can have

bears = FooBear, BarBear
foo_select = a, b, c
bar_ignore = e, f, g

We need to determine the best naming convention, based on existing bears if possible. Then after this, we can update the other bears to comply with the new naming convention.