Add support for escaped macros

NassimBtk commented 2 months ago

Description:

We need to extend our macro replacement functionality to support various escape formats. The goal is to allow the engine to automatically replace macros with their corresponding inputs in different escaped formats.

Current Functionality: Currently, our code supports the replacement of macros with their corresponding inputs. For example, the macro %{USERNAME} is replaced with the actual username from the user's configuration.

New Functionality: We want to introduce a new set of macros that apply different types of escaping to the input values. The new macros should follow the pattern %{SOURCE_ESC_FORMAT}.

Supported Escape Formats:

Regex: %{SOURCE_ESC_REGEX}
URL: %{SOURCE_ESC_URL}
JSON: %{SOURCE_ESC_JSON}
XML: %{SOURCE_ESC_XML}
Windows Shell: %{SOURCE_ESC_WINDOWS} and %{SOURCE_ESC_CMD}
Linux Shell: %{SOURCE_ESC_LINUX} and %{SOURCE_ESC_BASH}
SQL: %{SOURCE_ESC_SQL}

Supported Input Sources (`SOURCE`)

USERNAME
HOSTNAME
PASSWORD
AUTHENTICATIONTOKEN
BASIC_AUTH_BASE64
SHA256_AUTH
PASSWORD_BASE64

Example Usage:

ℹ️ We must ensure that all possible scenarios are accounted for in each escape format. These are just a few examples:

JSON Escape:
%{USERNAME_ESC_JSON}: Na"ssim→ Na\"ssim
Explanation: In JSON, the double quote character " must be escaped with a backslash, so Na"ssim becomes Na\"ssim.
URL Encoding:
%{USERNAME_ESC_URL}: Na ssim → Na%20ssim
Explanation: In URLs, spaces are encoded as %20, so Na ssim becomes Na%20ssim.
Regex Escape:
%{USERNAME_ESC_REGEX}: Na.ssim → Na\.ssim
Explanation: In regular expressions, the dot . is a special character and must be escaped with a backslash, so Na.ssim becomes Na\.ssim.
XML Escape:
%{USERNAME_ESC_XML}: Na<ssim & "N" → Na<ssim & "N"
Explanation: In XML, <, &, and " are special characters that must be escaped as <, &, and ", respectively.
Windows CMD Escape:
%{USERNAME_ESC_WINDOWS}: Na&ssim → Na^&ssim
Explanation: In Windows CMD, the ampersand & is a special character that must be escaped with a caret ^, so Na&ssim becomes Na^&ssim.
Linux Bash Escape:
%{USERNAME_ESC_BASH}: Na'ssim → Na\'ssim
Explanation: In Linux Bash, single quotes inside a quoted string must be escaped with a backslash, so Na'ssim becomes Na\'ssim.
SQL Escape:
%{USERNAME_ESC_SQL}: Na'ssim → Na''ssim
Explanation: In SQL, single quotes are escaped by doubling them, so Na'ssim becomes Na''ssim.

Additional Notes:

Ensure that the implementation does not break existing functionality.
Consider the security implications of these escapes, particularly in Shell and SQL escapes.
Remove %{USERNAME_JSON} and %{PASSWORD_JSON} and replace them with %{USERNAME_ESC_JSON} and %{PASSWORD_ESC_JSON}

Specification

The possible escape characters for all the source formats are as follow: (@bertysentry Can you please review the escape sequences below ?)

URL

Character	Escape Sequence	Context
(Space)	`%20`	Represents a space
`!`	`%21`	Represents the exclamation mark
`"`	`%22`	Represents a double quote
`#`	`%23`	Represents the hash or pound symbol
`$`	`%24`	Represents the dollar sign
`%`	`%25`	Represents the percent sign
`&`	`%26`	Represents the ampersand
`'`	`%27`	Represents the single quote or apostrophe
`(`	`%28`	Represents the opening parenthesis
`)`	`%29`	Represents the closing parenthesis
`*`	`%2A`	Represents the asterisk
`+`	`%2B`	Represents the plus sign
`,`	`%2C`	Represents the comma
`/`	`%2F`	Represents the forward slash
`:`	`%3A`	Represents the colon
`;`	`%3B`	Represents the semicolon
`=`	`%3D`	Represents the equals sign
`?`	`%3F`	Represents the question mark
`@`	`%40`	Represents the at symbol (@)
`[`	`%5B`	Represents the opening square bracket
`]`	`%5D`	Represents the closing square bracket
`^`	`%5E`	Represents the caret symbol
` \| `%60`	Represents the backtick or grave accent
`{`	`%7B`	Represents the opening curly brace
`}`	`%7D`	Represents the closing curly brace
`\\|`	`%7C`	Represents the pipe
`~`	`%7E`	Represents the tilde

Regex

Character	Escape Sequence	Context
`.`	`\.`	Represents any single character (wildcard)
`^`	`\^`	Represents the start of a line
`$`	`\$`	Represents the end of a line
`*`	`\*`	Represents zero or more occurrences
`+`	`\+`	Represents one or more occurrences
`?`	`\?`	Represents zero or one occurrence (optional)
`{`	`\{`	Represents the start of a quantifier
`}`	`\}`	Represents the end of a quantifier
`[`	`\[`	Represents the start of a character class
`]`	`\]`	Represents the end of a character class
`(`	`\(`	Represents the start of a capture group
`)`	`\)`	Represents the end of a capture group
`\\|`	`\\\|`	Represents alternation (logical OR)
`\`	`\\`	Represents a literal backslash
`/`	`\/`	Represents a literal forward slash

XML

Character	Escape Sequence	Context
`<`	`<`	Represents the less than symbol in XML tags
`>`	`>`	Represents the greater than symbol in XML tags
`&`	`&`	Represents the ampersand (used for entities)
`"`	`"`	Represents double quotes in attribute values
`'`	`'`	Represents single quotes in attribute values

Windows CMD

Character	Escape Sequence	Context
`&`	`^&`	Used to escape the ampersand, which is used for command chaining (e.g., `&&` for "AND") in CMD
`\\|`	`^\\|`	Used to escape the pipe, which is used for piping the output of one command to another
`<`	`^<`	Used to escape the less than symbol, which is used for input redirection in CMD
`>`	`^>`	Used to escape the greater than symbol, which is used for output redirection in CMD
`^`	`^^`	Used to escape the caret symbol itself, which is the escape character in CMD
`%`	`^%`	Used to escape the percent symbol, which is used for variable expansion in CMD
`(`	`^(`	Used to escape the opening parenthesis, which is used to group commands
`)`	`^)`	Used to escape the closing parenthesis, which is used to close grouped commands
`"`	`^"`	Used to escape the double quote, which is used for enclosing strings or paths in CMD

Linux Bash

Character	Escape Sequence	Context
`'`	`\'`	Used to escape the single quote inside single-quoted strings
`"`	`\"`	Used to escape the double quote inside double-quoted strings
`\`	`\\`	Used to escape the backslash itself
`$`	`\$`	Used to escape the dollar sign, which is used for variable expansion
`!`	`\!`	Used to escape the exclamation mark (for history expansion)
`*`	`\*`	Used to escape the asterisk, which is used as a wildcard
`?`	`\?`	Used to escape the question mark, which is used as a wildcard
`[`	`\[`	Used to escape the opening square bracket, which is part of character classes
`]`	`\]`	Used to escape the closing square bracket, which is part of character classes
`(`	`\(`	Used to escape the opening parenthesis, which is used for subshells or grouping
`)`	`\)`	Used to escape the closing parenthesis, which is used for subshells or grouping
`{`	`\{`	Used to escape the opening curly brace, which is used in brace expansion
`}`	`\}`	Used to escape the closing curly brace, which is used in brace expansion
`\\|`	`\\\|`	Used to escape the pipe, which is used to pipe commands
`&`	`\&`	Used to escape the ampersand, which is used to run commands in the background
`<`	`\<`	Used to escape the less than symbol, which is used for input redirection
`>`	`\>`	Used to escape the greater than symbol, which is used for output redirection
`~`	`\~`	Used to escape the tilde, which is used for home directory expansion

SQL queries

Character	Escape Sequence	Context
`'`	`''`	Used to escape single quotes inside string literals in SQL
`"`	`\"` (if applicable)	Used to escape double quotes in some SQL databases (not all databases require this)
`\`	`\\` (if applicable)	Used to escape the backslash in certain SQL databases (depends on the SQL dialect)
`%`	`%`	Represents the percent symbol used in SQL for pattern matching (wildcards in `LIKE`)
`_`	`_`	Represents the underscore symbol used in SQL for pattern matching (wildcards in `LIKE`)
Newline (`\n`)	`\\n`	Used to represent a newline in string literals or text fields in SQL
Carriage Return (`\r`)	`\\r`	Used to represent a carriage return in string literals or text fields in SQL
Tab (`\t`)	`\\t`	Used to represent a tab in string literals or text fields in SQL

bertysentry commented 2 months ago

For URL and Regex escaping, we can use native Java methods.

For Windows CMD, I think there is a problem in the table (2nd row), where we should see | must be replaced with ^|.

This will require some manual testing, to make sure the behavior is as intended, especially for Windows and Linux commands, before implementing unit tests.

Also, the source can be an actual source reference (${source::monitors.enclosure.discovery.source(1)_ESC_JSON}).

MedMaalej commented 1 month ago

Tested simple macros (Non-regression test) :

Tested the connector HPEGen9IloREST (%{BASIC_AUTH_BASE64} macro). The HTTP request was executed without errors and the metrics collection worked as expected :white_check_mark:
Tested the connector CiscoUCSRest (%{PASSWORD} and %{USERNAME} macros). The HTTP request was executed without errors and the metrics collection worked as expected :white_check_mark:
Tested the connector EMCuemcli (%{HOSTNAME}, %{PASSWORD} and %{USERNAME} macros). The OsCommand command was executed without errors and the metrics collection worked as expected :white_check_mark: