Analyze JS inside YAML for AWS Lambdas

[ilia-kebets-sonarsource]

• [ ] Plugin
  • [ ] new YAML sensor
  • [x] filter YAML input files
  • [ ] generate TSConfig (to include type-dependent rules)
  • [x] analyze YAML
    • [x] analyze file
    • [x] save issues
    • [x] handle errors
• [x] Bridge
  • [x] new YAML endpoint
  • [x] YAML parser
  • [x] test Node.js 12 version (requires refactor because different API in yaml v1.x, possibly not doable)
  • [x] test Node.js 14 version
  • [x] Fork yaml repository and transpile it into Node12 compatible
  • [x] parse YAML file
  • [x] YAML syntax parsing
    • [x] single document
    • [x] multiple document
  • [x] parsing error
  • [x] extract AWS lambdas
  • [x] AWS::Lambda::Function
  • [x] AWS::Serverless:Function
  • [x] parse AWS lambdas
  • [x] JS snippet parsing
  • [x] parsing error
  • [x] normalize AWS lambda ast
  • [x] tree
    • [x] node
    • [x] token
    • [x] comment
  • [x] normalization
    • [x] loc
    • [x] offset
    • [x] range
    • [x] helpers
    • [x] parsing error (patch location)
    • [x] explain our patching of location in the code
    • [x] refactor YAML parsing code in separate file
    • [x] don't analyze YAML file content outside of JS snippets (leads to irrelevant issues and duplicates in case of multiple snippets per file)
    • [x] write test for header issue with multiple snippets
    • [x] write test for trailing comma issue with multiple snippets
  • [x] format (https://yaml-multiline.info/) - we handle only single item in items array
    • [x] block folded ">"
    • [x] block literal "|"
    • [x] flow with plain
    • [ ] flow with single-quotes
    • [ ] flow with double-quotes
    • [x] fix 1-off offset
  • [x] checks
    • [x] primary location
    • [x] secondary location
    • [x] quickfixes
    • [x] regex rules
    • [x] parent node location reached by visit()
  • [x] analyze AWS lambdas
  • [x] ~disable quickfixes (if possible)~
  • [x] build source codes
  • [x] lint source codes
  • [x] aggregate issues
  • [x] aggregate parsing errors
• [x] ITs
  • [x] plugin
  • [x] IaC coexistence
    • [x] issues
    • [x] metrics
    • [x] meta metrics (cpd, symbol/syntax highlighting)
  • [x] issue finding
  • [x] ~analysis failure~ (dropped as done in unit test)
    • [x] ~YAML parsing error~
    • [x] ~JS parsing error~
  • [x] ruling
  • [x] find AWS lambdas projects
  • [x] check issues
• [x] Peach
  • [x] find AWS lambdas projects
  • [x] check Cirrus logs
  • [x] check Peach results
• [x] Commit with personal github on fork

Decisions

We have decided to drop metrics and highlighting from this sprint as it is not necessary and requires coordination with SQ and other bubbles. Quickfixes are dropped from the scope as they will not be used as long as sonar-iac is not supported by SonarLint.

In case of multiple JS snippets in a single YAML file, the API endpoint /analyze-yaml returns a parsingError property if at least one of them has an error, we have chosen this "fail fast" approach for ease of implementation and so that the user can easily fix this issue and proceed with a well working analysis.

[ilia-kebets-sonarsource]

The API of yaml v1.x is different from the v2.x, so it would require some refactor or impossible to make it work with Node 12.

[ilia-kebets-sonarsource]

Candidates for peach and ruling:

Lambda

• https://github.com/Sudheer8979/homestory4/blob/fba569898a248d4607b7ae682d1c448e3d5c92e6/public/data/lamdafunctions.yaml#L163

• https://github.com/stelligent/cfn_nag/blob/be9415a4a1ee4892a6869158c03551cb078bd0a7/spec/test_templates/yaml/lambda_function/lambda_with_cloudwatch_logs_permission.yaml#L11

• https://github.com/melscoop-test/check/blob/7d4cc42172d877092fb6e40bbea5d785743edd40/tests/cloudformation/checks/resource/aws/example_LambdaEnvironmentCredentials/LambdaEnvironmentCredentials.yaml#L15

• https://github.com/Sudheer8979/homestory3/blob/b89285d17ed20caf94dcb3843338dff5997934d1/public/data/lamdafunctions.yaml#L163

• https://github.com/Sudheer8979/homestory3/blob/b89285d17ed20caf94dcb3843338dff5997934d1/public/data/lamdafunctions.yaml#L163

• https://github.com/SnidermanIndustries/checkov-fork/blob/b76678fd30ec392345a3ed61fff33df4236ae8c5/tests/cloudformation/checks/resource/aws/example_LambdaEnvironmentCredentials/LambdaEnvironmentCredentials.yaml#L14

• https://github.com/lifadev/archive_ingraph/blob/d3558eb28fa77c45caf7a4859e724185dc48b5aa/tests/e2e/fixture/complete_module/output.yaml#L24

• https://github.com/SnidermanIndustries/checkov-fork/blob/b76678fd30ec392345a3ed61fff33df4236ae8c5/tests/cloudformation/checks/resource/aws/example_LambdaEnvironmentCredentials/LambdaEnvironmentCredentials.yaml#L14

• https://github.com/lifadev/archive_ingraph/blob/d3558eb28fa77c45caf7a4859e724185dc48b5aa/tests/e2e/fixture/complete_module/output.yaml#L24

• https://github.com/aws-samples/aws-serverless-vending-machine/blob/b2898f3ac097e91fedfd021c87968c4266800f15/scripts/deployServerlessVendingMachine.yaml#L260

• https://github.com/caoheyang/cognito-wildrydes-website-1/blob/2d67103fed8ec1ef7ce1d1c8c4a8ac83a07be9c6/ServerlessBackend.yaml

• https://github.com/caoheyang/tibcas-cognito-website-1/blob/ba4da3041684907f43455f40de33c7b76b1cca9f/ServerlessBackend.yaml#L71

• https://github.com/ucipass/awsmanager/blob/9ba51eb67719c774f51c03cb9fca6a30861839ec/src/assets/lambdaServerless.yaml#L95

• https://github.com/hoaphanFox/sceptre/blob/ad8908330e8469f8f41f3b7889d7880de6d1b8df/lambda/templates/serverless.yaml#L60

• https://github.com/openrideproject/www/blob/e63c3a9d58707d46cb2d22180a40ac12264411e8/Auth/2_ServerlessAPI/ServerlessBackend.yaml#L64

Serverless

• https://github.com/aws-samples/global-serverless-image-gallery/blob/5edf0e1583033b8c26fbb9ff0522c3183f4968de/blog/cf/serverlessImageGallery.yaml#L157
• https://github.com/aws-samples/amazon-ivs-auto-record-to-s3-web-demo/blob/60fd48203d865c1a22672647d63a3f0dded2ca76/serverless/r2s3-serverless.yaml#L77

[ilia-kebets-sonarsource]

As the repositories where we found JS in YAML code are:

• in another language
• potentially unstable (most have ~0 stars)

We will simply take the YAML file for the ruling. I propose to add them in its/sources/yaml/, adding the source URL as a comment on the top of the file.

[ilia-kebets-sonarsource]

We will ignore the following formats because the parser handles spaces at the beginning of lines in a weird way:

• flow with single quotes
• flow with double quotes

Example:

single-quote: '
  Several lines of text,
  containing ''single quotes''. Escapes (like \n) don''t do anything.
  '

[ilia-kebets-sonarsource]

Since the lib yaml v1.x does not have the following features:

• node's location and range
• line starts array
• visite(doc, callback) function

we estimate that refactoring this would be too much work, and we go towards the fork the lib and transpile it into Node12 path.

[ilia-kebets-sonarsource]

Elected for Peachees because from aws-samples org which should be stable:

Lambda (block-literal formats and unsupported ones):

• https://github.com/aws-samples/aws-serverless-vending-machine/blob/b2898f3ac097e91fedfd021c87968c4266800f15/scripts/deployServerlessVendingMachine.yaml#L260

Serverless (plain double-quote format):

• https://github.com/aws-samples/global-serverless-image-gallery/blob/5edf0e1583033b8c26fbb9ff0522c3183f4968de/blog/cf/serverlessImageGallery.yaml#L157