java_maven workflow not up-to-date fails on utf-8 encoding issues of Maven output

[zoladkow]

Description:

This issue is not about the original encoding handling problem, but the fact, that current installer does not include the package which seems to fix that issue.

sam build fails with Error: JavaMavenWorkflow:MavenBuild - 'utf-8' codec can't decode byte 0xdf in position 76813: invalid continuation byte. --debug shows the error originates from \aws_lambda_builders\workflows\java_maven\maven.py", line 31 which is this: LOG.debug("Maven logs: %s", stdout.decode("utf8").strip())

This is weird as there are preexisting issues (ie. #1164 ) related to this, which got closed already. And fun fact is, that the aws_lambda_builders included with CLI installer does not seem to be the same as current content of the repository for that package. at least maven.py now uses an util for decoding maven output, instead direcly using decode() with hardcoded utf-8.

Steps to reproduce:

1. install SAM CLI as instructed in https://docs.aws.amazon.com/serverless-application-model/latest/developerguide/install-sam-cli.html#install-sam-cli-instructions
2. run sam.cmd build Function --template project\.aws-sam\temp-template.yaml --build-dir project\.aws-sam\build

Observed result:

SAM build fails after Maven finishes successfully:

2023-10-12 13:02:46,559 | JavaMavenWorkflow:CopySource succeeded
2023-10-12 13:02:46,560 |  Running JavaMavenWorkflow:MavenBuild
2023-10-12 13:04:01,892 | JavaMavenWorkflow:MavenBuild raised unhandled exception
Traceback (most recent call last):
  File "C:\Soft\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflow.py", line 371, in run
    action.execute()
  File "C:\Soft\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\java_maven\actions.py", line 39, in execute
    self.subprocess_maven.build(self.scratch_dir)
  File "C:\Soft\Amazon\AWSSAMCLI\runtime\lib\site-packages\aws_lambda_builders\workflows\java_maven\maven.py", line 31, in build
    LOG.debug("Maven logs: %s", stdout.decode("utf8").strip())
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xdf in position 76813: invalid continuation byte

Build Failed

If, however, I update the installed maven.py to say LOG.debug("Maven logs: %s", stdout.decode(encoding="utf8", errors="replace").strip()) the whole build passes without issue, also cannot identify where the replacement was done.

[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  01:13 min
[INFO] Finished at: 2023-10-12T13:27:13+02:00
[INFO] ------------------------------------------------------------------------
2023-10-12 13:27:15,963 | JavaMavenWorkflow:MavenBuild succeeded
2023-10-12 13:27:15,965 |  Running JavaMavenWorkflow:MavenCopyDependency
2023-10-12 13:27:15,965 | Running copy_dependency with scope: runtime
2023-10-12 13:27:21,176 | JavaMavenWorkflow:MavenCopyDependency succeeded

Expected result:

I'd expect Maven output to be copied as-is, and in general, console output encoding should have no impact on the build process...

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

1. OS: win10
2. sam --version: SAM CLI, version 1.98.0
3. AWS region: eu-west-1
4. sam --info

   {
   "version": "1.98.0",
   "system": {
   "python": "3.8.8",
   "os": "Windows-10-10.0.19041-SP0"
   },
   "additional_dependencies": {
   "docker_engine": "24.0.6",
   "aws_cdk": "Not available",
   "terraform": "Not available"
   },
   "available_beta_feature_env_vars": [
   "SAM_CLI_BETA_FEATURES",
   "SAM_CLI_BETA_BUILD_PERFORMANCE",
   "SAM_CLI_BETA_TERRAFORM_SUPPORT",
   "SAM_CLI_BETA_RUST_CARGO_LAMBDA"
   ]
   }

[zoladkow]

Oh, my bad! I just noticed, that maven.py was fixed JUST TWO DAYS ago 🤯 as part of https://github.com/aws/aws-lambda-builders/pull/554

I think I assumed it must have been fixed a long time ago since #1164 was from 2020, was already closed and contained some PRs. but those were only for .NET workflow - so not even directly related to that issue 🤦‍♂️

[lucashuy]

Hi thanks for opening this issue. The Lambda Builders PR you've linked does indeed fix the crashing, but you bring up a good point in your OP about using errors="replace". The linked fix doesn't utilize that argument, so its a good thing you opened this issue (even if it might be a duplicate/fixed already)!

I'll keep this issue open in the mean time to see if we can use that argument to better improve our logs.

[github-actions[bot]]

Patch is released in v1.99.0. Closing