Inconsistent behavior of sam build on windows and linux - Python

[sferhan]

Description:

Using sam build for my application is behaving differently on linux and windows systems. I am trying to use python package jsonschema in my project. Jsonschema has a dependency pyrsistent which does not have a wheel package. Sam tries to create a wheel package succeeds regardless of the OS but then on windows it doesn't detect the built wheel and shows pyrsistent as incompatible.

Steps to reproduce the issue:

1. Clone [https://github.com/sferhan/sam-sample]
2. Use Python 3.7 & Pip 19.0.3
3. Run Sam Build

Observed result: Failure on windows:

2019-03-27 18:42:25 Using SAM Template at F:\temp\myApp\template.yaml 2019-03-27 18:42:25 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-03-27 18:42:25 Changing event name from before-call.apigateway to before-call.api-gateway 2019-03-27 18:42:25 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-03-27 18:42:25 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-03-27 18:42:25 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-03-27 18:42:25 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-03-27 18:42:25 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-03-27 18:42:25 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-03-27 18:42:25 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-03-27 18:42:25 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-03-27 18:42:25 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-03-27 18:42:25 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-03-27 18:42:25 Changing event name from before-call.apigateway to before-call.api-gateway 2019-03-27 18:42:25 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-03-27 18:42:25 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-03-27 18:42:25 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-03-27 18:42:25 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-03-27 18:42:25 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-03-27 18:42:25 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-03-27 18:42:25 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-03-27 18:42:25 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-03-27 18:42:25 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-03-27 18:42:25 'build' command is called 2019-03-27 18:42:25 No Parameters detected in the template 2019-03-27 18:42:25 2 resources found in the template 2019-03-27 18:42:25 Found Serverless function with name='helloworldpython' and CodeUri='./src' 2019-03-27 18:42:25 Building resource 'helloworldpython' 2019-03-27 18:42:25 Loading workflow module 'aws_lambda_builders.workflows' 2019-03-27 18:42:25 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)' 2019-03-27 18:42:25 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)' 2019-03-27 18:42:25 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' 2019-03-27 18:42:25 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)' 2019-03-27 18:42:25 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)' 2019-03-27 18:42:25 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)' 2019-03-27 18:42:25 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)' 2019-03-27 18:42:25 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)' 2019-03-27 18:42:25 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)' 2019-03-27 18:42:25 Running workflow 'PythonPipBuilder' 2019-03-27 18:42:25 Running PythonPipBuilder:ResolveDependencies 2019-03-27 18:42:25 calling pip download -r F:\temp\myApp\src\requirements.txt --dest C:\Users\syedf\AppData\Local\Temp\tmpxkt1sbkv 2019-03-27 18:42:35 Full dependency closure: {attrs==19.1.0(wheel), six==1.12.0(wheel), setuptools==40.8.0(wheel), pyrsistent==0.14.11(sdist), jsonschema==3.0.1(wheel)} 2019-03-27 18:42:35 initial compatible: {six==1.12.0(wheel), attrs==19.1.0(wheel), setuptools==40.8.0(wheel), jsonschema==3.0.1(wheel)} 2019-03-27 18:42:35 initial incompatible: {pyrsistent==0.14.11(sdist)} 2019-03-27 18:42:35 Downloading missing wheels: {pyrsistent==0.14.11(sdist)} 2019-03-27 18:42:35 calling pip download --only-binary=:all: --no-deps --platform manylinux1_x86_64 --implementation cp --abi cp37m --dest C:\Users\syedf\AppData\Local\Temp\tmpxkt1sbkv pyrsistent==0.14.11 2019-03-27 18:42:36 compatible wheels after second download pass: {six==1.12.0(wheel), attrs==19.1.0(wheel), setuptools==40.8.0(wheel), jsonschema==3.0.1(wheel)} 2019-03-27 18:42:36 Build missing wheels from sdists (C compiling True): {pyrsistent==0.14.11(sdist)} 2019-03-27 18:42:36 calling pip wheel --no-deps --wheel-dir C:\Users\syedf\AppData\Local\Temp\tmpxkt1sbkv C:\Users\syedf\AppData\Local\Temp\tmpxkt1sbkv\pyrsistent-0.14.11.tar.gz 2019-03-27 18:42:38 compatible after building wheels (no C compiling): {six==1.12.0(wheel), attrs==19.1.0(wheel), setuptools==40.8.0(wheel), jsonschema==3.0.1(wheel)} 2019-03-27 18:42:38 Build missing wheels from sdists (C compiling False): {pyrsistent==0.14.11(sdist)} 2019-03-27 18:42:38 calling pip wheel --no-deps --wheel-dir C:\Users\syedf\AppData\Local\Temp\tmpxkt1sbkv C:\Users\syedf\AppData\Local\Temp\tmpxkt1sbkv\pyrsistent-0.14.11.tar.gz 2019-03-27 18:42:38 compatible after building wheels (C compiling): {six==1.12.0(wheel), attrs==19.1.0(wheel), setuptools==40.8.0(wheel), jsonschema==3.0.1(wheel)} 2019-03-27 18:42:38 Final compatible: {six==1.12.0(wheel), attrs==19.1.0(wheel), setuptools==40.8.0(wheel), jsonschema==3.0.1(wheel)} 2019-03-27 18:42:38 Final incompatible: {pyrsistent==0.14.11(wheel)} 2019-03-27 18:42:38 Final missing wheels: {pyrsistent==0.14.11(sdist)} 2019-03-27 18:42:38 PythonPipBuilder:ResolveDependencies failed Traceback (most recent call last): File "c:\users\syedf\appdata\local\programs\python\python37\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 39, in execute self.scratch_dir File "c:\users\syedf\appdata\local\programs\python\python37\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 143, in build_dependencies requirements_path, artifacts_dir_path, scratch_dir_path) File "c:\users\syedf\appdata\local\programs\python\python37\lib\site-packages\aws_lambda_builders\workflows\python_pip\packager.py", line 212, in build_site_packages raise MissingDependencyError(packages_without_wheels) aws_lambda_builders.workflows.python_pip.packager.MissingDependencyError: {pyrsistent==0.14.11(sdist)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last): File "c:\users\syedf\appdata\local\programs\python\python37\lib\site-packages\aws_lambda_builders\workflow.py", line 236, in run action.execute() File "c:\users\syedf\appdata\local\programs\python\python37\lib\site-packages\aws_lambda_builders\workflows\python_pip\actions.py", line 42, in execute raise ActionFailedError(str(ex)) aws_lambda_builders.actions.ActionFailedError: {pyrsistent==0.14.11(sdist)} Build Failed Error: PythonPipBuilder:ResolveDependencies - {pyrsistent==0.14.11(sdist)}

Expected result:

Build Succeeded

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

Causes problem on Windows and Build succeeds on Linux

Output of sam --version:

SAM CLI, version 0.14.1

[sriram-mv]

I woud suggest using sam build --use-container for such problematic dependencies. Are you able to install pyrsistent through pip on windows at all?

[sferhan]

Didn't know about the --use-container argument. Will try that out. Yes I can install pyrsistent on windows.

[wolfeidau]

Also encountered this same error on OSX, workaround sam build --use-container worked. Not sure what the issue is.

[sriram-mv]

Closing the issue. The built wheel needs to be compatible for a lambda like environment. so always best to build inside a container.