sam fails to build when build-dir or template-file exists on non-C drive

feldschlachtiv commented 5 years ago

Description

Sam fails when template or build-dir exists on non-C drive, and because of that pycharm cannot run lambda function locally.

Looking at SAM source code, it seems sam tries to use relative path on L161 in command.py. I guess this is to shorten the command to avoid hitting command length limit, however, it doesn't work when either ctx.build_dir or ctx.output_template_path is on non-C drive as there's no relative path exist, assuming sam is installed in C drive.

       msg = gen_success_msg(os.path.relpath(ctx.build_dir),
                             os.path.relpath(ctx.output_template_path),
                             os.path.abspath(ctx.build_dir) == os.path.abspath(DEFAULT_BUILD_DIR))

I replaced the two os.path.relpath calls above with os.path.abspath, and sam succeeded to build my function without error (and hence pycharm was able to run the function locally)

Steps to reproduce

Run the following command in windows 10 command prompt. (content of template.yaml is the "Hello World" one) sam build --template N:\projects\git\kuraberu\template.yaml --build-dir N:\projects\git\kuraberu.aws-sam\build HelloWorldFunction --debug

Observed result

2019-08-14 00:56:25 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-08-14 00:56:25 Changing event name from before-call.apigateway to before-call.api-gateway 2019-08-14 00:56:25 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-08-14 00:56:25 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-08-14 00:56:25 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-08-14 00:56:25 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-08-14 00:56:25 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-08-14 00:56:25 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-08-14 00:56:25 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-08-14 00:56:25 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-08-14 00:56:25 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-08-14 00:56:25 Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane 2019-08-14 00:56:25 Changing event name from before-call.apigateway to before-call.api-gateway 2019-08-14 00:56:25 Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict 2019-08-14 00:56:25 Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration 2019-08-14 00:56:25 Changing event name from before-parameter-build.route53 to before-parameter-build.route-53 2019-08-14 00:56:25 Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search 2019-08-14 00:56:25 Changing event name from docs..autoscaling.CreateLaunchConfiguration.complete-section to docs..auto-scaling.CreateLaunchConfiguration.complete-section 2019-08-14 00:56:25 Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask 2019-08-14 00:56:25 Changing event name from docs..logs.CreateExportTask.complete-section to docs..cloudwatch-logs.CreateExportTask.complete-section 2019-08-14 00:56:25 Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search 2019-08-14 00:56:25 Changing event name from docs..cloudsearchdomain.Search.complete-section to docs..cloudsearch-domain.Search.complete-section 2019-08-14 00:56:25 Telemetry endpoint configured to be https://aws-serverless-tools-telemetry.us-west-2.amazonaws.com/metrics 2019-08-14 00:56:25 'build' command is called 2019-08-14 00:56:25 No Parameters detected in the template 2019-08-14 00:56:25 2 resources found in the template 2019-08-14 00:56:25 Found Serverless function with name='HelloWorldFunction' and CodeUri='hello_world/' 2019-08-14 00:56:25 Building resource 'HelloWorldFunction' 2019-08-14 00:56:25 Loading workflow module 'aws_lambda_builders.workflows' 2019-08-14 00:56:25 Registering workflow 'PythonPipBuilder' with capability 'Capability(language='python', dependency_manager='pip', application_framework=None)' 2019-08-14 00:56:25 Registering workflow 'NodejsNpmBuilder' with capability 'Capability(language='nodejs', dependency_manager='npm', application_framework=None)' 2019-08-14 00:56:25 Registering workflow 'RubyBundlerBuilder' with capability 'Capability(language='ruby', dependency_manager='bundler', application_framework=None)' 2019-08-14 00:56:25 Registering workflow 'GoDepBuilder' with capability 'Capability(language='go', dependency_manager='dep', application_framework=None)' 2019-08-14 00:56:25 Registering workflow 'GoModulesBuilder' with capability 'Capability(language='go', dependency_manager='modules', application_framework=None)' 2019-08-14 00:56:25 Registering workflow 'JavaGradleWorkflow' with capability 'Capability(language='java', dependency_manager='gradle', application_framework=None)' 2019-08-14 00:56:25 Registering workflow 'JavaMavenWorkflow' with capability 'Capability(language='java', dependency_manager='maven', application_framework=None)' 2019-08-14 00:56:25 Registering workflow 'DotnetCliPackageBuilder' with capability 'Capability(language='dotnet', dependency_manager='cli-package', application_framework=None)' 2019-08-14 00:56:25 Found workflow 'PythonPipBuilder' to support capabilities 'Capability(language='python', dependency_manager='pip', application_framework=None)' 2019-08-14 00:56:26 Running workflow 'PythonPipBuilder' 2019-08-14 00:56:26 Running PythonPipBuilder:ResolveDependencies 2019-08-14 00:56:26 calling pip download -r N:\projects\git\kuraberu\hello_world\requirements.txt --dest C:\Users\yamshu\AppData\Local\Temp\tmp2o0n2xjb 2019-08-14 00:56:27 Full dependency closure: {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)} 2019-08-14 00:56:27 initial compatible: {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)} 2019-08-14 00:56:27 initial incompatible: set() 2019-08-14 00:56:27 Downloading missing wheels: set() 2019-08-14 00:56:27 compatible wheels after second download pass: {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)} 2019-08-14 00:56:27 Build missing wheels from sdists (C compiling True): set() 2019-08-14 00:56:27 compatible after building wheels (no C compiling): {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)} 2019-08-14 00:56:27 Build missing wheels from sdists (C compiling False): set() 2019-08-14 00:56:27 compatible after building wheels (C compiling): {idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel), requests==2.22.0(wheel), chardet==3.0.4(wheel)} 2019-08-14 00:56:27 Final compatible: {requests==2.22.0(wheel), chardet==3.0.4(wheel), idna==2.8(wheel), urllib3==1.25.3(wheel), certifi==2019.6.16(wheel)} 2019-08-14 00:56:27 Final incompatible: set() 2019-08-14 00:56:27 Final missing wheels: set() 2019-08-14 00:56:28 PythonPipBuilder:ResolveDependencies succeeded 2019-08-14 00:56:28 Running PythonPipBuilder:CopySource 2019-08-14 00:56:28 PythonPipBuilder:CopySource succeeded

Build Succeeded 2019-08-14 00:56:28 Sending Telemetry: {'metrics': [{'commandRun': {'awsProfileProvided': False, 'debugFlagProvided': True, 'region': '', 'commandName': 'sam build', 'duration': 2490, 'exitReason': 'ValueError', 'exitCode': 255, 'requestId': 'b5bac6f2-d2fc-4477-a1bd-1262c7838d88', 'installationId': 'a161c6e6-1e2c-4a3a-9c82-6034f85d4100', 'sessionId': '43fe2ffd-e622-414b-9a97-62b7c7582762', 'executionEnvironment': 'CLI', 'pyversion': '3.6.7', 'samcliVersion': '0.19.0'}}]} 2019-08-14 00:56:28 Starting new HTTPS connection (1): aws-serverless-tools-telemetry.us-west-2.amazonaws.com:443 2019-08-14 00:56:28 HTTPSConnectionPool(host='aws-serverless-tools-telemetry.us-west-2.amazonaws.com', port=443): Read timed out. (read timeout=0.1) Traceback (most recent call last): File "runpy.py", line 193, in _run_module_as_main File "runpy.py", line 85, in _run_code File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli__main__.py", line 12, in cli(prog_name="sam") File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 722, in call return self.main(args, kwargs) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 697, in main rv = self.invoke(ctx) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 1066, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 895, in invoke return ctx.invoke(self.callback, ctx.params) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 535, in invoke return callback(args, kwargs) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\decorators.py", line 64, in new_func return ctx.invoke(f, obj, *args[1:], *kwargs) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\click\core.py", line 535, in invoke return callback(args, kwargs) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 94, in wrapped raise exception # pylint: disable=raising-bad-type File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\lib\telemetry\metrics.py", line 65, in wrapped return_value = func(*args, **kwargs) File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 105, in cli skip_pull_image, parameter_overrides, mode) # pragma: no cover File "C:\Program Files\Amazon\AWSSAMCLI\runtime\lib\site-packages\samcli\commands\build\command.py", line 161, in do_cli msg = gen_success_msg(os.path.relpath(ctx.build_dir), File "ntpath.py", line 584, in relpath ValueError: path is on mount 'N:', start on mount 'C:'

Expected result

I expected sam can build functions located on non-C drive.

Additional environment details (Ex: Windows, Mac, Amazon Linux etc)

OS: Windows 10 Professional
sam --version: SAM CLI, version 0.19.0

Add --debug flag to command you are running

jfuss commented 5 years ago

@feldschlachtiv Can you describe how you have everything setup? I want to make sure I understand what is going on before going further.

It looks like you have your code on one drive (N) but are running command and PyCharm from a different drive (C)? I am not a windows person but why would you do that? Is this common?

We do relative paths here to make the output of the command easier to read and understand.

feldschlachtiv commented 5 years ago

@jfuss - thank you for your reply! Here is what I can tell as a Windows user at home, though I mostly use Linux at work.

I have my code in N drive - when you have multiple physical disks (SSD or HDD) in your PC, Windows assigns the drive letter(such as C, D, ..) per disk. You can confirm this by googling something like 'SSD "D drive"' or for example, https://qph.fs.quoracdn.net/main-qimg-9713ca1340c5e3f5cd92631f5d8ae0c0.webp. You can change the drive letter for disk, and 'N' is just my longtime favorite.
I am running sam command and pycharm from C drive (because aws-sam-cli and pycharm are installed in C drive) - this is also almost mandatory in Windows. People store OS and all applications in C drive (C:\Windows, and C:\Program Files\ are the convention). You need to conduct a series of non-trivial tasks to move them to another drive (such as https://www.backup-utility.com/backup-restore/move-program-files-to-another-drive-windows-10.html)

feldschlachtiv commented 5 years ago

@jfuss - let me know if you need any further info from me. I am happy to provide. Thank you!

jfuss commented 5 years ago

@feldschlachtiv Thank you for the explanation, really helped me understand your setup. So it looks like we need further support in our output. If I recall correctly, we did relative paths here to make the output easier to understand, @sanathkr might remember the exact reasoning. For this case, paths that are absolute makes sense. To get the best of both worlds, I think we should try to do relative paths and if that errors out, fall back to absolute paths. Thoughts?

feldschlachtiv commented 5 years ago

@jfuss - I think it's a good idea to try relative paths then absolute paths. I don't have bias on neither - as long as it works.

I made the change to my pull request. Can you have a look when you get a chance?

Thank you for the feedback by the way!

eth0izzle commented 5 years ago

Would be good to get this merged as I'm also experiencing the same issue. Workaround for now is to copy my code to the C:\ drive.

feldschlachtiv commented 5 years ago

@jfuss - did you, or anyone who can review, have a chance to look at my change in pull request #1337?

sanathkr commented 5 years ago

@feldschlachtiv I have commented on your PR. Should be a quick fix to get this merged.

feldschlachtiv commented 5 years ago

@sanathkr - thank you for looking into it. I made a couple of changes - can you have a look at the PR (#1337) when you get a chance?

sriram-mv commented 5 years ago

Closing, as #1337 is merged and released.

aws / aws-sam-cli