Build pip requirements.txt to specific runtime (cross macos-linux/architecture compilation)

[caioquirino]

Is your request related to a problem? Please describe.

I am having to use docker to build a lambda layer from a pip requirements.txt file, but since I am doing it locally from a macbook, the binaries are added for darwin architecture.

It would be great if I can force it to use linux/amd64 (manylinux) or something that is compatible with AWS Lambda when building it locally

Describe the solution you'd like.

It would be great if together with pip_requirements in the source_path list, I could use something like platform="linux/amd64" or even following the pypa manylinux standards as platform="manylinux2014_x86_64".

Or as another option, to use the compatible_runtimes and architectures or runtime fields already existing in the module.

Describe alternatives you've considered.

If I can fix the issue https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/458 that would be another alternative

Additional context

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[caioquirino]

Up

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[caioquirino]

Up...

[jcopps]

Up..

[atyndall]

Ran into this, would be very useful

[jcopps]

Two workaround options:

@atyndall , @caioquirino

1. One may try this: https://stackoverflow.com/a/74729370/3195195
2. Use docker with pip requirements.txt. In this case, the docker can be set with the right platform architecture and the build takes care of itself.

[atyndall]

Thanks @jcopps. It would be great if the module could set the --platform manylinux2014_x86_64 itself.

[jcopps]

Agreed. But the docker support was added for a reason like these I presume.

[caioquirino]

@jcopps it exists, but it is not working for my use case (cross compiling mac-os m1 to linux x64): https://github.com/terraform-aws-modules/terraform-aws-lambda/issues/458

[jcopps]

@caioquirino Did you try specifying the image of the docker from docker registry to be of linux x64.

[caioquirino]

@jcopps you can find a lot of information about what is happening in the #458 issue :) TLDR: Yes

[userhas404d]

I ran into this recently as a result of this issue. I was able to cheese it locally by manually cramming in the pip options required here but needed both --platform and --only-binary, as a result my recommendation for this issue (and solution) would be to allow the input of an arbitrary number of additional pip options

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[atyndall]

Not stale

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[atyndall]

Not stale

[github-actions[bot]]

This issue has been automatically marked as stale because it has been open 30 days with no activity. Remove stale label or comment or this issue will be closed in 10 days

[github-actions[bot]]

This issue was automatically closed because of stale in 10 days

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.