search

mojodna / marblecutter-virtual

Virtual catalogs for marblecutter
Other
48 stars 30 forks source link

Size of code package exceeds Lambda's 250MB limit #13

Closed mhiley closed 5 years ago

mhiley commented 5 years ago

Hi @mojodna ,

I tried to deploy a small change to our instance of marblecutter-virtual, and am now receiving an error from Lambda indicating that the uncompressed code package exceeds the limit of 250MB. The deploy has worked fine in the past, so I'm not sure exactly what changed to cause this.

It looks like the 3 largest dependencies are: rasterio (127MB) numpy (73MB) botocore (38MB) .. and the remainder of the packages cause the whole thing to just exceed the limit, about 258MB.

Before I dive into this much more - would you be able to try running a build/deploy and confirm whether you are also experiencing this issue?

moamenet commented 5 years ago

I do have the exact same issue. Help would be really appreciated.

j08lue commented 5 years ago

For a similarly scoped project with the same dependencies (https://github.com/DHI-GRAS/terracotta), we are using Zappa https://github.com/Miserlou/Zappa for deployment and you have a good deal of handles on the file size there - e.g. excluding packages that are already on AWS or activating slim_handler. We are actually using Zappa a lot and it is really easy to set up.

mhiley commented 5 years ago

This might provide at least a near-term solution to the problem:

https://twitter.com/sgillies/status/1166038763732004865

@mojodna Can you provide a bit of guidance on upgrading to to rasterio 1.0.26? Is it as simple as setting the version in requirements.txt? I should have time to take a shot at this in the near future.

mojodna commented 5 years ago

@mhiley yup, that should be everything. That's exactly what I was planning on trying. There are other ways to slim things down, but the wins are tiny.

mojodna commented 5 years ago

Oh, botocore may be able to be omitted--I seem to remember that it's part of the runtime. Thanks for taking a look!

mhiley commented 5 years ago

@mojodna @moamenet I believe this issue is resolved now. Currently the rasterio version requirement is:

rasterio[s3] ~= 1.0

When I opened this issue in July, that resulted in Rasterio 1.0.24, which resulted in a final uncompressed code package size of 258MB.

Now when I do the sam build step, it'll use the newer rasterio (1.0.28) and the final code package size is back down to 182MB (rasterio dependency is only 51MB now), which deploys to Lambda successfully.

In other words, things are working again, and I don't believe any PR is needed.

mojodna commented 5 years ago

Awesome!