Debug info seems to be lost when deployed to Lambda

[maxcountryman]

Hi there,

I've been back and forth with Sentry trying to figure out why debug information seems to be completely lost when in the deployed Lambda environment--very strangely, the same project compiled directly in the underlying Docker container built for release happily passes along its debug info. The trouble only starts when deployed.

Some interesting things about the project:

1. We organize our functions via workspaces, so we've defined the profile.release overload in the root Cargo.toml.
2. We only use Sentry in our library project crate and then use this crate in the other functions; this shouldn't be an issue as evidenced by the fact this works in virtually any environment aside from Lambda itself.
3. Frustratingly things work as expected locally and during CircleCI builds and even from the build container; just not Lambda itself.
4. As per Sentry's request we've tried uploading debug symbols manually, but this hasn't really helped.
5. Sentry complains that libc.6.so is not included in the aforementioned uploaded debug files, but despite this things work in e.g. CircleCI, just again not in Lambda.

I'm wondering if this could be partly to do with the fact that the Lambda executable is packaged and renamed to "bootstrap"? If there's any light you can shed on that piece of things and how it might affect something like this, I'd really appreciate it! :)

serverless version: ^1.51.0

rust-plugin version: ^0.3.6

[softprops]

This is likely related to building a release binary. I think in those cases debug info gets stripped by default. serverless-rust is a Serverless framework front end for lambda-rust a docker builder. There's an open issue there about making the release vs debug build configurable https://github.com/softprops/lambda-rust/issues/21 which may be helpful here. This defaults to release for efficiency of speed and size. I a believer that debug should also be an option though.

You mentioned something worked in a build container. Was that the lambda-rust container or a different container?

[maxcountryman]

This is likely related to building a release binary. I think in those cases debug info gets stripped by default.

I think this is right. However my understanding was that you could workaround this by specifying something like this in your Cargo.toml:

[profile.release]
debug = true

You mentioned something worked in a build container. Was that the lambda-rust container or a different container?

This is in the lambda-rust container. Interestingly enough that was with a release build. Again I think that by specifying the debug = true configuration for the release profile the binary should be including debug info.

[softprops]

Post a link here for my own reference and testing https://blog.sentry.io/2018/10/22/getting-started-with-rust-error-tracking

[jan-auer]

Debug information is stripped explicitly in the build script in the docker container: https://github.com/softprops/lambda-rust/blob/d3e55ce8a436c79c32921740ce1eea3475b999e6/build.sh#L33

While this makes sense to reduce the size of the final binary, it might make sense to put debug information in a companion file instead so that it can be used for debugging later, like so:

objcopy --only-keep-debug "$file" "$file.debug"
objcopy --strip-debug --strip-unneeded "$file"
objcopy --add-gnu-debuglink="$file.debug" "$file"

[jan-auer]

Also, it is a little strange that debug information is stripped even for debug builds. There, it should probably even be retained in the original file so that the builtin libbacktrace can symbolicate stack traces, etc.

[maxcountryman]

Also, it is a little strange that debug information is stripped even for debug builds. There, it should probably even be retained in the original file so that the builtin libbacktrace can symbolicate stack traces, etc.

Thanks so much for spotting this, @jan-auer--I ran out of bandwidth on my end but this is pretty clearly the source of my issue.

[softprops]

I'll try to publish an update soon but I've updated the underlying container image to add support for this. I just haven't wired it into the serverless plugin

[maxcountryman]

Fantastic! Looking forward to the update.

[softprops]

Hi folks. It's been a while but I wanted to post and update on the latest release which now has experimental support for local builds which takes a slightly more hands off approach when it comes to stripping anything

https://github.com/softprops/serverless-rust#-experimental-local-builds