uninitialized constant Rack::RACK_MULTITHREAD

[benbonnet]

Using rails 7.1.1, rack (3.0.8) and lamby (~> 5.2) Hard to get why those happen; I created a fresh app following the getting started and deploying via github actions. It does deploy without issues, but invariably getting the following error :

Executing 'config/environment.Lamby.cmd' in function directory '/app'
...
"errorMessage": "uninitialized constant Rack::RACK_MULTITHREAD"
...
2023-10-30T01:36:10.282+01:00 "/app/vendor/bundle/ruby/3.2.0/gems/lamby-5.2.0/lib/lamby/rack_http.rb:45:in `env_base'"
...

Went through the rack code trying to find Rack::RACK_MULTITHREAD; not to be found

Then further below :

Init error when loading handler
--
  | 2023-10-30T01:36:10.282+01:00 | {
  | 2023-10-30T01:36:10.282+01:00 | "errorMessage": "wrong number of arguments (given 4, expected 2..3)",

This one looks related

[hungnv-sr]

Same issue: https://github.com/rails-lambda/lamby-cookiecutter/issues/41

[hungnv-sr]

any updates?

[benbonnet]

downgrading / forcing down rack to 2.2.6.4 clears the issue; but it surely does not work with latest rack (= default installed for a new rails app seems like 3.0.x)

[arinhouck]

I experienced the same issue here.

[albertski]

I experienced this as well. I removed the constants in https://github.com/rails-lambda/lamby/pull/173 and it fixed the issue.

[jeremiahlukus]

Going to look at this tomorrow and get it merged after testing on lower rack versions to make sure it doesn't break anything. If it does i need to set a rack restraint and it will be a major version change with a breaking change.

[thenano]

Hi @jeremiahlukus thanks for looking into this and pushing a fix! I'm not sure if I'm doing anything wrong, but I updated to lamby 5.1.2, and I'm still getting errors when trying to run with Rack 3.1.4, getting the following in my logs:

2024-06-27T13:27:52.055+10:00 | {
-- | --
  | 2024-06-27T13:27:52.055+10:00 | "errorMessage": "uninitialized constant Rack::VERSION",
  | 2024-06-27T13:27:52.055+10:00 | "errorType": "Function<NameError>",
  | 2024-06-27T13:27:52.055+10:00 | "stackTrace": [
  | 2024-06-27T13:27:52.055+10:00 | "/usr/local/bundle/ruby/3.3.0/gems/lamby-5.2.1/lib/lamby/rack_http.rb:41:in `env_base'",
  | 2024-06-27T13:27:52.055+10:00 | "/usr/local/bundle/ruby/3.3.0/gems/lamby-5.2.1/lib/lamby/rack.rb:33:in `env'",
  | 2024-06-27T13:27:52.055+10:00 | "/usr/local/bundle/ruby/3.3.0/gems/lamby-5.2.1/lib/lamby/handler.rb:89:in `call_app'",
...

Let me know if I'm doing something wrong or if I can provide any additional information.

thanks again!

[jeremiahlukus]

Seems like more conta were removed

https://github.com/rails-lambda/lamby/blob/master/lib/lamby/rack_http.rb#L41

I only tested on older rack versions since my app uses an older rack version. I should have tested on older and newer to verify the PR actually fixes the issue. I should have some time tomorrow to spin up a new app and test. I’ll fix this in the next couple days.

More than happy to look over a PR if you want it in faster.

[thenano]

Thanks @jeremiahlukus I'm not in any rush, was just going through my open PRs for dependency upgrading. I'm happy to wait a couple of days. I you do need help after that let me know.

[jeremiahlukus]

Im able to reproduce

[jeremiahlukus]

lol so looking at the rack changelog

https://github.com/rack/rack/blob/main/CHANGELOG.md#added

• Introduce Rack::VERSION constant. (https://github.com/rack/rack/pull/2199, @ioquatix)

They added back the Rack::VERSION constant.

[jeremiahlukus]

for some reason getting status 0 when using the newest rack version no idea why.

[jeremiahlukus]

@thenano can you see if my branch fixes it for you?

gem 'lamby', git: 'https://github.com/jeremiahlukus/lamby'

[jeremiahlukus]

Ok finally fixed it that took longer than i'd like to admit. Ill send another PR

[jeremiahlukus]

https://github.com/rails-lambda/lamby/pull/180

[thenano]

Hey @jeremiahlukus thanks so much for looking into this! Sorry about the delay in getting back to you. I have tested your branch and although in the end I'm getting a 500 response, checking lamby logs it all seems to be running fine, so something else might have broken in my setup which I'll look into. Happy to do any more testing if you need.

[thenano]

it looks like API Gateway is not happy with the response sent from lambda:

{"httpMethod":"GET","integrationErrorMessage":"The response from the Lambda function doesn't match the format that API Gateway expects. Lambda body contains the wrong type for field "multiValueHeaders"","protocol":"HTTP/1.1","requestId":"aDkFrj1KywMEPUA=","requestTime":"28/Jun/2024:02:04:20 +0000","resourcePath":"-","responseLength":"35","routeKey":"$default","sourceIp":"220.245.90.202","status":"500"}

Not sure if related, or maybe something that rack has changed?

[jeremiahlukus]

Hm i think i need to ensure multiValueHeaders is always a hash. I just pushed do you mind trying the latest?

[thenano]

@jeremiahlukus I'm still getting the same error, I'm pretty sure it built with the latest from your branch, but if you wanted to add a tag or something like that I could force a version and check?

[jeremiahlukus]

I closed the PR still have a weird issue on my end as well. I’m not seeing the error you are seeing however. If you can debug a bit that would be helpful.

[thenano]

@jeremiahlukus interesting, happy to help with debugging. essentially I keep getting that same error on the GW side. Can you point me in the right direction of where I could potentially dump some information into logs to get more insight into it?

[metaskills]

Ok finally fixed it that took longer than i'd like to admit. Ill send another PR

I'm in this statement too 😅

[jeremiahlukus]

Whats weird is some requests a passing and some are returning status 0 and 302. Having a hard time figuring out if its because of my app or lamby ha

[jeremiahlukus]

working on it again today making progress

[jeremiahlukus]

@thenano hey I got it working on my app now

https://github.com/jeremiahlukus/lamby

Can you try again. I realized I was hitting my cloud front app so some things were getting cached when i hit api gateway directly i was able to reproduce consistently. After a good bit of fiddling I finally got it to work with rack v3 (pulling from master).

[jeremiahlukus]

Or @metaskills if you have an app running on rack v2 and want to test v3 that would be great.

[jeremiahlukus]

Deployed in 3 apps so far no issues, I also tested on a rack v2 app and it seems to be working.

[thenano]

Hey @jeremiahlukus thanks again for all your work on this. I have tested with rack 3.1.4 and 3.1.6. "Normal" rails controller calls are returning correctly (i.e. I can get html and data) but I'm getting errors when serving any assets, I get 500s for all js and css files served via the app:

2024-07-10T12:31:30.089+10:00 | Error raised from handler method
-- | --
  | 2024-07-10T12:31:30.089+10:00 | {
  | 2024-07-10T12:31:30.089+10:00 | "errorMessage": "source sequence is illegal/malformed utf-8",
  | 2024-07-10T12:31:30.089+10:00 | "errorType": "Function<JSON::GeneratorError>",
  | 2024-07-10T12:31:30.089+10:00 | "stackTrace": [
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/lib/ruby/3.3.0/json/common.rb:305:in `generate'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/lib/ruby/3.3.0/json/common.rb:305:in `generate'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/ruby/3.3.0/gems/activesupport-7.1.3.4/lib/active_support/json/encoding.rb:92:in `stringify'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/ruby/3.3.0/gems/activesupport-7.1.3.4/lib/active_support/json/encoding.rb:41:in `encode'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/ruby/3.3.0/gems/activesupport-7.1.3.4/lib/active_support/json/encoding.rb:23:in `encode'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/ruby/3.3.0/gems/activesupport-7.1.3.4/lib/active_support/core_ext/object/json.rb:42:in `to_json'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/aws_lambda_marshaller.rb:30:in `marshall_response'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/lambda_handler.rb:31:in `call_handler'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:81:in `run_user_code'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:59:in `start_runtime_loop'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric.rb:42:in `run'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/bootstrap.rb:35:in `bootstrap_handler'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/lib/aws_lambda_ric/bootstrap.rb:8:in `start'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/gems/aws_lambda_ric-2.0.0/bin/aws_lambda_ric:10:in `<top (required)>'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/bin/aws_lambda_ric:25:in `load'",
  | 2024-07-10T12:31:30.089+10:00 | "/usr/local/bundle/bin/aws_lambda_ric:25:in `<main>'"
  | 2024-07-10T12:31:30.089+10:00 | ]
  | 2024-07-10T12:31:30.089+10:00 | }

Let me know if you need any more info or if I need to change anything.

Thanks!

[jeremiahlukus]

Just to make sure does your app work with the published version using rack v2?

and does your app differ from the configuration https://github.com/rails-lambda/lamby-cookiecutter/blob/master/%7B%7Bcookiecutter.project_name%7D%7D/config/environments/production.rb

And are you pre compiling in your deploy script and coping those files in your dockerfile?

I also have config.public_file_server.enabled = true in my config.

my app uses active admin and that css and is is loading so either your app is configured wrong or you are loading assets a different way and I’m not accounting for that in my fork of lamby

[jeremiahlukus]

In the attached pic i have css to make the panels red then some js to turn them a light blue on hover.

added some custom js and css to active admin. Still no issue

Assets all are 200's as well

[thenano]

Hey @jeremiahlukus yep I can confirm that the application works with rack 2.2.9 and lamby 5.2.1. The production.rb env config file is the same in my app as the cookie cutter, only difference is the lograge config at the end which I doubt would make a difference? I'm precompiling assets during the container build, with this line in my dockerfile: RUN RAILS_ENV=production bundle exec rake assets:precompile

This is very odd because my application is also an activeadmin application (3.2.2 with rails 7.1.3.4). From the stack trace, it seems like for some reason aws_lambda_ric is trying to transform the response to JSON? The browser is sending the correct accepts header for each file so it's pretty strange. Let me know if I can test any other change or send any more info.

Here's my full Dockerfile for reference, if it might help:

# syntax = docker/dockerfile:1

# Make sure RUBY_VERSION matches the Ruby version in .ruby-version and Gemfile
ARG RUBY_VERSION=3.3.2
FROM registry.docker.com/library/ruby:$RUBY_VERSION-slim as base

# Set production environment
ENV LAMBDA_TASK_ROOT=/var/task \
    LAMBDA_RUNTIME_DIR=/var/runtime \
    RAILS_ENV="production" \
    BUNDLE_DEPLOYMENT="1" \
    BUNDLE_PATH="/usr/local/bundle" \
    BUNDLE_WITHOUT="development test"

# Rails app lives here
WORKDIR ${LAMBDA_TASK_ROOT}

# Throw-away build stage to reduce size of final image
FROM base as build

# Install packages needed to build gems
RUN apt-get update -qq && \
    apt-get -y upgrade && \
    apt-get install --no-install-recommends -y build-essential nodejs libpq-dev git

RUN gem install bundler -v '=2.5.12'
RUN gem install aws_lambda_ric

# Install application gems
COPY Gemfile Gemfile.lock ./
RUN bundle install --jobs 4 --retry 3 && \
    rm -rf ~/.bundle/ "${BUNDLE_PATH}"/ruby/*/cache "${BUNDLE_PATH}"/ruby/*/bundler/gems/*/.git && \
    bundle exec bootsnap precompile --gemfile

# Copy application code
COPY . .

# Precompile assets
ARG SECRET_KEY_BASE
RUN RAILS_ENV=production bundle exec rake assets:precompile

# Precompile bootsnap code for faster boot times
RUN bundle exec bootsnap precompile app/ lib/

# Final stage for app image
FROM base

# Install packages needed for deployment
RUN apt-get update -qq && \
    apt-get install --no-install-recommends -y nodejs libpq-dev && \
    rm -rf /var/lib/apt/lists /var/cache/apt/archives

# Copy built artifacts: gems, application
COPY --from=ghcr.io/rails-lambda/crypteia-extension-debian:1 /opt /opt
COPY --from=build /usr/local/bundle /usr/local/bundle
COPY --from=build ${LAMBDA_TASK_ROOT} ${LAMBDA_TASK_ROOT}

# Run and own only the runtime files as a non-root user for security
RUN useradd rails --create-home --shell /bin/bash && \
    chown -R rails:rails db tmp
USER rails:rails

# Entrypoint prepares the database.
ENTRYPOINT [ "/usr/local/bundle/bin/aws_lambda_ric" ]
ENV LD_PRELOAD=/opt/lib/libcrypteia.so

CMD ["config/environment.Lamby.cmd"]

[jeremiahlukus]

Do you have

  config.public_file_server.headers = {
    'Cache-Control' => "public, max-age=#{30.days.seconds.to_i}",
    'X-Lamby-Base64' => '1'
  }

In your config?

All your gems are update date as well (ie json)

It def sounds like something in your app is off, but i could be wrong. Its sending something not UTF-8 as a response. I am now forcing UTF encoding and add some logging can you bundle update lamby from my fork and try again? Maybe something will be in the logs to help.

If that doesnt work can you add some middleware in your app and see if that response is off?

Something i added in my app to debug things was

class EnsureValidStatus
  def initialize(app)
    @app = app
  end

  def call(env)
    # Log the incoming request details
    request = Rack::Request.new(env)
    puts "EnsureValidStatus: Incoming request - Method: #{request.request_method}, Path: #{request.path}, Params: #{request.params}"

    status, headers, response = @app.call(env)

    # Log the status before and after modification
    puts "EnsureValidStatus: Original status: #{status}"

    # Ensure the status is an integer >= 100
    status = 500 unless status.is_a?(Integer) && status >= 100

    puts "EnsureValidStatus: Modified status: #{status}"
    puts. "::: RESPONSE FROM RAILS ::::"
    puts response 
    puts ":::::::::::::::::::::
    [status, headers, response]
  end
end

Rails.application.config.middleware.insert_before Rack::Cors, EnsureValidStatus

[thenano]

@jeremiahlukus I added the extra public server config but it didn't change anything. I also made sure all gems were up to date. I updated to your latest branch and the problem actually got worse, and I think I know what the underlying issue is: compression.

With your latest, even the index file was failing in the browser due to net::ERR_CONTENT_DECODING_FAILED, I think your UTF forcing has now broken my compressed files. I have this line in my application.rb file:

config.middleware.insert_before Rack::ETag, Rack::Deflater

This compresses all content in rails itself, although I don't think it runs for public files. When I removed that line, the app started sending the HTML correctly again. Now with the latest, instead of getting the previous 500 error for the assets, I'm getting the net::ERR_CONTENT_DECODING_FAILED which I think means my assets have also been compressed and are being served as gzip (not sure where this happens, maybe during assets:precompile?) This assumption seems to be further corroborated by the extra logs you added and the EnsureValidStatus middleware logs:

  | 2024-07-12T11:53:20.164+10:00 | START RequestId: 0ba2115c-2965-4eb1-8889-f76fe5591a0e Version: $LATEST -- | --
  | 2024-07-12T11:53:20.165+10:00 | EnsureValidStatus: Incoming request - Method: GET, Path: /assets/active_admin-871448167ef6ec861e8f2907a99fb0aa02eaa4d42e372cac0d899a2a07e8ef03.css, Params: {}
  | 2024-07-12T11:53:20.165+10:00 | EnsureValidStatus: Original status: 200
  | 2024-07-12T11:53:20.165+10:00 | EnsureValidStatus: Modified status: 200
  | 2024-07-12T11:53:20.165+10:00 | ::: RESPONSE FROM RAILS ::::
  | 2024-07-12T11:53:20.166+10:00 | #<Rack::Files::Iterator:0x0000ffff7c10e858>
  | 2024-07-12T11:53:20.166+10:00 | :::::::::::::::::::::
  | 2024-07-12T11:53:20.204+10:00 | Encoded part: "\u001F\b\u0000f\u0002\u0003=isF߯Zr4\b\u0010<kS+۲\u001D\u001FI9˥1$\u0006\t\u0000-)\|\u0002\u0002\bʊy\"9zfznrD٪\b\u0015Y\u0014IlP\u001A\u0014\a4\|\u0006דdBNf\a)(H;AWe\u0012)ΠN&=o}5hQbVhU>߅\u00150P\u0003\u0018J'&[WP\u0006a\u0002Ml=9\u001Fd\u0015+sI\\,sJ\u0012['A\u0010\u001D\\ۙA~ɶxr2ͦ\u0004E\u0010g\u0013CE\u001F3&n?Fiي\nxjӧu\u0000'\u001C\u0015a\u0018,ϩ>62)\u0016d\|`I\aȄFk3EMm&,Y\u0015hc\u001C\u0005hс\u0017I.\u001E\nȮ\u001C'\nf:d(&09\fpb\\-\u0000\u0019jzH\nA\u0000M\u0016P\nɓ%\"-2HQ(N\u007FGb\u0012`GVhULMMA\u001CCNfą)\b\u000F\bCU6-\u0530^R'\r8?#\|A\u0005kM\u0010'\|BʕaIPBqMg?-\u05F9o['\u0001X>\u0010\|<\u000Fb}\|`\eDZ\u0016Xu\u0015ltWSTQm\u00022w\u0018M(Eh\u0012\u001F@ë\u0012YM]bmW҂Wœ;Az},m`t\u0002QJFm\fpSCO1@3VI\v\bݤ=\frX!s\u0018xE0@1\nY<\u001A\u0003g1\u001AwFa0\fy({h\u0014~^ONex:\eFA!r#`k\u0013к\u0014Q44YbW\aa?\v8Pp\u000E[6Ǽf\u0002I{\u000Epgl\u0012sXO\u0006[0yj\u0002\t0\u0010fIx{\n&\u0013dL0%]\u0002\u0005\\oߊ5=L\u0018c[\u0010+,m\u000E\u000F\u001Dy&\u0012A-0\u0005\u008B/\u0015@d\u000E\u0006_܏hI\u007F͂e^O`ìlD\u0019ӂcIQA\u0005:WXT+Q^\aW9?^V,ؠ\u001CX\u0013xϜt8\u007F*\\S\u0004HgFA*\u007FZ\u0006W\u0015\nZ>Z'6\u0019p6Udhq\u0018n+MJJBx۰NBuhA\u0011[zdk*w.\e\u0014%\u0004\u000E\bn)l:Y\eDx\u0002ٓ\u0012R\u0002\vlyCCt1(\u001D0i]\a\v\u0014O\fw獆~<\u0014Q?Y\u0006s\v \u0005`52_%H\u0004\tMT\u001Fx\u0011!]TMQ\b4ќ>5X;I-ת\r\v-w\u00102,&$Npah@RRL\|Weit\"\u0006fr\u0015:,zC\r \|>tێ\u0011\t\u0018PPL\u{14AF4}V8KP\u001Ayi[ޚ;brG\\g\"X\u0015\u0016)6:\u0006fY*Q)]NVȅߘ\r3q\u007Fpr\u0013\u0003ֲœ&\u0010{]Ei\u001E\u000Egԇ\u0011_\vLfz/Ӎ\u001D\b\u007FzL\u0010^ՍYzc\u0019\n*,M1\nkR\u001DYv^{'sM<\u001Fha꧉\u0005\u07FCI'\u0006\u0014t?P?bn_mK6Hg\u07BC)\fcs}3\u001C`0\u0018\|,;Q\u001A-\u0011cKV:/;Xژi\u0003@\u000FQh\n̪S\v\u007F4\u001EBiuﴶzsymCt\u001F\u001D/ȖVR\u001E?\u0014+W[jeNVb\u000ET}Dx8AW$sCF\fV\u00029\u0011\u0015Gubf\u0004Ƨ\u001A#A\\3P&\"\u001C\f\nR^2 zt}Q:?5]\u001C6\u001CM.BO6\bދ/UJ\u001DC'\u0000[\b:\r\rjnhcg-Ҍm$DF,҆\u001A,1\u0010n\u0000\u0006oM:U\\6VҴ=\u0002\u0013\u000FOO}POzTfSg/zЬ29W]\u0001\u000Eiʎ$)N3S\rӫ\u0004UBZh\u0012V\u0012nú\u00185q\a\n&>1PDC\u001Fō\\Ͽ(N\u0002+[V\u001Em\u0010ZY*>fe\u000F\u000E?`^'W(\u0010uS[K\u0003a5\u0005wcx)tw\u0014ǜsG-}\r܂\al\u0002\u0001i`E\u0018A*5\u001CV^uf^/\u001A<wN\e3?\u0019\u001Â\aN\eܾ?p\u000Fqo \rЈn,>Mo\b\v1A5\u00053ŵaQ\u000E4܍0:ݞ~^<T\t氇a7ʖ\u000FW\b\u0004 \u007F(X;ɚ\u001D\v\u0012\u0018N\u0010{\u00030K\u0012_jȧEoW.h\u001Epw0DK 틍(<5iALUdޠhZfl*\u001FP\u0003l.\nxV$E$ѵ\u00167V\fQ\u000FdNY>bd\u000Fc;\u000FcGnϷv-<\u001D%\vc8KR<`\u0000.lvl\u0016:Nq89\u0003!\u0018~&9.v\en&\u0006LسME9\eO\u0018y \r7Y-\u0010l?՜!cib\u0017\u001E\bL\u0016{\u0006\u0018/\u001A\u0005\u00062'2\f`\r\u0012\u001AЄVeM&D$N\u0002ʤP_Y_CЗ@\u00180Ȫ:\exe~a2-\u0001_︹\eut\u0013XH??\u0001\"B\u0003꜏)AwH\f\bhVهY/8pS`\u0005J5\u0017\u001D\u0003`\at$\u0018]\u000E(D1,\u0004D\v\u0014\ax\u0004\u0001?\u0006jQXv\u0019\rYRYd=\u0001H\u001Ed\u0014Ջ4Ew\u0089\u000Fr\\3\a\u0014l6tܤ4P˘xELmMVՙ%mנ(\u001CڽD\u0000_}ŧU\u0012Hn؂\u0015[\u0093\u0002IG;)ax\u0012:=d\n*Ҫ{Q\u0001atF[t϶6IVܶ\u0017`µ\u0017\v^\fvN?\u0018C\u0015\v\b+T\u007F<O\u00008f=`\u0012z8#v/\\#g\n\rI)O!\\i̘\u0016}V\u0001N֢e\f^%1Z\u001A\u001C_J&\u0004\u0004V\u0006;\u0012\u0015آ[K;byDK\u001A6%\u0012mG\u0018o\tn\n$$\u0018asR\u0000TJ0hY=+a~\f]\f\u0010d\a :\u0014$rb\u0012©N\u0001\u0018;e7y\u001E%\fq\u0011FEA`ǫj=\u0000*\u0000\u0004C\u000FlA8@`\u0004%\v=\u0010$\u0019}XԆ\u007F1Di^{\r@0fec~Fq\u0002$g(J0{Sm~@S+.cE;%,mb}\u001FbqbV,P\u0010[F<x<;(%\b\u0013]U4\"뎆\u001AI`\u001F@QH\n\\2FZ\u0005\u0010\u0004\u0015\".ul\e\a+?\u000F`\f\u00188\u0006-eƎ\\flBZ)G3/rUP*\u0005\n٫*A~t:J\a\u0013 `\e\u000E\u000EQFN*~t\u0000f~d~xJU$V\fN=\u0014#FipOt+sJGOX8>7\u0016pGNҴ&\u000FbQ\u001DD'`VbVs\u001FQz\nb*1X;/`\a\u0002\noĠk\u007FD\u0012\u001F/\u00197+wb]]D'Gz²Ui\u0017!o(\u0003TTiJ\u0013:\u000F_\u0002і\u0005G䗂\u0014\u0005\b[זW\u0002:\u0013kV\u001C@=8?L0Q7n@\"npS\u0003\bU\u0017A~B9~6\u074B\"\u0010pO49oGgg/:;{5zO?\u000F\u001Fz\u0010]=~?ggo^'ׯ~z#u_y9??'ڭ\u0001Y\"X۱SL[fV5NuP<<O_\|7o:/\u001ECeᣖjC\u001DBuK\\\|W\"_\u0018d\u007F\f\\Ǵ݁\u001C\u0019&Ӡϭi\u0015MB$_\u001D[a⛬ue3q/!֩J2\u0005gO=\\\u000E;\ts4p1~<(\u0003Ѽ+h-K\u0015CLWTi.'8ꫴd\t/\u001F?^\u001F\u007Fq_\u007FzӛyNް+?`D\u0017x<{z$?7dJ\en\u001Du\u0012=;f_\|k}6\u0019:{lxxD\u0013&\u000FBJ'\a&B$BuI5Qc6S5\u0011o#\u0004D7DD'.9\aJB>!2\u0018/E&**\e3\u0006Qr\u0019\v*٘:?$eey&1\ax\u0019&/N\u001A۽/vrCM^RGO\uA63B}U\r\u0010\u0011QdjB!]\u0002II\u0015!/7WǔN,Hq,̦A\u001D{S\u0011v?Gݴ\u0002\|g\\Eaƞ\\af3G$l\f\u0002*\v\u000E\u0000\u007Fߊ\u000E#\u0013+8\u0017j\f\u0084\u001C\u0002\u0016+o\u0001{\ftU[4Rd\r肞LX\u0013QTJ\u0010X,\u000E\t2!J/VZ\f\u0000eE2X]_\b>بHtE\f\u0015-rV,lzF˖6_MT-pLcMu)\u0011[0\u0015ĨRQmk뺼RN\u001A*N\u001A\b\u001FdࡥЂΘ}J96 >;\u0006,\rUjo\u000F&:\" qǁbh\u0019$28@!\u0010\u000E(Dyl+H\u000F#2.·`6_E26}I7buF\u0013\u001A)!g\u007Fc\bpP\u0018}eLWeG\a\u001A\u0004V56j\e!nJRЦ\u001ESj\u000E\u001F(`\\u!(\u0004P4YHqj\nt\u0019N<0DU4}%\|>@:\u0018fGBl\u001Et\u0011g^CN)\u0010\u007FyO%[x)j^\u0005yE\u001A^\u001C\u001D_\t\vVd<\u000E\\\u0013G\u000F B\u001D}7S\b=2K\"Yi^kS<-\tqb.w41\u0015SuDqz}\u0015}\t\u0003J菄#o<\nE4\u0015\n\u001AO{x+\u001D&6Ӧ\u0005=f`\u0010'$r\b\u0380K\u0003C3'\\\u0016Q$bu׀\u0015N\u0019.P\tjT=7r}l߮[3a\b@DZ0\eBl\u0000b18@!a\u0010$v\u0003k\u05C8\u0015EM)sK=\e\u05F9\u0003a\u000FfjV\u0016MZ4=o\u001F57i\u0014$+rf\u0018JHu\"g\u001D*.\n_J\u0012١\u0095vxp%K!ɒq\u0016_O\u0018\u0014\u00012\u0019ZQ \u001E\u001F5\b!aK\\\v\u0010&z,udϙ\u000F<RqHy8[\eZq3\u0001VҵFS\u0005\u0011\u000Fzk\u0006S[Q0v\n؉\u0006\u0013jeH\\hht}\u0001;e,6\u001F)9`\e6'בms\v6jVI%Qde\n:,T\n\u0018бkUckН갼Sj\u0003\rCj\u0014\u0018\"\u0015l;m;\u001C\u001ATv<bihakj\u001F©7\u001C\u000F\\\n\u0017RNGm/ٍWR@S\u0001\t\u0002_^괾YC\u0001Y^ZU&ղX7\u0002\u0005F\u0000+\u0000d\u0013DQ4ՅYNiWeyY\u0011f\u0011O\u007F\u0003\azJwĮ\u0005\u000FX.c1z\u001E\u0013\rG1C\u001E\u0018y\\G9qcX\u00033\u000F\u0013\u0095\u00140g6M\u001D6p\e\u001E\u001E\u000F8hk'\n\u000Ec1ࠌL\u0012\u000Eud\u001C\fcA^K}\u000F\u007F}Z\u0001<\u0019龛'1\nEN፧v]&\nrKJvDS\u0019\v_d}9H-M%*m\u0014S\u0012\u0010u\u0019\u0016S\u0001\u0002GY8zA٢BL')hQ\nk\b\u0013W,ڴХs\t\u001C6\u001AܠtB\u001F\u0001iBZӶ!D4QQysV覝6\vID\e]\ej\n\r\u000F\u001AkŃ\u001E\u001F1;\u0003scD\u007F64dz$8<05\u001EtƹXpw2q2!j-\u001Dj\u0013Yǂr'2ՈLQ\\Bb/x.c\nNp*S>th&)\t\u000F#^<E,\tP\u00189ԉ\u0012\u0015\\\u0000JV36\u0002 7\fhk^\u0005bm3\u0014e\f\u001E!\u001D\f7+HZ\u0003bHn7jHs\u001754+\t[u#\u001C\a\u0017Sv@Y\u0004I@M\u001C\u0017\n\u00044\u0017\u00067\ez1V_\|\u0014Gi\u001Alj\bD\b9Q\u0019\u0015\u0019G=Y6i!\u008E\u00102+${=Sy>\u0017@q;}r\bɡ\u0015l6EqCk\u0010Ga0+4©@c\"\u0002rV\vH*/=\u0000#rg:$\u0018mQb_X7jbZ]iU9qtxN\u0012rk\u0019\eõ\u001322]pՏ\u0010>\u0006\u0002\u0003Gc_-}#R_\u0018]n=\u0005\\\rY*\\߹\u0013\u000FP+)\u0002eh\\!\u001C.o1U\u0010\u0001yEbfBv4̤R*D\b!eԭb\u000FET\u000E\u0017\u0013\f\u0014\|\a\u0001\u001Az>%vǗ\u0006\u001746nMduж\u0001V\u001F.\u0005g\u001Drnk\u00137m?\"']c9:\u0017\u0013䢦\ez>NxNNن\u000EۨӲD!U\u0011\u0012T\em+%pV=bN1\u001E\u000F٤\u007Fp\u001E\u0013}'nJ(H*\u000E6uЦP<1\u00123\u000E4rpjE\f\u0012\v\u0003o0\u001AiyݥW\u0012BLUQ\u00170,ҙсƴ7L\u0018q<\u0013.g\u00186$\u0017\u001E;\u0015D'$%0RHp-&Fro\u0002\u0017\u001Dz\u0014\u0005I\u0011_o\f\u0004(H[V\u0003\u007F6Kxnw}{\tE\b͜oސ\u0006I\u0004{\u0016)p\u0005nz!d\u0012f\"E2VA7k-]H\u007FfHTˀdN\u0013\u0014/:i٭6?&9a\u001FD\u0006x*pꗂ1ems$?>KyS\u0006\|[q\u0004\u000Ex/$Y?\u007Fӟ\u007FC_<zq^?=](M%9J\u0017ߝ?}^'ޫE뷽gWO~<WO'ntuY~٣\u0017ϟ~jz=\u007FC~v\u001EG_\u007F}{8N\u0014__\u001F\u0006ɾo\u001F>\r/y~\e]a:\u000E\u001Fׯ\u0018g?/?>\u007F\u0017~=\f˗o$\u007F\u2EF7\|dSqļ\fw\u001AD!KNGҢT\u001D\u000E\u000FԎQSu\"\u001C>Լs\f!#\u00150!s,\fN@Mp¹NHX@j#CM\u0017}@+\"aȁ7\nu1T'\u0002-bv\u0018/\r6,r\u001Fv\u0010Qz\f\u0001%IfTc!A֬\u000E!jZ6zԎ0:07,f\u0018ă\u0018*\r+\rb\u0002i+\a\u0014z\t.\u0006iUB\u001D\b\|Mba\u05FEc\rы\bq\b\u001A@l*jpQ@\u00055l\u0018TNPV\u0017-EUa,(D\u0019\u0004h:\u0017L#\a1'uJ˲h7\u0014]\u0010켝c\|o;˶\b\u0005pJ\u0017Bj\u0018\u001C\u0001*{\\!`\"+ݧub,/gQ=\tl/\u0014sD\|/AR4R\u0011x+N0i^IjRb\u001DC\u0012ET\u0011,UNtC\u0013\\PsNIm\u0000&G;ɗk]]Ӎ\u009EbV+ǥm}\u0018\e\u0002\u001ACv8\u0006\u0383\u0383\a㱴@D[ \vUkȻ\u0001uu6\vhL1L\u001CZ#d\u0012!y%>6\b1q/]\u001EO\u0006EXw\u001D\u00100\u0016sΟAa*l\t`Ta\u0016lb\u0004\t梾JmՓ\u0017L9>\u0011i!ܠ 6ex\f>T銫Iص\u0015/jS\u001F1S}.\u0004\u0018M.QR=JjoǓJ\vA\u0017nq \b0w5&ȋ%Zmħ[Kyߣ=\t\v$,Lk\u0013\u0000\u{B42AE}\|\u0000\u0002!)\u001D-\u001AT\r@ j$6sx=>\u001E\u0017\\@s\u0011\\*Je^\u00015$\u0017ؖX@vPYd\u0001JgiOj\u0006\\7\b \u001FH\u001A`.n5\u00158\u0004m F'\tbg\u0013Or\u0004!no\u0000=\b6Q\rB:\u001A\u0001~efد`JLi#'2/sLќL1\u0002\|]\u007F\ao\u000F@)zטp!Oe\u001DN8debZ\u0018ƖqHj`p95YŻ%?jU\u0001lA0P\u0014g\u0005Z\u0378b\u00109˪I,aN\u001E6w>0Z\u0001k\a\u0018/\u000FZ\u0001/g\u0014}x\a#/\bT\u0001\u0015\u0010V\u0006QnIa98\npNo\u0015\txڧ*U5\bm\bB9ׁYo\u001A\rW3wA\u0006%¹\e/}b\u0003q\f!`16B\u0003/n\u001A0d2Ϻ;;)\u001DǷ*\rDF걄5\f\u0018\u001E"
  | 2024-07-12T11:53:20.205+10:00 | Encoded part: "\u0098;JA\a0ͥHl\u001ESJM\u0013Q8OjbZ_lyu`*[7I\u0006.l\u0012~\u0018W}Ʋ*\u0017O Hwb_Dfe/\e\rpZo\u000E3\u0018S\u000E\u0000\u0000ڻ\u0010\u0000gY\u0012_l-\u0006v)C}EyŻw&ΓEig\u0015k\|*mSw\u001A0iص\u000EF;\u0001N=\r\t&\u001DZ\v\u00180c\u001DDh\u001F\u001Du\tK\u0013RCl\"Yy].ѯ/\u0013Qԫk\nˀ lOvcϻ\u001Am.L0\u0010Y+Twѝ`\\3a(v,5T\u007Fz\u0010\u0004wBF(aTg,C,ɦ\u0011trbϗ\vu\u0005lFq6\u0002Z]'\"\u0010QG\u007F!2ժ~\|5k\u0019╁/ذ\u001AM6ԅ5G\u0013[\u000Efȭ-_3.+\u000F\"\u0012Lz,\u001As\u0003n\u001Eek\u0010=\u0006-H\\A\u0018\u001DJ3V\u0013Qm*\u0010gWع\u0000\u007FČlo{b\u001Cc:\u001A+\u0002нFg\u0003g<y$\u0015Yeb\u0006]Z\|QdYM0$̺Bd<\u00161'2;JrJrk?*q41TG\u0019?F~Ckd \|\"\u0018bNO\u0016Z\u000Fﵖǣz'\u0013\u001D~\"%\rdx>w\u0093\u0001\|*<\b\u007FL\u0004e?f:*Q\b?F:k\u0019xpiMY\u0015N%Y\u00019ɩd\u0005ZZiEЛ/;\\:\u0010f_`JH&`Et\u001DnF֣.Ѳ3I{\u0002\u0000\u0019E##\u0000Y\u0013Z;\u00104Ri\u0012\u0006{\f!\u0015's\u000E9&#6\u001F;=u@\u0003mW9()$yE.Asv-XGnz}S\u0016\u0000\rƮ\u0005E\u0005R\u0001\u007F!M64\u000F'a\bEŎqJvTo3$\u001A\u0010#fl\u000F\u0002-2\u0017$(\u0016w,P\u0011.V(}\u00105Í\u0017J\u0012$?\u0014)GΝNEoV\u0011\bdNY^'H\u0018L}۵N#I1\u001ASjN>1KI\t_ !(\u000F\u0002\nUewB\u000F&2=:\u0005\u0004?zJ\u0016~\"7(\aɩ=ADv\t\u0003\u0019$+\a\u0010\u0016,\u001Cm(j\u001Dڢt(]>upδF[6jS敡ba&\e\u001D\u0018\u0004'+K`\u000EۺV͋7߱\u0014iWU\u001F[]L}l\u001DT~+{D}̭\u001CH\u0000\u000F%',+\u0004+j5u\u0010*>ޟЀ\u000FUZ$z\u0004\u0004C\u007FK'}INq\f\u0000_\u0014J['j\u0014=1ʹ^lɦ0N\u0000]_%\an]\"\u001CтT\u000EZpH&)2э;uP\u0016CQf8 '$\u001F]߿e\em\u0004\u001Di\e\u0003&\u008D--\fH\u0005'd\u00189Fl:\u0019.r7dw6IȮ_\u0002\f)gs\eS)X\u0016DBm\u0001;Ќ0QhK\u001Dn.\u0002-sx.<2ۗ{ٽt&^\t\erSw\fV^\u00057GV#`2\u001F.\u001COd\u0011\eQl\u0014[\u0014t'd)!\u007Fn\u007FΒr\u0017%j\u0001DVvEdaT{ޗl\u001F\u0018\u0015yp\u0012I`7\t\u000F?E\u000F\rv7sè\vd_W?X\b¯9ȳ\u0010;?\\\u0001N\\dZdz\u0014\u0018M>):.AN\u0010{S^\u0017n\a\u0000Kq^:sXn \u000E!Z2f\u001C$P8<o#\u0002\vh\e\r\u007F2!Ŷʯ7\u001CZ7\u0006alwr6\u0017!G)\u0012\u0015C}\u0014C;/6jS6,F0`>vD4nl\u0004ˑoZx:\u0017\b\eGE\u000E7(\u007FhmAH`5\u00147NdzĚ,;)\u007F\eI\u0000ARog\u0002ڦG60@\u001A.X\b$!cib\u0017;\r?[\u0016\u00058 R=;{\u0016V?i{v\|/`/\u0006#4UY{Igљe6\a{\u0005\fK顾\u0004볾\u0000B//\u0001AV\u0001120ߖ\u0000ɯw6ㅣ&\u001E~~ؗf\u0012ĮcJ\u001D?\u0003h)UlҌ\"rUep\u0011R\u0013)<M61滺\u001DP\"tg\u0013\u0010-P\u001E\u0011\u0010\axOv]\rբ-NDa\u0016YO\u0000\u0012+\u0012Ea\\\u0012A\u0001O\u0019ujLيu\fV%6PK\b46dUY\u0002vC^9{\u0010ƨ\b4WJ]\u0002).\u0012\u000E,\u0013-$}7߾\u0012'ӻ/p\f^`\u0015\a\u0006h/\\{ً`gp\b\u001D\u0001\u0016UDxO\u0001S]zNkbA76А\u0003Mc2\u0019w}\u0016e-\v\u007Fw<V#\u0006\aj\u0004 ܒJkE$\u0012\u0015\u0005055H\u001Dp4\\l{֎`w\u0016\u0012}YلW;3H\|R\u0000TJ/hY=+an\f]\v\bd\a\u0011wg)c80ZB6Չ5\u0000\u0018TMHn<Jt\u0019^\"6\u0011Wz\u0010 ED\u0018\u0002A8@\\\u0004\u0005\u001Eە\b4\rficӼ̥`&. %\n1%N\u0001q7N*%ҩtMf}g\u0016^,\fwcgx<;(X\u0013-8,F)%9Pa\u0019\r5@>\u0014\u0004e\vZwm\n\u0011i-SVpa=؈/g\u0004/o\f\u001C\rZ\u0002ˌ\u001D\fqVJ\u0016L\u0000\\=\nPcl-=HN8r\n'$^ӨxMe}\u0005(2`'2_8yI=E'H\u007FB#\eEo:\u001E\u00196Օ~l\vbɾΞ\u0012\u0012IJޤ\u0014߬+5\u0016ѓW,K,赒\u0005)&.w^5\r.bD6n8/+\\qJDGyg\u000F@t\u000F\u0016BxnM.9ҰQ6 G\u0019e\tE$<\"\u001ClX~Ev\t\u0003\u001D\u0004g9Ծ\u0015\u0018\u0010`עmrS\u0003\u001C\e_O[9\u0017I]7\v.\tw\u0010_,0&C1pm\u0012<Nj\u0010Htʻ)Myx^1;o&ea\u007FDy9\|?\u0000\u001E<\|\f<\t>dI,u)2=[?Rϧm<gS`\nE;'=O>>l5݇@tx\n\u0002<rڅn@&d\u000F!Pe\u0016Q&yMi]$HhRs[\u0017KPP-Y1.z\u0014*\v\u0018d c Q\u0012r\u001D7\u001F:\u0011)v&F\ena\u0018Q<0\u0015\u0002d%X\u00063NfI\n-\u0017j\u0006\u000F\t7U//#\a'{HD]\u00177~Km\u001D/,w/\u0019;\"FLiO\bDRﮯZr\u007F/[xFK\u0019\u0001N^-\u001Cj=\tը=Zx{39ȟH\ri\u0002A\\R\u001F:M\u0018\u0002\u0001\u0000"
  | 2024-07-12T11:53:20.207+10:00 | END RequestId: 0ba2115c-2965-4eb1-8889-f76fe5591a0e

Hope this helps find a way forward? I think we just need to deal with compressed responses?

As an additional note, I had to add the middleware as this: config.middleware.insert_before Rack::Sendfile, EnsureValidStatus so that it runs for assets, I actually couldn't build it with Rack 3.1.7 because it would fail saying Rack::Cors doesn't exist.

[jeremiahlukus]

AH I found the issue! Rack::Deflater set https://github.com/rack/rack/blob/main/lib/rack/deflater.rb#L66 the header "content-encoding" but im looking for "Content-Encoding"

https://github.com/jeremiahlukus/lamby/commit/839cacdc6f58de567d924d0c33e9d06e6ca7ebc3

This should fix your issue. Thanks for the info man I would have missed this one. Can you bundle update lamby from my fork and try again?

[thenano]

@jeremiahlukus spot on, everything works as expected! Thanks so much for all your time fixing this. I just did a bunch of browsing, download, etc, on my app and no errors.

[jeremiahlukus]

Awesome! Thanks for debugging with me ha. Ill send another PR to really support rack 3after a bit more testing.

[jeremiahlukus]

https://github.com/rails-lambda/lamby/pull/181

Will merge this soon going to test a bit more and try to add a bit more to the test suite.

[jeremiahlukus]

In case someone searches for ERR_CONTENT_DECODING_FAILED be sure you have caching disabled in CloudFront. Im not sure why but updating to rack v3 gives this error on the css.

[jeremiahlukus]

@thenano hopefully will get the PR merged tomorrow or the next day and then cut a release sometime this week

[thenano]

@jeremiahlukus thank you so much. No rush from our side either, take your time.

[jeremiahlukus]

merged waiting release

[metaskills]

merged waiting release

I do that right?

[metaskills]

I just did a commit to main add v6 to the Gemfile.lock. I also just tagged and release v6 to ruby gems. 🥳 Great work team!

[jeremiahlukus]

Yes that part is not automated, if you want i could create a workflow for that but you would have to enter your keys in the secrets.

Awesome i forgot to bundle install after I upped the version ha.

Indeed thank you @thenano for helping debug some issues

[thenano]

Thank you very much @jeremiahlukus and @metaskills! we've updated to v6 on our side 💪