403 Forbidden response from flipper ui

[krzcho]

I am sorry in advance for maybe a noobish question but I am stuck After deploying flipper (with ui) to production and trying to flip feature I got 403 Forbidden from every action I am running two thins, http and ssl one with force ssl option on I see ui just fine but I can't trigger any action getting 403 Forbidden Going back to development environment (with single ssl thin) and actions work fine Any hints?

uncomfortable solution is to flip on development and copy flipper.pstore to production

Gems included by the bundle:

• actionmailer (4.1.14)
• actionpack (4.1.14)
• actionview (4.1.14)
• activemodel (4.1.14)
• activerecord (4.1.14)
• activerecord-sqlserver-adapter (4.1.1)
• activesupport (4.1.14)
• addressable (2.3.8)
• ar-octopus (0.8.5)
• arel (5.0.1.20140414130214)
• bcrypt (3.1.10)
• builder (3.2.2)
• bundler (1.10.3)
• coffee-rails (4.1.0)
• coffee-script (2.4.1)
• coffee-script-source (1.9.1.1)
• concurrent-ruby (1.0.0)
• daemons (1.2.3)
• devise (3.5.2)
• devise_ldap_authenticatable (0.8.5)
• erubis (2.7.0)
• eventmachine (1.0.8)
• exception_notification (4.1.1)
• execjs (2.6.0)
• faraday (0.9.2)
• flipper (0.7.4)
• flipper-ui (0.7.4)
• hashie (3.4.3)
• i18n (0.7.0)
• json (1.8.3)
• jwt (1.5.2)
• mail (2.6.3)
• mime-types (2.99)
• mini_portile (0.6.2)
• minitest (5.8.4)
• multi_json (1.11.2)
• multi_xml (0.5.5)
• multipart-post (2.0.0)
• net-ldap (0.11)
• oauth2 (1.0.0)
• omniauth (1.2.2)
• omniauth-google-oauth2 (0.2.9 b861c83)
• omniauth-oauth2 (1.3.1)
• orm_adapter (0.5.0)
• protected_attributes (1.1.3)
• rack (1.5.5)
• rack-protection (1.5.3)
• rack-test (0.6.3)
• rails (4.1.14)
• railties (4.1.14)
• rake (10.5.0)
• responders (1.1.2)
• sass (3.4.19)
• sass-rails (5.0.4)
• sprockets (3.5.2)
• sprockets-rails (2.3.3)
• thin (1.6.4)
• thor (0.19.1)
• thread_safe (0.3.5)
• tilt (2.0.1)
• tiny_tds (0.7.0)
• tzinfo (1.2.2)
• tzinfo-data (1.2015.7)
• uglifier (2.7.2)
• warden (1.2.3)

[krzcho]

warden was updated to latest 1.2.6 with no luck access constraint was updated and it works fine from localhost (remote flipping still fails but I guess I will need to live with it)

class CanAccessFlipper
  def self.matches?(request)
    current_user = request.env['warden'].user
    (current_user.present? && current_user.respond_to?(:is_admin?) && current_user.is_admin?) || request.remote_ip == '127.0.0.1'
  end
end

[jnunemaker]

Could you add some logging to get some more information? Check to see which of the conditionals is returning false maybe (ie: request ip or current user or is admin). Also, not much I can do to help without more information. I'd probably need you to paste in here anything related to flipper in your app.

[greyblake]

I ran staging locally, and I have this in logs:

W, [2016-04-12T17:29:59.801209 #16401]  WARN -- : attack prevented by Rack::Protection::AuthenticityToken
W, [2016-04-12T17:30:01.924352 #16401]  WARN -- : attack prevented by Rack::Protection::AuthenticityToken
W, [2016-04-12T17:30:04.102610 #16401]  WARN -- : attack prevented by Rack::Protection::AuthenticityToken

Looks like middleware related issue...

[greyblake]

I was expecting to find this middleware somewhere in rails.. But it was a wrong path. It turned out it comes from FlipperUI: https://github.com/jnunemaker/flipper/blob/master/lib/flipper/ui.rb#L30-L40

Now I need to investigate why it does not work for me..

[greyblake]

I went crazy with this error..I want to provide the information, that I found out her:

• It can happen in case if session does not work properly (because session saves csrf token, see implementation of https://github.com/sinatra/rack-protection/blob/master/lib/rack/protection/authenticity_token.rb)
• In my case it's can be reproduced on staging and production only... Even locally with RAILS_ENV=production cannot be reproduced (in my previous post, I did it occassionaly by deleting cookies, what actually resets sesion)
• It does not work in Google-Chrome, but works perfectly in Firefox

Hope, those fact I provided can be useful for somebody else, who'll get this error and try to track it down.

[jnunemaker]

Without more information there isn't anything I can do. If anyone could setup an example app where this happens or figure out what there problem is and post here I'm happy to add docs. The key really is that you have a session before flipper. If you are getting a 403, I'd put good money on rack protection. Closing, but please drop any more thoughts or help here and I'll keep following along.

[sajmoon]

I had a similar problem. Soved it by modifying nginx config.

See sidekiq discussion: http://stackoverflow.com/questions/37270614/forbidden-sign-out-using-sidekiq-devise-activeadmin-on-production-server-with

Shameless copypaste from that thread:

In Nginx server configuration you need:

proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;

[shimms]

Don't have much to add, other than we're experiencing the same thing, and it looks to be as you thought with an existing session.

Our setup:

We use basic auth on the flipper stuff and Devise/warden on the non-flipper stuff. These two "apps" are constrained by subdomain routes, but once you've logged into the devise "app" (subdomain A), the flipper UI (with basic auth) (subdomain B) starts throwing this forbidden error at us.

Opening flipper UI up in private browsing window avoids the issue (as you'd expect as it doesn't have the existing devise session from subdomain A).

The warden cookie's domain is set to work on all subdomains of our domain.

[jnunemaker]

@shimms how are you authenticating the flipper ui?

[shimms]

Custom middleware which performs basic_auth if the subdomain is matched, otherwise continues normal processing:

class AdminAuth < Rack::Auth::Basic
  def call(env)
    # perform auth if admin domain
    return super if request_subdomain(Rack::Request.new(env)) == ENV.fetch('ADMIN_SUBDOMAIN')

    @app.call(env)  # skip auth
  end

  private

  def request_subdomain(request)
    request.env['HTTP_HOST'].split('.').first
  end
end

Flipper mounted as:

constraints :subdomain => ENV.fetch('ADMIN_SUBDOMAIN') do
  ...
  mount Flipper::UI.app(Teamsquare::Application.features) => '/'
  ...
end

[panozzaj]

I ran into this issue today, and solved it by clearing my cookies for the site in question. Thanks to @greyblake for mentioning the CSRF issue, that helped.

[tobico]

I'm having this same issue in Safari in production environment only (no issue in staging or development), accessing with Firefox instead seems to be a workaround.

I think I had similar issues with sidekiq-web, and have the following code in my routes.rb as a fix:

Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]

Is there any way to set this same option for Flipper UI, or should this already be happening automatically?

[calebeoliveira]

I'm having exactly the same issue mentioned here with Google Chrome in production environment.

Also seeing this on my server logs:

WARN -- : attack prevented by Rack::Protection::AuthenticityToken

Tried to disable the related rack protection with:

mount Flipper::UI.app(Flipper, {:rack_protection => {}}) => '/flipper'

and also tried to use the same cookie currently used by my application with:

mount Flipper::UI.app(Flipper) { |builder|
  builder.use Rack::Session::Cookie, :secret => Rails.application.secrets[:secret_key_base],
              :key => Rails.application.config.session_options[:key]
} => '/flipper'

and tried to set headers on my nginx proxy:

proxy_set_header  X-Forwarded-Proto $scheme;
proxy_set_header  X-Forwarded-Ssl on;
proxy_set_header  X-Forwarded-Port $server_port;
proxy_set_header  X-Forwarded-Host $host;
proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header  Host $http_host;
proxy_set_header  X-Real-IP $remote_addr;

also cleared my browser data as suggested before but nothing seems to work 😞 I'm really frustrated with this issue.

I exported the Chrome HTTP request toggling one of the flags that I have as CURL and imported it on Postman app, it works like a charm. Also works on Firefox. Looks like Chrome is doing something wrong.

The only difference that I could spot between chrome and firefox requests was the Host header, it is present on firefox request but is not on chrome.

[calebeoliveira]

I am almost certain that is a nginx related issue. I could trace it down to HTTP_ORIGIN header being passed wrong (?) by nginx because I set a blank Rails project on my local machine with flipper and enabled production mode and could not reproduce the issue, also locally running my application in production mode can not reproduce the issue.

Currently I'm using cloudflare -> nginx proxy -> puma socket and I could make it work on Google Chrome doing a dark evil magic simple trick like this:

# routes.rb
constraints CanAccessFlipperUI do
    mount Flipper::UI.app(Flipper) => '/dolphin_evil_flags'
end

# flipper.rb
class CanAccessFlipperUI
  def self.matches?(request)
    # the trick (do not use in production, is unsafe)
    request.set_header('HTTP_ORIGIN', nil)

    current_user = request.env['warden'].user
    current_user.present? && current_user.id.to_s == '1'
  end
end

I know it's far from ideal and can potentially be a security breach but I'm still debugging my nginx stack (specially headers) to find out how to fix this issue once by all.

[joshpuetz]

We ran into this problem on https://dev.to (and I spent HOURS trying to fix it), but our stack runs on Heroku so there is no nginx proxy to configure. We worked around it by disabling CSRF protections in the Rack middleware running Flipper UI, and relying upon protections in our main Rails application (https://github.com/forem/forem/pull/9360/files).

mount Flipper::UI.app(Flipper,
    { rack_protection: 
      { except: %i[authenticity_token form_token json_csrf remote_token http_origin session_hijacking] } }
), at: "flipper"

[jnunemaker]

@joshpuetz ugh. So sorry. Super weird. I use flipper ui on multiple heroku apps with no issues. I'd love to know more about what caused your problem. I'd be happy to even meet up for a short zoom or something if there was anything I could do to help.

[yagudaev]

I just got the same error on a brand new Mac. Downloaded our code base and tried this, never seen it on my older Mac. Same OS, 2018 vs 2020 MBP, so not so different.

Refreshing the form and resubmitting again worked fine.

Started POST "/flipper/features" for ::1 at 2020-08-27 14:22:10 -0700
W, [2020-08-27T14:22:10.047338 #44411]  WARN -- : attack prevented by Rack::Protection::AuthenticityToken

[OrionWH]

If you are using haproxy in front of your nginx/rails, this option in haproxy configuration may help.

From haproxy documentation:

If you’d like to inform the backend server whether HTTPS was used, you can append an X-Forwarded-Proto request header by adding the http-request set-header directive:

http-request set-header X-Forwarded-Proto https if { ssl_fc }
http-request set-header X-Forwarded-Proto http if !{ ssl_fc }

At least this helped me to solve forbidden error. Adding such header could help with others than haproxy ssl terminators/offloaders also.

[jclusso]

Is this still any issue for anyone else? We use the same constraint for Sidekiq and have no issues.

[krzcho]

No longer an issue for me

[jclusso]

I had to use @joshpuetz way using rack_protection to get it to work in staging / production environments.

[matheussilvasantos]

I'm still having the same problem. I was able to create a small application that reproduces it: https://github.com/matheussilvasantos/flipper_ui_broken.

[matheussilvasantos]

Using Rack::Session::Cookie instead of Rack::Session::Pool for the session middleware fixed the error for me: https://github.com/matheussilvasantos/flipper_ui_broken/commit/b430e4d5ae2953f67e032f1c5962e618a179b460.

I still don't know why, but when you use Rack::Session::Pool, the csrf token isn't set in the session in https://github.com/sinatra/sinatra/blob/master/rack-protection/lib/rack/protection/authenticity_token.rb#L135.

[matheussilvasantos]

I've just found the problem for my scenario. https://github.com/jnunemaker/flipper/pull/606 fixes it for me.

[ilvez]

Letting you know that we have the same issue as above @joshpuetz. Disabling rack_protection concerning CSRF checks did cure this situation for us. Our application is behind Cloudflare and Nginx reverse proxy. Flipper (all gems) is 0.25.2.

Didn't know good ways how to debug it, so I'm currently setting it aside, but I'm interested in solving it. If you have good hints how to make Rack or whatever needed to spit out more logs to understand the situation, let me know. I don't know too much about Rack internals, my experience with it is very basic.

[ikropotov]

@ilvez We have similar configuration: Cloudflare is resolving TLS, and then sends http to our GCP load balancer, which proxies it to the Rails app. this was in the logs WARN -- : attack prevented by Rack::Protection::HttpOrigin Digging it down it comes to this check: https://github.com/sinatra/sinatra/blob/main/rack-protection/lib/rack/protection/http_origin.rb#L35

     def base_url(env)
        request = Rack::Request.new(env)
        port = ":#{request.port}" unless request.port == DEFAULT_PORTS[request.scheme]
        "#{request.scheme}://#{request.host}#{port}"
      end

      def accepts?(env)
        return true if safe? env
        return true unless (origin = env['HTTP_ORIGIN'])
        return true if base_url(env) == origin
        return true if options[:allow_if]&.call(env)

        permitted_origins = options[:permitted_origins]
        Array(permitted_origins).include? origin
      end

so origin is https:// version proxied request however was resolving base_url to http://

so this line is giving False: return true if base_url(env) == origin

now digging base_url, it was the request.scheme that was misaligned, and this one is calculated here: https://github.com/rack/rack/blob/main/lib/rack/request.rb#L249

def scheme
        if get_header(HTTPS) == 'on'
          'https'
        elsif get_header(HTTP_X_FORWARDED_SSL) == 'on'
          'https'
        elsif forwarded_scheme
          forwarded_scheme
        else
          get_header(RACK_URL_SCHEME)
        end
      end

and here digging into forwarded_scheme I see that it expects this header: https://github.com/rack/rack/blob/main/lib/rack/request.rb#L746

  HTTP_X_FORWARDED_SCHEME = 'HTTP_X_FORWARDED_SCHEME'

but neither GCP proxy, nor CloudFlare were not sending this, only: HTTP_X_FORWARDED_FOR: HTTP_X_FORWARDED_PROTO:

So I decided to resolve on this level, ie def scheme I couln't figure out what is the value for HTTPS header, however HTTP_X_FORWARDED_SSL is clearly defined, so I set this header to be added on the GCP loadbalancer: X-FORWARDED-SSL: 'on'

And it worked like a charm. Cheers.

[jnunemaker]

I wonder if I should change builder.use Rack::Protection, rack_protection_options to use Rack::Protection::AuthenticityToken. @ikropotov do you think that would have helped you? I thought that by setting rack protection options to rack_protection_options = options.fetch(:rack_protection, use: :authenticity_token) that is what I was doing but now I'm thinking it doesn't work that way.

[coalest]

@jnunemaker I work with @ikropotov so I can try to answer that.

I'm not positive because I haven't tested that, but yes I think that would have helped us. Because like @ikropotov said, initially Rack::Protection::HttpOrigin was failing, but even when I just excluded that middleware with { except: %i[http_origin] } the issue was still occurring and a different Rack Protection middleware started causing an issue.

Looking at this, I think when you use builder.use Rack::Protection, rack_protection_options, you are adding Rack::Protection::AuthenticityToken (because it is not a default) but also adding all the default Rack Protection middlewares:

Rack::Protection::FrameOptions
Rack::Protection::HttpOrigin
Rack::Protection::IPSpoofing
Rack::Protection::JsonCsrf
Rack::Protection::PathTraversal
Rack::Protection::RemoteToken
Rack::Protection::SessionHijacking
Rack::Protection::XSSHeader

I'm not sure which Rack protections you want/intend to apply though.

[gillisd]

I was experiencing this issue and determined it was caused by how nginx & AWS ALB handle client IP's

The IPSpoofing class of RackProtection checks to make sure the value of X-Client-IP and X-Real-IP are contained in the list of IP's found in the X-Forwarded-For header. After ensuring that this condition was met, everything worked as normal.

[MiralDesai]

I had this issue and resolved it by effectively copying what I had for sidekiq, hopefully it helps someone in the future.

mount Flipper::UI.app(Flipper) { |builder|
    builder.use ActionDispatch::Cookies
    builder.use ActionDispatch::Session::CookieStore, key: Rails.application.config.session_options[:key]
    builder.use Rack::Auth::Basic do |username, password|
      ActiveSupport::SecurityUtils.secure_compare(username, "hello") &
        ActiveSupport::SecurityUtils.secure_compare(password, "world")
    end
  } => "/flipper"`

Adding builder.use ActionDispatch::Cookies was what stopped the 403's for me. Using rails in api mode so having to set session stuff explicitly.

[davidrunger]

Sorry to add to this thread, but I also was getting 403 responses when trying to create a Flipper feature in production via the web UI; there was WARN -- : attack prevented by Rack::Protection::HttpOrigin message in my logs for each attempt to create a Flipper feature.

What fixed it for me was changing the Referrer-Policy header set by NGINX from no-referrer to strict-origin-when-cross-origin. (When the Referrer-Policy header was no-referrer, the Origin header sent with my requests to create a new Flipper feature flag was set to null, which is what caused the rack-protection HttpOrigin check to fail.)

[jnunemaker]

@davidrunger no apology needed. Thanks for adding more context. I really need to just lock rack protection down to the minimum it seems.

[davidrunger]

I really need to just lock rack protection down to the minimum it seems.

@jnunemaker Yeah, that might be a good idea.

Alternatively, for what it's worth, in my particular case, I have NGINX configured to respect the Referrer-Policy header, if it is set by the "upstream" server that NGINX is proxying. The no-referrer value that I mentioned that NGINX was setting for the Referrer-Policy header for Flipper responses was just the default/fallback value that I was having NGINX add to the response headers if the upstream server has not itself already set such a header.

So, in my particular situation, I would not have encountered a 403 response when trying to create a new Flipper feature if Flipper would set a Referrer-Policy header that is compatible with the requirements of the Rack::Protection::HttpOrigin check. That could be accomplished via the following change:

diff --git a/lib/flipper/ui/action.rb b/lib/flipper/ui/action.rb
index 0ff4053e..e0a6a618 100644
--- a/lib/flipper/ui/action.rb
+++ b/lib/flipper/ui/action.rb
@@ -38,6 +38,7 @@ module Flipper
         style-src-elem 'self';
         connect-src https://www.flippercloud.io;
       CSP
+      REFERRER_POLICY = 'strict-origin-when-cross-origin'.freeze

       # Public: Call this in subclasses so the action knows its route.
       #
@@ -138,6 +139,7 @@ module Flipper
       def view_response(name)
         header Rack::CONTENT_TYPE, 'text/html'
         header 'content-security-policy', CONTENT_SECURITY_POLICY
+        header 'referrer-policy', REFERRER_POLICY
         body = view_with_layout { view_without_layout name }
         halt [@code, @headers, [body]]
       end

That all being said, as you suggested, it's very possible that an easier and/or more generally reliable solution might instead be to dial down Flipper's usage of rack-protection to only perform checks deemed essential (which presumably might not include the Rack::Protection::HttpOrigin check that was an issue for me and a few others who have commented in this thread).