Closed krzcho closed 8 years ago
warden was updated to latest 1.2.6 with no luck access constraint was updated and it works fine from localhost (remote flipping still fails but I guess I will need to live with it)
class CanAccessFlipper
def self.matches?(request)
current_user = request.env['warden'].user
(current_user.present? && current_user.respond_to?(:is_admin?) && current_user.is_admin?) || request.remote_ip == '127.0.0.1'
end
end
Could you add some logging to get some more information? Check to see which of the conditionals is returning false maybe (ie: request ip or current user or is admin). Also, not much I can do to help without more information. I'd probably need you to paste in here anything related to flipper in your app.
I ran staging locally, and I have this in logs:
W, [2016-04-12T17:29:59.801209 #16401] WARN -- : attack prevented by Rack::Protection::AuthenticityToken
W, [2016-04-12T17:30:01.924352 #16401] WARN -- : attack prevented by Rack::Protection::AuthenticityToken
W, [2016-04-12T17:30:04.102610 #16401] WARN -- : attack prevented by Rack::Protection::AuthenticityToken
Looks like middleware related issue...
I was expecting to find this middleware somewhere in rails.. But it was a wrong path. It turned out it comes from FlipperUI: https://github.com/jnunemaker/flipper/blob/master/lib/flipper/ui.rb#L30-L40
Now I need to investigate why it does not work for me..
I went crazy with this error..I want to provide the information, that I found out her:
csrf
token, see implementation of https://github.com/sinatra/rack-protection/blob/master/lib/rack/protection/authenticity_token.rb)Hope, those fact I provided can be useful for somebody else, who'll get this error and try to track it down.
Without more information there isn't anything I can do. If anyone could setup an example app where this happens or figure out what there problem is and post here I'm happy to add docs. The key really is that you have a session before flipper. If you are getting a 403, I'd put good money on rack protection. Closing, but please drop any more thoughts or help here and I'll keep following along.
I had a similar problem. Soved it by modifying nginx config.
See sidekiq discussion: http://stackoverflow.com/questions/37270614/forbidden-sign-out-using-sidekiq-devise-activeadmin-on-production-server-with
Shameless copypaste from that thread:
In Nginx server configuration you need:
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
Don't have much to add, other than we're experiencing the same thing, and it looks to be as you thought with an existing session.
Our setup:
We use basic auth on the flipper stuff and Devise/warden on the non-flipper stuff. These two "apps" are constrained by subdomain routes, but once you've logged into the devise "app" (subdomain A), the flipper UI (with basic auth) (subdomain B) starts throwing this forbidden error at us.
Opening flipper UI up in private browsing window avoids the issue (as you'd expect as it doesn't have the existing devise session from subdomain A).
The warden cookie's domain is set to work on all subdomains of our domain.
@shimms how are you authenticating the flipper ui?
Custom middleware which performs basic_auth if the subdomain is matched, otherwise continues normal processing:
class AdminAuth < Rack::Auth::Basic
def call(env)
# perform auth if admin domain
return super if request_subdomain(Rack::Request.new(env)) == ENV.fetch('ADMIN_SUBDOMAIN')
@app.call(env) # skip auth
end
private
def request_subdomain(request)
request.env['HTTP_HOST'].split('.').first
end
end
Flipper mounted as:
constraints :subdomain => ENV.fetch('ADMIN_SUBDOMAIN') do
...
mount Flipper::UI.app(Teamsquare::Application.features) => '/'
...
end
I ran into this issue today, and solved it by clearing my cookies for the site in question. Thanks to @greyblake for mentioning the CSRF issue, that helped.
I'm having this same issue in Safari in production environment only (no issue in staging or development), accessing with Firefox instead seems to be a workaround.
I think I had similar issues with sidekiq-web, and have the following code in my routes.rb as a fix:
Sidekiq::Web.set :session_secret, Rails.application.secrets[:secret_key_base]
Is there any way to set this same option for Flipper UI, or should this already be happening automatically?
I'm having exactly the same issue mentioned here with Google Chrome in production environment.
Also seeing this on my server logs:
WARN -- : attack prevented by Rack::Protection::AuthenticityToken
Tried to disable the related rack protection with:
mount Flipper::UI.app(Flipper, {:rack_protection => {}}) => '/flipper'
and also tried to use the same cookie currently used by my application with:
mount Flipper::UI.app(Flipper) { |builder|
builder.use Rack::Session::Cookie, :secret => Rails.application.secrets[:secret_key_base],
:key => Rails.application.config.session_options[:key]
} => '/flipper'
and tried to set headers on my nginx proxy:
proxy_set_header X-Forwarded-Proto $scheme;
proxy_set_header X-Forwarded-Ssl on;
proxy_set_header X-Forwarded-Port $server_port;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $http_host;
proxy_set_header X-Real-IP $remote_addr;
also cleared my browser data as suggested before but nothing seems to work 😞 I'm really frustrated with this issue.
I exported the Chrome HTTP request toggling one of the flags
that I have as CURL and imported it on Postman app, it works like a charm. Also works on Firefox. Looks like Chrome is doing something wrong.
The only difference that I could spot between chrome and firefox requests was the Host
header, it is present on firefox request but is not on chrome.
I am almost certain that is a nginx related issue. I could trace it down to HTTP_ORIGIN
header being passed wrong (?) by nginx because I set a blank Rails project on my local machine with flipper and enabled production mode and could not reproduce the issue, also locally running my application in production mode can not reproduce the issue.
Currently I'm using cloudflare -> nginx proxy -> puma socket
and I could make it work on Google Chrome doing a dark evil magic simple trick like this:
# routes.rb
constraints CanAccessFlipperUI do
mount Flipper::UI.app(Flipper) => '/dolphin_evil_flags'
end
# flipper.rb
class CanAccessFlipperUI
def self.matches?(request)
# the trick (do not use in production, is unsafe)
request.set_header('HTTP_ORIGIN', nil)
current_user = request.env['warden'].user
current_user.present? && current_user.id.to_s == '1'
end
end
I know it's far from ideal and can potentially be a security breach but I'm still debugging my nginx stack (specially headers) to find out how to fix this issue once by all.
We ran into this problem on https://dev.to (and I spent HOURS trying to fix it), but our stack runs on Heroku so there is no nginx proxy to configure. We worked around it by disabling CSRF protections in the Rack middleware running Flipper UI, and relying upon protections in our main Rails application (https://github.com/forem/forem/pull/9360/files).
mount Flipper::UI.app(Flipper,
{ rack_protection:
{ except: %i[authenticity_token form_token json_csrf remote_token http_origin session_hijacking] } }
), at: "flipper"
@joshpuetz ugh. So sorry. Super weird. I use flipper ui on multiple heroku apps with no issues. I'd love to know more about what caused your problem. I'd be happy to even meet up for a short zoom or something if there was anything I could do to help.
I just got the same error on a brand new Mac. Downloaded our code base and tried this, never seen it on my older Mac. Same OS, 2018 vs 2020 MBP, so not so different.
Refreshing the form and resubmitting again worked fine.
Started POST "/flipper/features" for ::1 at 2020-08-27 14:22:10 -0700
W, [2020-08-27T14:22:10.047338 #44411] WARN -- : attack prevented by Rack::Protection::AuthenticityToken
If you are using haproxy in front of your nginx/rails, this option in haproxy configuration may help.
From haproxy documentation:
If you’d like to inform the backend server whether HTTPS was used, you can append an X-Forwarded-Proto request header by adding the http-request set-header directive:
http-request set-header X-Forwarded-Proto https if { ssl_fc } http-request set-header X-Forwarded-Proto http if !{ ssl_fc }
At least this helped me to solve forbidden error. Adding such header could help with others than haproxy ssl terminators/offloaders also.
Is this still any issue for anyone else? We use the same constraint for Sidekiq and have no issues.
No longer an issue for me
I had to use @joshpuetz way using rack_protection to get it to work in staging / production environments.
I'm still having the same problem. I was able to create a small application that reproduces it: https://github.com/matheussilvasantos/flipper_ui_broken.
Using Rack::Session::Cookie
instead of Rack::Session::Pool
for the session middleware fixed the error for me: https://github.com/matheussilvasantos/flipper_ui_broken/commit/b430e4d5ae2953f67e032f1c5962e618a179b460.
I still don't know why, but when you use Rack::Session::Pool
, the csrf
token isn't set in the session in https://github.com/sinatra/sinatra/blob/master/rack-protection/lib/rack/protection/authenticity_token.rb#L135.
I've just found the problem for my scenario. https://github.com/jnunemaker/flipper/pull/606 fixes it for me.
Letting you know that we have the same issue as above @joshpuetz. Disabling rack_protection
concerning CSRF checks did cure this situation for us. Our application is behind Cloudflare and Nginx reverse proxy. Flipper (all gems) is 0.25.2.
Didn't know good ways how to debug it, so I'm currently setting it aside, but I'm interested in solving it. If you have good hints how to make Rack or whatever needed to spit out more logs to understand the situation, let me know. I don't know too much about Rack internals, my experience with it is very basic.
@ilvez We have similar configuration:
Cloudflare is resolving TLS, and then sends http to our GCP load balancer, which proxies it to the Rails app.
this was in the logs
WARN -- : attack prevented by Rack::Protection::HttpOrigin
Digging it down it comes to this check:
https://github.com/sinatra/sinatra/blob/main/rack-protection/lib/rack/protection/http_origin.rb#L35
def base_url(env)
request = Rack::Request.new(env)
port = ":#{request.port}" unless request.port == DEFAULT_PORTS[request.scheme]
"#{request.scheme}://#{request.host}#{port}"
end
def accepts?(env)
return true if safe? env
return true unless (origin = env['HTTP_ORIGIN'])
return true if base_url(env) == origin
return true if options[:allow_if]&.call(env)
permitted_origins = options[:permitted_origins]
Array(permitted_origins).include? origin
end
so origin
is https:// version
proxied request however was resolving base_url to http://
so this line is giving False: return true if base_url(env) == origin
now digging base_url, it was the request.scheme
that was misaligned, and this one is calculated here:
https://github.com/rack/rack/blob/main/lib/rack/request.rb#L249
def scheme
if get_header(HTTPS) == 'on'
'https'
elsif get_header(HTTP_X_FORWARDED_SSL) == 'on'
'https'
elsif forwarded_scheme
forwarded_scheme
else
get_header(RACK_URL_SCHEME)
end
end
and here digging into forwarded_scheme
I see that it expects this header:
https://github.com/rack/rack/blob/main/lib/rack/request.rb#L746
HTTP_X_FORWARDED_SCHEME = 'HTTP_X_FORWARDED_SCHEME'
but neither GCP proxy, nor CloudFlare were not sending this, only: HTTP_X_FORWARDED_FOR: HTTP_X_FORWARDED_PROTO:
So I decided to resolve on this level, ie def scheme
I couln't figure out what is the value for HTTPS
header, however
HTTP_X_FORWARDED_SSL
is clearly defined, so I set this header to be added on the GCP loadbalancer:
X-FORWARDED-SSL: 'on'
And it worked like a charm. Cheers.
I wonder if I should change builder.use Rack::Protection, rack_protection_options
to use Rack::Protection::AuthenticityToken
. @ikropotov do you think that would have helped you? I thought that by setting rack protection options to rack_protection_options = options.fetch(:rack_protection, use: :authenticity_token)
that is what I was doing but now I'm thinking it doesn't work that way.
@jnunemaker I work with @ikropotov so I can try to answer that.
I'm not positive because I haven't tested that, but yes I think that would have helped us. Because like @ikropotov said, initially Rack::Protection::HttpOrigin
was failing, but even when I just excluded that middleware with { except: %i[http_origin] }
the issue was still occurring and a different Rack Protection middleware started causing an issue.
Looking at this, I think when you use builder.use Rack::Protection, rack_protection_options
, you are adding Rack::Protection::AuthenticityToken
(because it is not a default) but also adding all the default Rack Protection middlewares:
Rack::Protection::FrameOptions
Rack::Protection::HttpOrigin
Rack::Protection::IPSpoofing
Rack::Protection::JsonCsrf
Rack::Protection::PathTraversal
Rack::Protection::RemoteToken
Rack::Protection::SessionHijacking
Rack::Protection::XSSHeader
I'm not sure which Rack protections you want/intend to apply though.
I was experiencing this issue and determined it was caused by how nginx & AWS ALB handle client IP's
The IPSpoofing class of RackProtection
checks to make sure the value of X-Client-IP
and X-Real-IP
are contained in the list of IP's found in the X-Forwarded-For
header. After ensuring that this condition was met, everything worked as normal.
I had this issue and resolved it by effectively copying what I had for sidekiq, hopefully it helps someone in the future.
mount Flipper::UI.app(Flipper) { |builder|
builder.use ActionDispatch::Cookies
builder.use ActionDispatch::Session::CookieStore, key: Rails.application.config.session_options[:key]
builder.use Rack::Auth::Basic do |username, password|
ActiveSupport::SecurityUtils.secure_compare(username, "hello") &
ActiveSupport::SecurityUtils.secure_compare(password, "world")
end
} => "/flipper"`
Adding builder.use ActionDispatch::Cookies
was what stopped the 403's for me. Using rails in api mode so having to set session stuff explicitly.
Sorry to add to this thread, but I also was getting 403 responses when trying to create a Flipper feature in production via the web UI; there was WARN -- : attack prevented by Rack::Protection::HttpOrigin
message in my logs for each attempt to create a Flipper feature.
What fixed it for me was changing the Referrer-Policy
header set by NGINX from no-referrer
to strict-origin-when-cross-origin
. (When the Referrer-Policy
header was no-referrer
, the Origin
header sent with my requests to create a new Flipper feature flag was set to null
, which is what caused the rack-protection
HttpOrigin
check to fail.)
@davidrunger no apology needed. Thanks for adding more context. I really need to just lock rack protection down to the minimum it seems.
I really need to just lock rack protection down to the minimum it seems.
@jnunemaker Yeah, that might be a good idea.
Alternatively, for what it's worth, in my particular case, I have NGINX configured to respect the Referrer-Policy
header, if it is set by the "upstream" server that NGINX is proxying. The no-referrer
value that I mentioned that NGINX was setting for the Referrer-Policy
header for Flipper responses was just the default/fallback value that I was having NGINX add to the response headers if the upstream server has not itself already set such a header.
So, in my particular situation, I would not have encountered a 403 response when trying to create a new Flipper feature if Flipper would set a Referrer-Policy
header that is compatible with the requirements of the Rack::Protection::HttpOrigin
check. That could be accomplished via the following change:
diff --git a/lib/flipper/ui/action.rb b/lib/flipper/ui/action.rb
index 0ff4053e..e0a6a618 100644
--- a/lib/flipper/ui/action.rb
+++ b/lib/flipper/ui/action.rb
@@ -38,6 +38,7 @@ module Flipper
style-src-elem 'self';
connect-src https://www.flippercloud.io;
CSP
+ REFERRER_POLICY = 'strict-origin-when-cross-origin'.freeze
# Public: Call this in subclasses so the action knows its route.
#
@@ -138,6 +139,7 @@ module Flipper
def view_response(name)
header Rack::CONTENT_TYPE, 'text/html'
header 'content-security-policy', CONTENT_SECURITY_POLICY
+ header 'referrer-policy', REFERRER_POLICY
body = view_with_layout { view_without_layout name }
halt [@code, @headers, [body]]
end
That all being said, as you suggested, it's very possible that an easier and/or more generally reliable solution might instead be to dial down Flipper's usage of rack-protection
to only perform checks deemed essential (which presumably might not include the Rack::Protection::HttpOrigin
check that was an issue for me and a few others who have commented in this thread).
I am sorry in advance for maybe a noobish question but I am stuck After deploying flipper (with ui) to production and trying to flip feature I got 403 Forbidden from every action I am running two thins, http and ssl one with force ssl option on I see ui just fine but I can't trigger any action getting 403 Forbidden Going back to development environment (with single ssl thin) and actions work fine Any hints?
uncomfortable solution is to flip on development and copy flipper.pstore to production
Gems included by the bundle: