[JENKINS-12875] "No valid crumb was included in the request" errors all around

[timja]

After upgrading Jenkins from 1.448 to 1.451 (with yum, upgraded plugins too) several parts of Jenkins have become unusable with error "No valid crumb was included in the request":

Console output

• When building, the log is unavailable, see console_log.png
• Once the build is done, the complete log is available, but the same error appears on the side console_side.png

Build configuration

• Builds cannot be launched anymore from the build page (The firebug error console displays the "No valid crumb was included in the request" error). They still can be launched from the dashboard though.
• Cannot add build steps (when choosing a build step type an error is reported in background), see add_build_step.png

Manage Jenkins configuration

• The page loads only partially (first half is permanently greyed out, see configuration.png). The firebug error console displays the "No valid crumb was included in the request" error.

Manage Jenkins plugins

• Cannot upgrade nor downgrade, see error shown plugins.png

---

Originally reported by naloy, imported from: "No valid crumb was included in the request" errors all around

• assignee: danielbeck
• status: Resolved
• priority: Critical
• resolution: Fixed
• resolved: 2016-04-01T23:57:26+00:00
• imported: 2022/01/10

[timja]

naloy:

After yet another restart things seems to be back to normal (???!!)

[timja]

valentine:

Not completely normal. The loading thing in the management section is still happening

[timja]

naloy:

In the Jenkins configuration page:

Error: Form.findMatchingInput is not a function
Source File: http://jenkins.rightnow.com:8080/static/6dcd23a9/scripts/hudson-behavior.js
Line: 338

[timja]

windyroad:

I'm not as blocked as the reporting user, but I'm having the same problem (no valid crumb..). In my case it only seems to affect the console output, though the error can be seen for "ajaxBuildQueue" and "ajaxExecuters" on the config page, the home page (and I'm assuming others).

What's interesting is that I only get this problem through a reverse proxy. http://ve.mbqblqmj.vesrv.com:8080/ works fine, but http://ci.windyroad.org/ (which is RPed to http://ve.mbqblqmj.vesrv.com:8080/) does not.

[timja]

windyroad:

I've offered USD100.00 at FreedomSponsors to have this fixed (http://www.freedomsponsors.org/core/issue/31/no-valid-crumb-was-included-in-the-request-errors-all-around).
Once you solve it (according to the acceptance criteria described there), just create a FreedomSponsors account and mark it as resolved (oh, you'll need a Paypal account too)
I'll then check it out and will gladly pay.

[timja]

onemanbucket:

I reproduced the problem with a simple nginx reverse proxy and csrf enabled.

The problem is that jenkins stores its' csrf token in a http header called '.crumb', AFAIK headers must only contain alphanumerics and dashes, and nginx will remove invalid headers from the request (unless configured not to).

Sooo, a workaround is to use ignore_invalid_headers in nginx or disable csrf protection until patched.

I'll post a pull request shortly

[timja]

onemanbucket:

Actually, it seems that you can set the header name with a system property to jenkins (hudson.security.csrf.requestfield). So another workaround could be to start jenkins with -Dhudson.security.csrf.requestfield=crumb

[timja]

onemanbucket:

Pull request: https://github.com/jenkinsci/jenkins/pull/596

[timja]

algob:

I've same issue, if I try to disable a project (use Jenkins v1.502):
http://rhodan:8080/jenkins/job/myproject_3562/disable

It looks like, I would have the same issue, if I would try to enable it, if it is disabled: JENKINS-14501

I tried to added these line to /opt/tomcat/bin/catalina.sh, but it didn't solve the issue:
— SNIP —
JAVA_OPTS="$JAVA_OPTS -Dhudson.security.csrf.requestfield=crumb"
— SNAP —

[timja]

ozawa:

Today, I've had also the same issue. Delete jobs or clone from github hasn't worked correctly.
This problem is caused after upgrading 1.515.

[timja]

johnnygoods:

Also seeing this issue when using jenkins 1.517 behind nginx reverse proxy. Setting nginx ignore_invalid_headers had no effect. Disabling CSRF did.

[timja]

vadakr:

To disable CSRF (Screenshot) -

Jenkins > Manage Jenkins > Configure Global Security

At the very bottom, you will see a checkbox for "Prevent Cross Site Request Forgery exploits". Uncheck this and apply.

[timja]

daemoen:

Tested multiple scenarios.

Following configs work:
OS: Centos 6 x86_64
Jenkins: 1.561 from Jenkins Repo

NGINX Reverse Proxy Setup test #1 (works):
/etc/nginx/conf.d/jenkins.conf

#
# Jenkins over SSL
#

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  ignore_invalid_headers off;
  listen 443;
  server_name jenkins.domain.com;

  ssl on;
  ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt;
  ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key;

  location / {
    proxy_set_headerHost $host;
    proxy_set_headerX-Real-IP $remote_addr;
    proxy_set_headerX-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_redirect http:// https://;
    proxy_pass      http://jenkins;
  }
}

NGINX Reverse Proxy Setup test #2 (does NOT work):
/etc/nginx/conf.d/jenkins.conf

#
# Jenkins over SSL
#

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  ignore_invalid_headers on; # Or simply leave it unspecified; This is the DEFAULT behavior of ignore_invalid_headers
  listen 443;
  server_name jenkins.domain.com;

  ssl on;
  ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt;
  ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key;

  location / {
    proxy_set_headerHost $host;
    proxy_set_headerX-Real-IP $remote_addr;
    proxy_set_headerX-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_redirect http:// https://;
    proxy_pass      http://jenkins;
  }
}

NGINX Reverse Proxy Setup test #3 (works):
/etc/nginx/conf.d/jenkins.conf

#
# Jenkins over SSL
#

upstream jenkins {
  server 127.0.0.1:8080 fail_timeout=0;
}

server {
  listen 80;
  server_name jenkins.domain.com;
  return 301 https://$host$request_uri;
}

server {
  ignore_invalid_headers on;
  listen 443;
  server_name jenkins.domain.com;

  ssl on;
  ssl_certificate /etc/nginx/ssl/wildcard.domain.chained.crt;
  ssl_certificate_key /etc/nginx/ssl/wildcard.domain.key;

  location / {
    proxy_set_headerHost $host;
    proxy_set_headerX-Real-IP $remote_addr;
    proxy_set_headerX-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_redirect http:// https://;
    proxy_pass      http://jenkins;
  }
}

/etc/sysconfig/jenkins

JENKINS_JAVA_OPTIONS="-Djava.awt.headless=true -Dhudson.security.csrf.requestfield=crumb"

[timja]

daemoen:

Further testing of this shows that even with the working setups above, chrome still does not like it. Firefox works fine.

[timja]

davea_br:

I can confirm that this also occurs on Centos 6.5 with Nginx 1.6 using Firefox.

[timja]

jglick:

Be warned that -Dhudson.security.csrf.requestfield=crumb will cause the attempts by some REST API clients to obtain a crumb to be rejected by the fix of SECURITY-47, as crumb is a valid JavaScript identifier that would open you to attack.

[timja]

elephantjim:

I'm using Jenkins behind a reverse proxy using nginx, too. I tried to set ignore_invalid_headers to "off" on my server's nginx settings to no avail. But then I found this: http://trac.nginx.org/nginx/ticket/117

It turns out that nginx needs to process headers before selecting a virtual server: the "Host" header determines which server to use. Unless the ignore_invalid_headers is set to off on the default server, it won't get read. The other way is to set it in nginx.conf in the http context. That worked for me.

[timja]

blaisorblade:

I ran into this yesterday. What's the status? Somebody proposed an X-Jenkins-Crumb header (in JENKINS-7518), which doesn't seem a legal JavaScript identifier (though it might be "the start of a legal JavaScript statement").
Would X-Jenkins-Crumb-0NoJS help, since 0NoJs is an illegal JS token?

Also, SECURITY-47 isn't visible in the bug tracker, apparently on purpose — I've only found the fix (https://github.com/jenkinsci/jenkins/commit/0de3e9b14ed75f70279435e78eb9f6a3a1a179df) and maybe the advisory (https://wiki.jenkins-ci.org/display/SECURITY/Jenkins+Security+Advisory+2013-02-16).

For my part, I tried to document this on https://wiki.jenkins-ci.org/display/JENKINS/Jenkins+behind+an+NGinX+reverse+proxy; I only mentioned the most secure workaround, not the -Dhudson.security.csrf.requestfield= setting. Improvements are welcome.

[timja]

timboudreau:

Running into this too, with NginX, on https://timboudreau.com/builds

I've set ignore_invalid_headers to off, but I'm not keen on leaving it that way, especially for an NginX instance that is proxying far more than just Jenkins - there are other forms of nastiness that would be better not to pass on to back end applications that makes possible.

[timja]

tomreyn:

The number of people watching this issue is pretty high compared to that of other issues. This may provide an indication as to how much of an issue it is.

C.f. http://seclists.org/bugtraq/2015/Aug/161

Adding a couple HTTP headers would also help to harden the security of a Jenkins installation (may break some Jenkins plugins - works for me). Assuming your domain name is EXAMPLE.ORG and you use HTTPS:

Content-Security-Policy: "default-src 'self' EXAMPLE.ORG *.EXAMPLE.ORG; script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src *; media-src *; object-src 'self'; frame-src 'self' updates.jenkins-ci.org;"
X-Frame-Options: "sameorigin, DENY"
X-XSS-Protection: "1; mode=block"
X-Content-Type-Options:"nosniff"
Strict-Transport-Security:"max-age=15768000"

[timja]

timboudreau:

> as crumb is a valid Javascript identifier that would open you to attack

Would like to understand better how a header name being a Javascript identifier opens one to attack - there are plenty of standard HTTP header names that are valid identifiers - if I make an HTTP request containing

Date: eval("alert('Hey')")

is something foolish enough to execute it?

At any rate, the problem here is simple - ".crumb" is not a legal HTTP header, but there are plenty of options that are legal HTTP headers and illegal as js identifiers - one has already been suggested. I don't think any of us want to leave our NginX installs set to accept whatever garbage a client decides to turn into headers.

Re the suggested additional headers: Some are not a bad idea, but I could write a NodeJS proxy that would strip those out in about 10 minutes and go hog wild (or just use a programmatic HTTP client) - you can't rely on that sort of thing to give you real security, just deter low-hanging-fruit bad behavior.

[timja]

jglick:

Would like to understand better how a header name being a Javascript identifier opens one to attack

It is complicated but has to do with how the crumb is offered to legitimate clients.

Anyway the fix is well understood and filed, “just” needs to be tested and reviewed by someone qualified in web security.

[timja]

danielbeck:

timboudreau You wrote:

".crumb" is not a legal HTTP header, but there are plenty of options that are legal HTTP headers and illegal as js identifiers

Just curious, where did you get this information from? Neither RFC 2822 (section 2.6.8) nor RFC 7230 seem to impose any header name restrictions beyond being printable ascii and not including spaces or colons. (Not saying this shouldn't be fixed due to compatibility, but I looked repeatedly, even back when I worked on this, and couldn't find anything like what you're saying).

[timja]

blaisorblade:

On header names: I mostly agree with Daniel Beck about the spec (after reading both specs, I've also looked at http://stackoverflow.com/a/3569667/53974 — beyond colons, other "separators" are forbidden, but not dots). I think our source is just what the nginx docs imply without giving any reference (http://nginx.org/en/docs/http/ngx_http_core_module.html#ignore_invalid_headers).

> Re the suggested additional headers: Some are not a bad idea, but I could write a NodeJS proxy that would strip those out in about 10 minutes and go hog wild (or just use a programmatic HTTP client) - you can't rely on that sort of thing to give you real security, just deter low-hanging-fruit bad behavior.

At least Strict-Transport-Security is there to offer some protection for clients (browsers) that do honor them (and I think some other ones too). Not sure how you propose to use the proxy, but that headers assumes Jenkins is behind HTTPS, and then attackers can't drop the header.

However, please don't silently enable Strict-Transport-Security — that's something for the sysadmin to decide, and it's not fun if one HTTP application enables it on a host where other services don't support HTTPS.

Anyway, this is OT here, maybe somebody (Tom Reynolds) can open a different bug for them? (I don't understand the headers well enough to do that).

[timja]

danielbeck:

We already got some enthusiastic feedback how broken Jenkins is with nginx. Looks like we actually should address this before shipping with CSRF protection enabled by default.

[timja]

jglick:

Assigning to danielbeck on the basis that he filed PR 1319 which possibly just needs to have some merge conflicts resolved and be retested, but feel free to reassign if you think someone else should pick it up.

[timja]

scm_issue_link:

Code changed in jenkins
User: Daniel Beck
Path:
core/src/main/java/hudson/security/csrf/CrumbFilter.java
core/src/main/java/hudson/security/csrf/CrumbIssuer.java
core/src/main/java/hudson/security/csrf/CrumbIssuerDescriptor.java
core/src/main/java/hudson/security/csrf/DefaultCrumbIssuer.java
test/src/test/groovy/hudson/model/SlaveTest.groovy
http://jenkins-ci.org/commit/jenkins/ed0ea631cd4af0e678da96d7297910ba1612749d
Log:
[FIXED JENKINS-12875] Change default crumb name to Jenkins-Crumb

[timja]

scm_issue_link:

Code changed in jenkins
User: Daniel Beck
Path:
test/src/test/java/lib/form/RowSetTest.java
http://jenkins-ci.org/commit/jenkins/5b1642252ebd977c21832eb12cae633b3616429f
Log:
JENKINS-12875 Fix test

[timja]

danielbeck:

Fixed towards Jenkins 2.0 which is scheduled to be released in April.

[timja]

scm_issue_link:

Code changed in jenkins
User: Daniel Beck
Path:
core/src/main/java/hudson/security/csrf/CrumbIssuer.java
http://jenkins-ci.org/commit/jenkins/8b6d38de67154fb96c59791082d5cbe743b2b693
Log:
JENKINS-12875 Restrict access to constant

[timja]

scm_issue_link:

Code changed in jenkins
User: Daniel Beck
Path:
core/src/main/java/hudson/security/csrf/CrumbFilter.java
core/src/main/java/hudson/security/csrf/CrumbIssuer.java
core/src/main/java/hudson/security/csrf/CrumbIssuerDescriptor.java
core/src/main/java/hudson/security/csrf/DefaultCrumbIssuer.java
test/src/test/groovy/hudson/model/SlaveTest.groovy
test/src/test/java/lib/form/RowSetTest.java
http://jenkins-ci.org/commit/jenkins/a87c38c5052cba745127a0205c40490b30c8bdbf
Log:
Merge pull request #2198 from daniel-beck/JENKINS-12875

[FIX JENKINS-12875] Change default crumb name to Jenkins-Crumb

Compare: https://github.com/jenkinsci/jenkins/compare/4649e04cbe58...a87c38c5052c

[timja]

jknurek:

Maybe it's worth noting in this ticket:
I installed Jenkins 2.0-rc-1 and created a view using Build Monitor View plugin (1.8+build.201601112328), and get this error (which leaves the UI in an infinite "sorry" loop

Apr 14, 2016 8:16:03 AM hudson.security.csrf.CrumbFilter doFilter
WARNING: No valid crumb was included in request for /$stapler/bound/18964be5-e86d-4242-a57b-35ee573d52c9/fetchJobViews. Returning 403.

[timja]

danielbeck:

jknurek That's unrelated to this issue and looks like a bug in the plugin that cannot handle the CSRF protection option (around since 2010 or so) being enabled.

[timja]

jknurek:

My first thought was that it was a bug specific to the Build Monitor View plugin, but when I run Jenkins 1.651.1, the same configuration works ok. I'm only getting this error with the 2.0-rc-1.
(but I'm not sure what the underlying issue is in this case, so it is likely a separate issue)

[timja]

janek:

danielbeck Build Monitor has supported CSRF protection since its early days

The problem is caused by change https://github.com/jenkinsci/jenkins/compare/4649e04cbe58...a87c38c5052c , which replaced the hard-coded '.crumb' header with 'Jenkins-Crumb', that's why they no longer match.

I can work around that in Build Monitor.

jknurek you can watch the progress of this issue on github at https://github.com/jan-molak/jenkins-build-monitor-plugin/issues/215, which is the official issue tracker for Build Monitor.

Thanks!
Jan

[timja]

danielbeck:

janek

which replaced the hard-coded '.crumb' header with 'Jenkins-Crumb', that's why they no longer match.

The crumb name has been configurable since 1.310 (hudson.security.csrf.requestfield), and that was necessary for anyone running Jenkins behind nginx, as it dropped the .crumb header (JENKINS-12875). So we renamed the default to something nginx accepts. Maybe it's a good idea to check which crumb name is used by Jenkins in your plugin to adapt to user configuration.

[timja]

janek:

jknurek - Fixed in Build Monitor 1.9+build.2016043001​28, which you can download here before it hits the official update centre.

[timja]

janek:

Awesome, thanks danielbeck, that's exactly what I did. The latest Build Monitor picks up the crumb name from CrumbIssuer::getCrumbRequestField so it should be a more appropriate solution

[timja]

nvn:

Hi,
I have Jenkins 2.7, I have created the service hook in TFS to trigger the automatic build but while testing the hook I get " No valid crumb was included in the request (403)". I can run the same build from Jenkins dashboard with no issue.
I have also installed the downgrade Build - Plugin-Version: 1.9+build.201606171408 as suggested by Jan Molak above but I am still facing the issue.

P.s - I have installed Jenkins on my local windows 2007 machine.

Thanks,

[timja]

janek:

Hey Naveen - from what you're saying it seems like the problem is related to Jenkins and not the Build Monitor plugin?
If you think it's related to Build Monitor please feel free to raise a ticket on github at https://github.com/jan-molak/jenkins-build-monitor-plugin

Thanks,
Jan

[timja]

nvn:

Hi Jan, yeah I think the issue is either with Jenkins or may be with Tfs-jenkins plugin, I am not sure. Pls suggest.

Thanks

[timja]

12niner:

Having just wasted a few hours debugging why I was seeing this: if you re-install Jenkins you need to clear your cookies or you'll hit this.

[timja]

danielbeck:

12niner Too many cookies is tracked as JENKINS-25046.

[timja]

[Originally duplicated by: JENKINS-24249]

[timja]

[Originally duplicated by: JENKINS-27972]

[timja]

[Originally duplicated by: JENKINS-25116]

[timja]

[Originally duplicated by: JENKINS-23793]

[timja]

[Originally related to: JENKINS-7518]

[timja]

[Originally related to: JENKINS-12787]

[timja]

[Epic: JENKINS-33810]