Instruct user agents to report invalid policies

[neilstuartcraig]

Hi

I’m currently working on a phased plan to implement CSP for BBC online (as we’re currently rolling out HTTPS) and have come across an issue for which I have a suggestion - an amendment to the current implementation of CSP (apologies if you’ve had this suggested previously and I missed it).

We, like many other organisations of similar type and scale, want to comprehensively monitor our output to ensure it’s doing what we intended. For us, currently, CSP has a gap in just that area – namely ensuring that the CSP policies we instate are valid and thus will be enforced by compatible user agents. We can obviously pre-test (prior to deployment) our policies but we ideally want to continuously be sure they’re valid and being implemented, otherwise they’re completely ineffective. As things stand right now, achieving this ongoing policy validity monitoring is a little tricky – we could develop some tooling to help but that’d be synthetic so potentially error prone and would require infrastructure to run which means costs and peoples time and so on. Anyway, to cut to the chase, I’d love to see an addition to CSP to instruct user agents to report an invalid policy via the existing report-uri directive, perhaps something like this (invalid) example:

Content-Security-Policy: script-src 'something-invalid';
    report-invalid-policy true;
    report-to csp-reporting-endpoint

Which might generate a violation report like this for the above invalid policy:

{
  "csp-report":
  {
    "document-uri": "http://www.example.com/a/b",
    "referrer": "http://www.example.co.uk/a",
    "policy-errors": [
        "Invalid script-src directive",
        "..."
    ],
    "original-policy": "...",
    "source-file": "http://static.example.co.uk",
    "status-code": 200
  }
}

If report-invalid-policy defaulted to “off”, current behaviour would not be affected and thus the change would be backwards compatible. It would be sensible to recommend that user agents only report a broken policy once per session, to avoid inundating report-uri endpoints.

I believe this change would be relatively simple, backwards compatible and would negate the need for a lot of monitoring/tooling and thus time and cost for many individuals and organisations. Hopefully others might agree? I’ve made the same suggestion to the HPKP working group as this suffers the same problem from our perspective.

Cheers Neil

[michaelficarra]

I don't see how this could possibly work. If an old user-agent sees directives or values that it doesn't support, it needs to silently ignore them. If this proposal was accepted, those old user agents would be constantly reporting that they do not recognise the latest and greatest CSP additions.

Relatedly, new directives that both impose restrictions and fall back to default-src are breaking. This would be, in at least one way, an even worse form of that.

[neilstuartcraig]

Sorry for the slow reply @michaelficarra

I've perhaps not been too clear here, what I was intending was not that the user agent would report any directive or directive value it doesn't understand, but that it reports a policy which it cannot implement. This information would be invaluable and I don't see any other practical way of gather such data - equally the reports would be opt-in and thus anyone who's not interested doesn't need to do anything more than they currently do and would not receive excessive numbers of reports.

From the BBC (and likely many other similar size organiations) perspective, we'd definitely want to know if we were issuing broken CSP configurations so that we can deal with that and retain the safeguards which CSP provides. Without a change in this regard, CSP is something of a blindspot - we won't know if our policies are being applied or not.

Cheers

[hillbrad]

The philosophy of CSP has always been that it is a supplemental measure, and should not be counted upon precisely because in any sufficiently large population there will be users with browsers that don't support it.

It's not clear to me what you would do differently with this kind of report vs the information that a tool like Salvation ( http://engineering.shapesecurity.com/2015/09/announcing-salvation-csp-parser.html) or https://cspvalidator.org/ could provide as part of a CI process. What is the value add of the additional platform complexity?

-Brad

On Fri, Jul 1, 2016 at 6:10 AM Neil Craig notifications@github.com wrote:

Sorry for the slow reply @michaelficarra https://github.com/michaelficarra

I've perhaps not been too clear here, what I was intending was not that the user agent would report any directive or directive value it doesn't understand, but that it reports a policy which it cannot implement. This information would be invaluable and I don't see any other practical way of gather such data - equally the reports would be opt-in and thus anyone who's not interested doesn't need to do anything more than they currently do and would not receive excessive numbers of reports.

From the BBC (and likely many other similar size organiations) perspective, we'd definitely want to know if we were issuing broken CSP configurations so that we can deal with that and retain the safeguards which CSP provides. Without a change in this regard, CSP is something of a blindspot - we won't know if our policies are being applied or not.

Cheers

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/w3c/webappsec-csp/issues/96#issuecomment-229942293, or mute the thread https://github.com/notifications/unsubscribe/ACFbcNnXi_InI2iaKSfjOlsiAFpNiZgAks5qRRGrgaJpZM4JA9vk .

[neilstuartcraig]

Hi @hillbrad

Yes, I totally understand that CSP is supplemental and not to be relied upon. Again, I have probably not been 100% clear and my apologies for that. Dealing with your Q's in turn:

Our organisation (central platform + many product teams) means that we have a wide variety of implementations and Java is rarely a good fit (few of our web teams know it well enough or have existing Java in their stacks) so the specific example of Salvation is unlikely to be a good fit in most of our use cases.

In terms of the CSP validator, that's useful but we'll almost always automate creation of our CSP configs so we need to be able to validate them programmatically - but we can cope with that via tooling.

The data that we want to access is really around how effective our CSP configs are in terms of whether they're implementable by user agents. For example, if one of our teams builds a CSP generator for one of their websites, they'd implement some parsing/validation of the semantics as far as is practical but when they come to deploy the CSP config, we want to be able to give them feedback to validate that in the real world, their config is implemented - otherwise they're blindly pushing out CSP configs and just hoping that they never make a mistake - there's no feedback loop. The only way around this would be to test every page (or at least section) of every website against every user agent which is difficult and time consuming. Does that help clarify at all?

Cheers

[dveditz]

what I was intending was not that the user agent would report any directive or directive value it doesn't understand, but that it reports a policy which it cannot implement.

could you define what that means? A policy has directives which the implementation might know or not. If it doesn't know "scrpt-src" it obviously can't implement it, but how does it know which ones are mistakes and which are directives understood by future implementations? A known directive has host names (with or without paths) and quoted keywords. Same problem there, really.

It would help me understand what you mean if you gave an example of a policy an implementation "can't implement" and should report. Most mistakes I can imagine are incorrect hosts (extra, forgetting, or mistyping) and implementations can't know when those kinds of problems occur.

[neilstuartcraig]

Hi @dveditz

I'm thinking of this perhaps from a slightly wider angle. We'll automate the generation of our policies and since we have literally millions of pages, we can't lint all the policies before serving, especially on our long tail content. These two factors mean that we have the potential to generate broken policies, in the sense that perhaps they're inadvertently invalid JSON (which might vary depending on how strict the parsing is, e.g. trailing commas) or they contain directive values which are unimplementable (or perhaps what you're saying here is that this isn't realistically check-able?).

Does that help? I can see other large orgs likely being in a similar situation.

Cheers

[ScottHelme]

Checking myself in on this thread, interested to see where it goes.