Remote Event Receivers not getting a valid token when using the "Copy to" action cross site collection

Target SharePoint environment

SharePoint Online

What SharePoint development model, framework, SDK or API is this about?

SharePoint Add-ins

Developer environment

Windows

What browser(s) / client(s) have you tested

[ ] 💥 Internet Explorer
[ ] 💥 Microsoft Edge
[ ] 💥 Google Chrome
[ ] 💥 FireFox
[ ] 💥 Safari
[ ] mobile (iOS/iPadOS)
[ ] mobile (Android)
[X] not applicable
[ ] other (enter in the "Additional environment details" area below)

Additional environment details

SP Addin built with .Net 4.8 and VS 2022

Describe the bug / error

Remote Event Receivers allows to intercept item updating or item adding event, and item updated or item added event.

However, these event has empty context token when action is issued from another site collection, especially using the standard OOB "Copy to action" from a document library. These leads to issues when some operation has to be made on the target site collection.

I have two site collections (let call them src and dest)

In the dest site collection I have a remote event receiver that triggers whenever a document is updated in the default document library (ItemAdded, ItemAdding, ItemUpdated and ItemUpdating event are tracked).

This works perfectly when I work whithin the site collection dest (drag and drop file from windows explorer or using the "copy to" action within the same site collection.

However, if I copy a file from src to dest using the OOB copy to action, the RER fails, having in the payload this error:

Failed to get the context token for the remote event receiver url.

After some tests, I land to the conclusion this is due to the browser session creating an auth token for the site src to call the copy file api.

Then the RER triggers using the context token for site src not site dest and fails because of that.

How to overcome this issue ?

Some details:

the RER is registered using an application. It handles AppInstalled event to register the receivers onto the host site's document library under the RER app context.
the error is taken from the incoming payload, in the SPRemoteEventProperties.ErrorMessage node
The RER is built using the addin model (.app package), uploaded in the site collection's app catalog, then activated (from add an app menu)

the manifest declare this permissions request :

<AppPermissionRequests AllowAppOnlyPolicy="true">
<AppPermissionRequest Scope="http://sharepoint/content/sitecollection"
                      Right="FullControl" />
</AppPermissionRequests>

As I said, it's working perfectly when working within the dest site.

sample SOAP payloads:

Payload of the ItemAdded event, if file copy within the same site collection :

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Body>
        <ProcessOneWayEvent xmlns="http://schemas.microsoft.com/sharepoint/remoteapp/">
            <properties xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
                <AppEventProperties i:nil="true"/>
                <ContextToken>eyJ0eXAiOiJKV1QiLC .........</ContextToken>
                <CorrelationId>8d8d6ba0-f0d7-5000-72d5-187dc612d273</CorrelationId>
                <CultureLCID>1033</CultureLCID>
                <EntityInstanceEventProperties i:nil="true" />
                <ErrorCode />
                <ErrorMessage />
                <EventType>ItemAdded</EventType>
                <ItemEventProperties>
                    <AfterProperties xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
                        <a:KeyValueOfstringanyType>
                            <a:Key>somefield</a:Key>
                            <a:Value i:type="b:int" xmlns:b="http://www.w3.org/2001/XMLSchema">14</a:Value>
                        </a:KeyValueOfstringanyType>
                        ... othter fields omitted for brievety ...
                    </AfterProperties>
                    <AfterUrl>Documents partages/myfile.pdf</AfterUrl>
                    <BeforeProperties xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays"/>
                    <BeforeUrl i:nil="true"/>
                    <CurrentUserId>14</CurrentUserId>
                    <ExternalNotificationMessage i:nil="true"/>
                    <IsBackgroundSave>false</IsBackgroundSave>
                    <ListId>96e3b5dc-7e94-4db9-b2cd-f0bbda1a59ab</ListId>
                    <ListItemId>64</ListItemId>
                    <ListTitle>Documents</ListTitle>
                    <UserDisplayName>Steve BEAUGÉ</UserDisplayName>
                    <UserLoginName>i:0#.f|membership|steve@mytenant.onmicrosoft.com</UserLoginName>
                    <Versionless>false</Versionless>
                    <WebUrl>https://mytenant.sharepoint.com/sites/mycoll</WebUrl>
                </ItemEventProperties>
                <ListEventProperties i:nil="true"/>
                <SecurityEventProperties i:nil="true"/>
                <UICultureLCID>1033</UICultureLCID>
                <WebEventProperties i:nil="true"/>
            </properties>
        </ProcessOneWayEvent>
    </s:Body>
</s:Envelope>

Payload when copied from different site collections :

<s:Envelope xmlns:s="http://schemas.xmlsoap.org/soap/envelope/">
    <s:Body>
        <ProcessOneWayEvent xmlns="http://schemas.microsoft.com/sharepoint/remoteapp/">
            <properties xmlns:i="http://www.w3.org/2001/XMLSchema-instance">
                <AppEventProperties i:nil="true"/>
                <ContextToken/>
                <CorrelationId>cbf86ba0-0089-5000-5f44-207718e56ee9</CorrelationId>
                <CultureLCID>1033</CultureLCID>
                <EntityInstanceEventProperties i:nil="true"/>
                <ErrorCode>GetContextTokenError</ErrorCode>
                <ErrorMessage>Failed to get the context token for the remote event receiver url https://url/of/my/function.</ErrorMessage>
                <EventType>ItemAdded</EventType>
                <ItemEventProperties>
                    <AfterProperties xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays">
                        <a:KeyValueOfstringanyType>
                            <a:Key>somefield</a:Key>
                            <a:Value i:type="b:int" xmlns:b="http://www.w3.org/2001/XMLSchema">14</a:Value>
                        </a:KeyValueOfstringanyType>
                        ... othter fields omitted for brievety ...
                    </AfterProperties>
                    <AfterUrl>Documents partages/myfile.pdf</AfterUrl>
                    <BeforeProperties xmlns:a="http://schemas.microsoft.com/2003/10/Serialization/Arrays"/>
                    <BeforeUrl i:nil="true"/>
                    <CurrentUserId>1073741823</CurrentUserId>
                    <ExternalNotificationMessage i:nil="true"/>
                    <IsBackgroundSave>false</IsBackgroundSave>
                    <ListId>96e3b5dc-7e94-4db9-b2cd-f0bbda1a59ab</ListId>
                    <ListItemId>64</ListItemId>
                    <ListTitle>Documents</ListTitle>
                    <UserDisplayName>Compte système</UserDisplayName>
                    <UserLoginName>SHAREPOINT\system</UserLoginName>
                    <Versionless>false</Versionless>
                    <WebUrl>https://mytenant.sharepoint.com/sites/mycoll</WebUrl>
                </ItemEventProperties>
                <ListEventProperties i:nil="true"/>
                <SecurityEventProperties i:nil="true"/>
                <UICultureLCID>1033</UICultureLCID>
                <WebEventProperties i:nil="true"/>
            </properties>
        </ProcessOneWayEvent>
    </s:Body>
</s:Envelope>

Differences :

when same site, my actual user account is set in the UserDisplayName and UserLoginName. When different sites, its system account
When same site, ContextToken is provider, no error message. When different sites, ContextToken is empty, but ErrorMessage contains the error

Steps to reproduce

Implement a RER project that subscrive to some event on host web
Copy file within the same site collection using Copy file action in the command bar ==> working
Copy file from another site collection into the library where the RER is set up ==> not working.

A full repro sample is available here : https://github.com/stevebeauge/Repro.MissingContext.

Press F5 from the solution to deploy and add the application to a site, a sample MyList library will be created with RER plugged.

Expected behavior

Remote Event Receivers SOAP payloads should contains a valid context token when triggerd by a user.

Whether the action is executed from the site collection or not.

Hi @stevebeauge, about a year ago I raised a support ticket with Microsoft regarding this exact scenario and got the following final response after a lot of discussions back and forth:

"As P... mentioned I took ownership of this case as he is out of office, but before he left he chased Engineering to ask them to come to a conclusion on this request. They just did: The current behavior is expected and is a consequence of the current design of the CreateCopyJobs API, which does require the elevation of privileges to work correctly. This design has always been present, is expected, and your remote event receiver must handle this possible scenario. As P... suggested, should you need to connect to SharePoint on those events in the remote event receiver, you should do so using app-only permissions."

I then also raised some concerns on how to validate that the call was valid since there is no token to validate and got the following response:

"As per your concerns: How do you propose I could verify the user using an app-only request to SharePoint when I don’t have a token to validate but rather just the username which could easily be spoofed in the request? And sure the request could come from a malicious server but the token would still need to be valid since it is validated by the TokenHelper class when the SharePoint context is being constructed or are you talking about something else? The preferred method would be adopting web hooks as this is the latest technology to achieve the results you intend, as they have replaced Remote Event Receivers, as documented here: Important As of January 2017 SharePoint Online does support list webhooks which you can use instead of "-ed" remote event receivers. Also please take into consideration that Remove Event Receivers were created for the older legacy models, such as SharePoint Add-Ins, and it the alternative shared previously by my colleague is explained here: Event receivers added to host web without app context, for example with SharePointOnlineCredentials or other means, will not return access token and you'll have to access the host web with app-only access token In your scenario if you want to keep such a solution with the change we suggested, and make sure there aren't any concerns, the alternative would be to, before any action, verify and confirm the change in SPO, that it actually happened, in that way, any theoretical issue would be mitigated."

As you can see they state that this is by design.

Just posting this here for your knowledge not as a real solution and I am still hoping for someone to provide a better solution for this issue. I feel that webhooks are not a replacement for remote event receivers, just a complement, since there are a lot of scenarios where we need synchronous event handling.

SharePoint / sp-dev-docs