bug: consecutive source-hub acp operations fail for http client due to an identity auth error

[shahzadlone]

While implementing #2762 in #2907 I came across a bug where calling a sourcehub operation on two different nodes failed i.e. attempting to share a document by the same owner with a 2nd owner on each 2 different nodes causes the first node's operation to succeed and the second nodes operation to fail.

My hunch was that this is not specific to the relationship sharing feature, as this is only an issue for http client with sourcehub.

I tested a basic example of creating 2 private documents using the same identity (which uses acp identity operations underneath ofc.), one on node 1 and second on node 2 using sourcehub with http client. As suspected the tests fail, hence this is an existing bug.

Here is a test that documents the failure:

func TestDEBUG(t *testing.T) {
    expectedPolicyID := "fc56b7509c20ac8ce682b3b9b4fdaad868a9c70dda6ec16720298be64f16e9a4"

    test := testUtils.TestCase{

        Description: "Test source-hub acp with http, consecutive operations on different nodes, identity bug",

        SupportedACPTypes: immutable.Some(
            []testUtils.ACPType{
                testUtils.SourceHubACPType,
            },
        ),

        Actions: []any{
            testUtils.RandomNetworkingConfig(),

            testUtils.RandomNetworkingConfig(),

            testUtils.AddPolicy{

                Identity: immutable.Some(1),

                Policy: `
                    name: Test Policy

                    description: A Policy

                    actor:
                      name: actor

                    resources:
                      users:
                        permissions:
                          read:
                            expr: owner + reader + writer

                          write:
                            expr: owner + writer

                          nothing:
                            expr: dummy

                        relations:
                          owner:
                            types:
                              - actor

                          reader:
                            types:
                              - actor

                          writer:
                            types:
                              - actor

                          admin:
                            manages:
                              - reader
                            types:
                              - actor

                          dummy:
                            types:
                              - actor
                `,

                ExpectedPolicyID: expectedPolicyID,
            },

            testUtils.SchemaUpdate{
                Schema: fmt.Sprintf(`
                        type Users @policy(
                            id: "%s",
                            resource: "users"
                        ) {
                            name: String
                            age: Int
                        }
                    `,
                    expectedPolicyID,
                ),
            },

            testUtils.CreateDoc{
                Identity: immutable.Some(1),

                NodeID: immutable.Some(0),

                CollectionID: 0,

                DocMap: map[string]any{
                    "name": "Shahzad",
                },
            },

            testUtils.CreateDoc{
                Identity: immutable.Some(1),

                NodeID: immutable.Some(1),

                CollectionID: 0,

                DocMap: map[string]any{
                    "name": "Shahzad Lone",
                },
            },
        },
    }

    testUtils.ExecuteTestCase(t, test)
}

Output:

Error: Received unexpected error: 403: forbidden

[shahzadlone]

Bumped down in priority, I think this is not a prod issue, might just be a testing framework issue the way we reuse identity bearer tokens between nodes, but they should have different tokens I believe due to different audience values (i.e. node host values) which might be causing this.