search

openfga / cli

A cross-platform CLI to interact with an OpenFGA server
https://openfga.dev
Apache License 2.0
50 stars 26 forks source link

feat(generate): generate pkl testing #345

Open toniphan21 opened 5 months ago

toniphan21 commented 5 months ago

Introduce new command generate to generate some code for testing using pkl-lang.

Motivation

I want to have a testing system for an Authorization Model that:

Solution

Using pkl-lang to write tests instead of yaml. The process of testing is:

However, written tests in pkl lang is not the completed solution, we need something to decouple the business logic and authorization model. It means if the authorization model changed, the changes we make in pkl files should be minimum (it's not possible with yaml because all tuples are tired to the authorization model structure). So the final architecture looks like this:

pkl-testing

Usage example:

By doing this way there are some advantages:

  1. Firstly the tests are much easier to understand compare to yaml file
  2. Developer can understand which kind of tuples need to be produced based on Type in the Adapter layer
  3. Non-Technical people can participate in early-phase of a model development
  4. All Type mistakes could be avoided when writing tests for the model

Changes description

Introduce a new command generate has a subcommand pkl which generate a small pkl code library with assertions and types from given openFGA model. The purpose of this command is provide functionality for Generate Pkl layer. The structure of generated directory looks like:

More about pkl-lang: https://pkl-lang.org/

Example

Run this command with the model custom-roles in sample-store.

fga generate pkl  --file=path-to-model/model.fga

The generated contents are:

testing/run

#!/bin/bash
param=$1
if [ "$param" == "all" ]; then
  tests=`ls ./test*.pkl`
  for test in $tests
  do
    echo "--- running test file: $test"
    pkl eval -f yaml $test | fga model test --tests /dev/stdin
    echo "--- finished"
  done
else
  pkl eval -f yaml $param | fga model test --tests /dev/stdin
fi

testing/type.pkl

import "lib/gen.pkl"
import "assertions.pkl"

class Asset extends gen.BaseAsset {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "asset"
  hidden fgaType: String = "asset"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class AssetCategory extends gen.BaseAssetCategory {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "asset-category"
  hidden fgaType: String = "asset-category"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class Org extends gen.BaseOrg {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "org"
  hidden fgaType: String = "org"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class Role extends gen.BaseRole {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "role"
  hidden fgaType: String = "role"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class Team extends gen.BaseTeam {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "team"
  hidden fgaType: String = "team"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

class User extends gen.BaseUser {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "user"
  hidden fgaType: String = "user"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  // this is where you make your tests look nicer, for example:
  // 
  // Using a function with descriptive name make setup of test is easier to understand
  // function has_user(i: User) = i.relation_member(this)
  // 
  // Using Assertions with descriptive name make the test easier to understand
  // function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
  //   assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

testing/assertions.pkl

import "lib/gen.pkl"

class AssetAssertions extends gen.BaseAssetAssertions {
  // you can put some custom assertions in this class

}

class AssetCategoryAssertions extends gen.BaseAssetCategoryAssertions {
  // you can put some custom assertions in this class

}

class OrgAssertions extends gen.BaseOrgAssertions {
  // you can put some custom assertions in this class

}

class RoleAssertions extends gen.BaseRoleAssertions {
  // you can put some custom assertions in this class

}

class TeamAssertions extends gen.BaseTeamAssertions {
  // you can put some custom assertions in this class

}

testing/lib/...

These files are too big, but it looks like this

....
abstract class BaseUser {
  id: String
  hidden type: String = "user"
  hidden fgaType: String = "user"

  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"

  function relation_assignee(i: BaseRole): Mapping = new Mapping {
    ["user"] = toFGAType()
    ["relation"] = "assignee"
    ["object"] = i.toFGAType()
  }

  function relation_member(i: BaseOrg|BaseTeam): Mapping = new Mapping {
    ["user"] = toFGAType()
    ["relation"] = "member"
    ["object"] = if (i.fgaType == "org") i.toFGAType()
      else if (i.fgaType == "team") i.toFGAType()
      else ""
  }

  function relation_owner(i: BaseOrg): Mapping = new Mapping {
    ["user"] = toFGAType()
    ["relation"] = "owner"
    ["object"] = i.toFGAType()
  }

  function assert_org(
    name: String,
    object: BaseOrg,
    customAssertions: BaseOrgAssertions,
    asserts: (BaseOrgAssertions) -> Listing
  ): Mapping = new Mapping {
    ["name"] = name
    ["check"] = new Listing {
      new Mapping {
        ["user"] = toFGAType()
        ["object"] = object.toFGAType()
        ["assertions"] = asserts.apply(customAssertions)
          .fold((new Mapping {}).toMap(), (r: Map, m: Mapping) -> r + m.toMap())
      }
    }
  }
....

Write Test after running the command

the type.pkl is where we put some setup helper functions, such as:

class Org extends gen.BaseOrg {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "org"
  hidden fgaType: String = "org"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  function has_user(i: User) = i.relation_member(this)

  // you could even set up more than 1 relation using this syntax
  function has_something(input: User) = new Listing {
    input.relation_member(this)
    input.relation_owner(this)
  }
}

In the code above, we use relation_member generated from the model to express that an org can has_user() with type safety. Then in the class User:

class User extends gen.BaseUser {
  // -------------------------------------------------------
  // pkl not support parent's properties yet, this help in case we want to use id
  // also if we use some union types we could change here.
  id: String
  hidden type: String = "user"
  hidden fgaType: String = "user"
  function toFGAType() = "\(type):\(id)"
  function toFGATypeWith(relation: String) = "\(type):\(id)#\(relation)"
  // -------------------------------------------------------

  function in_org(org: Org, asserts: (assertions.OrgAssertions) -> Listing) =
    assert_org("Test user \(id) in org \(org.id)", org, new assertions.OrgAssertions {}, asserts)

}

We express that we want to test a user in a specific org, and use generated assert helper handle the check.

Then we can start writing test for multiple-scenarios, each scenarios has it own setup and file, looks like test_example:

import "type.pkl"
import "lib/test.pkl"

// NOTE: this is where you define your type for testing, for example:
local Anna: type.User = new type.User { id = "Anna" }
local OpenFGA: type.Org = new type.Org { id = "OpenFGA" }

suite: test.OpenFGATestSuite = new {
  name = "Awesome test - RENAME ME"
  model = read("../model.fga")
  setup {
    // NOTE: setup your test here, for example:
    Anna.relation_owner(OpenFGA)
    OpenFGA.has_user(Anna)
  }
  tests {
    // NOTE: write your assertions here, for example:
    Anna.in_org(OpenFGA, (she) -> new Listing {
      she.should_be_member
      she.should_be_owner("comment or reason")
    })
  }
}

output { value = test.output_value(suite) }

as you see, in the setup we could use generated function or custom function (which is nicer), and in the tests we could test user Anna in a specific Organization with very nice syntax.

Now, to run the test use run utility, or by using raw command:

# raw command
pkl eval -f yaml test_example.pkl | fga model test --tests /dev/stdin

# run utility
./run test_example.pkl

# run all test file which start with `test*`
./run all

That's all.

Review Checklist

linux-foundation-easycla[bot] commented 5 months ago

CLA Signed


The committers listed above are authorized under a signed CLA.

stacklok-cloud[bot] commented 5 months ago

Minder Vulnerability Report ✅

Minder analyzed this PR and found it does not add any new vulnerable dependencies.

Vulnerability scan of 9f28e15e:

  • 🐞 vulnerable packages: 0
  • 🛠 fixes available for: 0
ewanharris commented 5 months ago

Hey @toniphan21, thanks for this PR and the well thought out description to bring us up to speed! ❤️

Just to set some expectations, it might take us a little while to get around to fully reviewing this PR. We'd like to make sure we fully understand the pieces that are new to us (pkl, generation of files outside the model) and how the full model development cycle looks with this change (e.g. local and CI usage, potentially expanding to how this may fit into areas like the playground).

We're definitely excited to see the suggestions you have for improving the testing of a model. Out of curiosity is this something you're using to author tests currently?

toniphan21 commented 5 months ago

Hey @ewanharris, thank you for reviewing this PR. I understand it's a big one, so I don't expect the review process to be fast. Also, you can see I haven't added any tests yet. The reason is that I want to gather your feedback before adding them.

To answer your question, yes, I do use these tests at work to verify the model with non-technical stakeholders. That's why readability is the main focus.

Let me walk you through the process of how I came up with the idea, and then I'll suggest some approaches for integrating pkl into openfga's ecosystem.

First, I started with the yaml file for testing a model:

name: test
model: ''
tuples:
- user: user:1
  relation: member
  object: org:1
tests:
- name: whatever
  check:
  - user: user:1
    object: object
    assertions:
      allow: true

Then I wrote a pkl file which could be converted to the exact yaml file above:

name: String = "test"
model: String = ""
tuples: Listing<Tuple> = new {
  new {
    user = "user:1"
    relation = "member"
    object = "org:1"
  }
}
tests: Listing<TestCase> = new {
  new {
    name = "whatever"
    check {
      new {
        user = "user:1"
        object = "object"
        assertions {
          ["allow"] = true
        }
      }
    }
  }
}

class Test {
  name: String
  model: String?
  tuples: Listing<Tuple>
  tests: Listing<TestCase>
}

class Tuple {
  user: String
  relation: String
  object: String
}

class TestCase {
  name: String?
  check: Listing<Check>
}

class Check {
  user: String
  object: String
  assertions: Mapping<String, Boolean>
}

As you can see, the raw pkl file is longer than the yaml file. However, there are some upsides to using pkl:

new {
    user = "user:1"
    relation = "member"
    object = "org:1"
  }

becomes

// somewhere at the top of the pkl file
local Anna = new User { id = "1" }
local OpenFGA = new Org { id = "1" }

Anna.member(OpenFGA) // member is a function of class User which returns the structure above

The benefit is that I can name member whatever I want and still produce the same output. I can condense every setup (producing a tuple) into one line. Another advantage is that in the function, we can produce more than one tuple (for example, a user belongs to an org - one tuple, and the org has the member - another tuple).

In short, the pkl file is just a nicer way to express ideas or business logic; in the end, it will be converted to yaml and tested using the openfga cli.

However, I faced a problem where I have to write a pkl class every time my model changes. That’s why I want to generate pkl files automatically based on the model, saving time and avoiding mistakes.

I suggest that:

I think this is a lot of work, and it’s not easy to integrate a new language into an ecosystem. However, I believe pkl offers a lot compared to yaml, and generating source code from a model is a good start. Please test this PR with any models in the sample store and give me feedback. I hope the idea is clearer after you've tried it.