search

Oudwins / zog

Go with Zod inspired simple schema validation
MIT License
153 stars 5 forks source link
error-handling golang runtime-validation schema-validation templ translation validation

readme


Zog, a Zod-like schema parser & validator

ZOG - A Zod & Yup like Schema Parser & Validator for GO

Go Reference License: MIT Go Report Card Coverage Status

Zog is a schema builder for runtime value parsing and validation. Define a schema, transform a value to match, assert the shape of an existing value, or both. Zog schemas are extremely expressive and allow modeling complex, interdependent validations, or value transformations.

Killer Features:

API Stability:

  • I will consider the API stable when we reach v1.0.0
  • However, I believe very little API changes will happen from the current implementation. The APIs are are most likely to change are the data providers (please don't make your own if possible use the helpers whose APIs will not change meaningfully) and the ParseCtx most other APIs should remain the same
  • Zog will not respect semver until v1.0.0 is released. Expect breaking changes (mainly in non basic apis) until then.

Introduction

1 Install

go get github.com/Oudwins/zog

2 Create a schema & a struct

import (
  z "github.com/Oudwins/zog"
   )

var nameSchema = z.Struct(z.Schema{
  // its very important that schema keys like "name" match the struct field name NOT the input data
  "name": z.String().Min(3, z.Message("Override default message")).Max(10),
})

var ageSchema = z.Struct(z.Schema{
  "age": z.Int().GT(18).Required(z.Message("is required")),
})

// Merge the schemas creating a new schema
var schema = nameSchema.Merge(ageSchema)

type User struct {
  Name string `zog:"firstname"` // tag is optional. If not set zog will check for "name" field in the input data
  Age int
}

3 Parse the struct

func main() {
  u := User{}
  m := map[string]string{
    "firstname": "", // won't return an error because fields are optional by default
    "age": "30", // will get casted to int
  }
  errsMap := schema.Parse(m, &u)
  if errsMap != nil {
    // handle errors -> see Errors section
  }
  u.Name // ""
  u.Age // 30
}

4 You can also parse individual fields

var t = time.Time
errsList := Time().Required().Parse("2020-01-01T00:00:00Z", &t)

5 And do stuff before and after parsing

var dest []string
Slice(String().Email().Required()).PreTransform(func(data any, ctx z.ParseCtx) (any, error) {
  s := val.(string)
  return strings.Split(s, ","), nil
}).PostTransform(func(destPtr any, ctx z.ParseCtx) error {
  s := val.(*[]string)
  for i, v := range s {
    s[i] = strings.TrimSpace(v)
  }
  return nil
}).Parse("foo@bar.com,bar@foo.com", &dest) // dest = [foo@bar.com bar@foo.com]

6 Use the zhttp package to parse JSON, Forms or Query Params

err := userSchema.Parse(zhttp.Request(r), &user)

Core Design Decisions

A WORD OF CAUTION. ZOG & PANICS Zog will never panic due to invalid input but will always panic if invalid destination is passed to the Parse function (i.e if the destination does not match the schema).

var schema = z.Struct(z.Schema{
  "name": z.String().Required(),
})
// This struct is a valid destionation for the schema
type User struct {
  Name string
  Age int // age will be ignored since it is not a field in the schema
}
// this struct is not a valid destination for the schema. It is missing the name field. This will cause a panic even if the input data is map[string]any{"name": "zog"}
type User2 struct {
  Email string,
  Age int
}

Changes from zod:

Limitations

Most of these things are issues we would like to address in future versions.

Helper Packages

For convenience zog provides three helper packages:

zhttp: helps parse http requests

import (
  z "github.com/Oudwins/zog"
  "github.com/Oudwins/zog/zhttp"
)
var userSchema = z.Struct(z.Schema{
  "name": z.String().Required(),
  "age": z.Int().Required().GT(18),
})

func handlePostRequest(w http.ResponseWriter, r *http.Request) {
  var user := struct {
    Name string
    Age int
  }
  // if using json (i.e json Content-Type header):
  errs := userSchema.Parse(zhttp.Request(r), &user)
  // if using form data (i.e Content-Type header = application/x-www-form-urlencoded)
  errs := userSchema.Parse(zhttp.Request(r), &user)
  // if using query params (i.e no http Content-Type header)
  errs := userSchema.Parse(zhttp.Request(r), &user)
  if errs != nil {
  }
  user.Name // defined
  user.Age // defined
}

zenv: helps validate environment variables

import (
  z "github.com/Oudwins/zog"
  "github.com/Oudwins/zog/zenv"
)

var envSchema = z.Struct(z.Schema{
    "PORT": z.Int().GT(1000).LT(65535).Default(3000),
    "Db": z.Struct(z.Schema{
        "Host": z.String().Default("localhost"),
        "User": z.String().Default("root"),
        "Pass": z.String().Default("root"),
    }),
})
var Env = struct {
    PORT int // zog will automatically coerce the PORT env to an int
    Db   struct {
        Host string `zog:"DB_HOST"` // we specify the zog tag to tell zog to parse the field from the DB_HOST environment variable
        User string `zog:"DB_USER"`
        Pass string `zog:"DB_PASS"`
    }
}{}

// Init our typesafe env vars, panic if any envs are missing
func Init() {
  errs := envSchema.Parse(zenv.NewDataProvider(), &Env)
  if errs != nil {
    log.Fatal(errs)
  }
}

// if you want to always panic on error
var Env = parse()
func Parse() env {
    var e env
    errs := envSchema.Parse(zenv.NewDataProvider(), &e)
    if errs != nil {
        fmt.Println("FAILURE TO PARSE ENV VARIABLES")
        log.Fatal(z.Errors.SanitizeMap(errs))
    }
    return e
}

zi18n: helps with having error messages in multiple languages There are two use cases for zi18n:

  1. You just want to change the language of your error messages
  2. You want to support having errors in multiple languages because your app is in 2+ languages

USE CASE 1:

// import one of our supported languages
import (
  "github.com/Oudwins/zog/i18n/es"
  "github.com/Oudwins/zog/i18n/en"
  "github.com/Oudwins/zog/conf" // import the zog configuration packag
)
// override the default error map
conf.DefaultErrMsgMap = es.Map // now all errors will be in spanish!

USE CASE 2:

// Somewhere when you start your app
import (
  "github.com/Oudwins/zog/i18n" // import the i18n library
  "github.com/Oudwins/zog/i18n/es" // import any of the supported language maps or build your own
  "github.com/Oudwins/zog/i18n/en"
)
i18n.SetLanguagesErrsMap(map[string]i18n.LangMap{
  "es": es.Map,
  "en": en.Map,
},
"es", // default language
i18n.WithLangKey("langKey"), // (optional) default lang key is "lang"
)

// Now when we parse
schema.Parse(data, &dest, z.WithCtxValue("langKey", "es")) // get spanish errors
schema.Parse(data, &dest, z.WithCtxValue("langKey", "en")) // get english errors
schema.Parse(data, &dest) // get default lang errors (spanish in this case)

Parsing Context

Zog uses a ParseCtx to pass around information related to a specific schema.Parse() call. Currently use of the parse context is quite limited but it will be expanded upon in the future.

uses

  1. Pass custom data to custom functions

Here is an example with a pretransform

nameSchema := z.String().Min(3).PreTransform(func(data any, ctx z.ParseCtx) (any, error) {
  char := ctx.Get("split_by")
  return strings.Split(data.(string), char), nil
})
nameSchema.Parse("Michael Jackson", &dest, z.WithCtxValue("split_by", " "))
  1. Change the error formatter
nameSchema := z.String().Min(3)
nameSchema.Parse(data, &dest, z.WithErrFormatter(MyCustomErrorMessageFormatter))

Errors

Zog creates its own error interface called ZogError which also implements the error interface.

type ZogError interface {
    Code() ZogErrCode // unique error code
    Value() any    // the value that caused the error
    Dtype() ZogType // type of the destionation
    Params() map[string]any // params for the error as defined by
    Message() string
    SetMessage(string)
    Error() string // returns the string representation of the ZogError. The same as String(). Printf will not print correctly otherwise.
  String() string // returns the string representation of the ZogError. Example: "ZogError{Code: '', ...}"
    Unwrap() error // returns the wrapped error.
}

This is what will be returned by the Parse function. To be precise:

For example:

// ! WARNING EXAMPLES HERE ARE SHOWN AS IF ZogError IS A STRUCT FOR EASIER READING BUT IT IS AN INTERFACE
errList := z.String().Min(5).Parse("foo", &dest) // can return []z.ZogError{z.ZogError{Message: "min length is 5"}} or nil
errMap := z.Struct(z.Schema{"name": z.String().Min(5)}).Parse(data, &dest) // can return map[string][]z.ZogError{"name": []z.ZogError{{Message: "min length is 5"}}} or nil

// Slice of 2 strings with min length of 5
errsMap2 := z.Slice(z.String().Min(5)).Len(2).Parse(data, &dest) // can return map[string][]z.ZogError{"$root": []z.ZogError{{Message: "slice length is not 2"}, "[0]": []z.ZogError{{Message: "min length is 5"}}}} or nil

Additionally, z.ZogErrMap will use the field path as the key. Meaning

// ! WARNING EXAMPLES HERE ARE SHOWN AS IF ZogError IS A STRUCT FOR EASIER READING BUT IT IS AN INTERFACE
errsMap := z.Struct(z.Schema{"inner": z.Struct(z.Schema{"name": z.String().Min(5)}), "slice": z.Slice(z.String().Min(5))}).Parse(data, &dest)
errsMap["inner.name"] // will return []z.ZogError{{Message: "min length is 5"}}
errsMap["slice[0]"] // will return []z.ZogError{{Message: "min length is 5"}}

$root & $first are reserved keys for both Struct & Slice validation, they are used for root level errors and for the first error found in a schema, for xample:

errsMap := z.Slice(z.String()).Min(2).Parse(data, &dest)
errsMap["$root"] // will return []z.ZogError{{Message: "slice length is not 2"}}
errsMap["$first"] || errsMap.First() // (equivalent) will return the same in this case []z.ZogError{{Message: "slice length is not 2"}}

Sanitizing Errors

If you want to return errors to the user without the possibility of exposing internal errors, you can use the Zog sanitizer functions z.Errors.SanitizeMap(errsMap) or z.Errors.SanitizeSlice(errsSlice). These functions will return a map or slice of strings of the error messages (stripping out the internal error).

Error Formatting & i18n

Please also look at the section on the i18n package above.

You have many different options for handling the formatting of error messages. Here are a few examples:

OPTION 1: Use z.Message() or z.MessageFunc() to customize the error messages

userSchema := z.Struct(z.Schema{
  "name": z.String().Min(3, z.Message("min length is 3")),
  "age": z.Int().GT(18, z.MessageFunc(func(e z.ZogError, p z.ParseCtx) {
    e.SetMessage("age must be greater than 18")
  })),
})

This is the simplest option but it has some limitations, for example you cannot customize the error messages for coercion errors.

OPTION 2: Iterate over the returned errors and create custom messages

errs := userSchema.Parse(data, &user)
msgs := formatZogErrors(errs)

func FormatZogErrors(errs z.ZogErrMap) map[string][]string {
  // iterate over errors and create custom messages based on the error code, the params and destination type
}

OPTION 3: Override the default error formatter Under the conf package you will find ErrorFormatter which is a function that is in charge of formatting the error messages. You can override this function to customize the error messages for all zog schemas.

You can also override the DefaultErrMsgMap which is a map[zogType][ErrCode] or just specific keys to customize the error messages for specific zog types or codes.

For more information on this see the overriding defaults section

OPTION 4: Override the error formatter for a specific parsing execution

userSchema.Parse(data, &user, z.WithErrFormatter(conf.ErrorFormatter)) // set the error formatter for this parse to be the default formatter. This doesn't make a lot of sense but you can pass a custom one

What do I recommend for i18n? If you are serius about i18n and want to translate all your messages and return meaningful error messages I recommend you choose option 3 and do something like this:

zconf.ErrorFormatter = func(e p.ZogError, p z.ParseCtx) {
  lang := p.Get("lang").(string)
  // generate a different error message based on the language
}

// then when you parse a schema you can pass the language
userSchema.Parse(data, &user, z.WithCtxValue("lang", "en"))

Example ways of delivering errors to users

Using HTML templates

(In this example I'll use go templ, but you can use any template engine)

Example use case: simplified Signup form validation Imagine our handler looks like this:

type SignupFormData struct {
  Email string
  Password string
}

schema := z.Struct(z.Schema{
  "email": z.String().Email().Required(),
  "password": z.String().Min(8).Required(),
})

func handleSignup(w http.ResponseWriter, r *http.Request) {
  var signupFormData = SignupFormData{}
  errs := schema.Parse(zhttp.NewRequestDataProvider(r), &signupFormData)

  if errs != nil {
    www.Render(signupFormTempl(&signupFormData, errs))
  }
  // handle successful signup
}

templ signupFormTempl(data *SignupFormData, errs z.ZogErrMap) {
  <input type="text" name="email" value={data.Email}>
  // display only the first error
  if e, ok := errs["email"]; ok {
    <p class="error">{e[0].Message()}</p>
  }
  <input type="text" name="password" value={data.Password}>
  // display only the first error
  if e, ok := errs["password"]; ok {
    <p class="error">{e[0].Message()}</p>
  }
}

PS: If you are using go html templates & tailwindcss you might be interesting in my port of tailwind-merge to go.

REST API Responses

Zog providers a helper function called z.Errors.SanitizeMap(errsMap) that will return a map of strings of the error messages (stripping out the internal error). So, if you do not mind sending errors to your users in the same form zog returns them, you can do something like this:

errs := schema.Parse(zhttp.Request(r), &userFormData)

if errs != nil {
  sanitized := z.Errors.SanitizeMap(errs)
  // sanitize will be map[string][]string
  // for example:
  // {"name": []string{"min length is 5", "max length is 10"}, "email": []string{"is not a valid email"}}

  // ... marshal sanitized to json and send to the user
}

Creating Custom Tests

Zog supports two main ways of creating custom tests a simple function and a struct.

OPTION 1: Full Test Definition

nameSchema := z.String().Test(z.Test{
  ErrCode: "custom_error_code", // this is the error code for all errors constructed from this test
  Params: map[string]any{"foo": "bar"}, // these will be passed to the error formatter via the constructed error
  ErrFmt: func(e z.ZogError, p z.ParseCtx) // This is what z.Message() sets, its a function to format the error message
  ValidateFunc: func(data any, ctx z.ParseCtx) bool{...}, // this is the validate func
})

Beware that if using the default error formatter you will get the default error message for any error code that is not defined see error formatting. You can define the ErrFmt function or use z.Message() like so:

nameSchema := z.String().Test(MyCustomTest(), z.Message("hello world") ) // MyCustomTest() returns the z.Test so it can be reused

OPTION 2: use z.TestFunc Helper

// here I will set a custom message so I don't have to add my message to the error messages map
nameSchema := z.String().Test(z.TestFunc("error_code", func (data any, ctx z.ParseCtx)bool{...}), z.Message("custom msg"))

Reference

Generic Schema Methods

These are methods that can be used on most types of schemas


// marks the schema as required. Remember fields are optional by default
schema.Required(z.Message("message or function"))
schema.Optional() // marks the schema as optional (default)
// optional & required are mutually exclusive
schema.Required().Optional() // marks the schema as optional

schema.Default(val) // sets the default value. See Zog execution flow
schema.Catch(val) // sets the catch value. A value to use if the validation fails. See Zog execution flow

schema.PreTransform(func(data any, ctx z.ParseCtx) (any, error) {}) // transforms the value before validation. returned value will override the input value. See Zog execution flow. errors returned from this will be wrapped in a ZogError under the "custom" code (You may also return a ZogError from this function)

schema.PostTransform(func(destPtr any, ctx z.ParseCtx) error {}) // transforms the value after validation. Receives a pointer to the destination value. errors returned from this will be wrapped in a ZogError under the "custom" code (you may also return a ZogError from this function)

Types

// Primtives. Calling .Parse() on these will return []ZogError
String()
Int()
Float()
Bool()
Time()

// Complex Types. Calling .Parse() on these will return map[string][]ZogError. Where the key is the field path ("user.email") & $root is the list of complex type level errors not the specific field errors
Struct(Schema{
  "name": String(),
})
Slice(String())

Strings

// Validations
String().Min(5)
String().Max(10)
String().Len(5)
String().Email()
String().URL()
String().UUID()
String().Match()
String().Contains(string)
String().ContainsUpper()
String().ContainsDigit()
String().ContainsSpecial()
String().HasPrefix(string)
String().HasSuffix(string)
String().OneOf([]string{"a", "b", "c"})

Numbers

// Validators
Int().GT(5)
Float().GTE(5)
Int().LT(5)
Float().LTE(5)
Int().EQ(5)
Float().OneOf([]float64{1.0, 2.0, 3.0})

Booleans

Bool().True()
Bool().False()

Times & Dates

Use Time to validate time.Time instances

Time().After(time.Now())
Time().Before(time.Now())
Time().Is(time.Now())

Structs

s := z.Struct(z.Schema{
  "name": String().Required(),
  "age": Int().Required(),
})
user := struct {
  Name string `zog:"firstname"` // name will be parsed from the firstname field
  Age int // since zog tag is not set, age will be parsed from the age field
}
s.Parse(map[string]any{"firstname": "hello", "age": 10}, &user)

Slices

s := Slice(String())

Slice(Int()).Min(5)
Slice(Float()).Max(5)
Slice(Bool()).Length(5)
Slice(String()).Contains("foo")

Overriding Defaults

Zog uses internal functions to handle many aspects of validation & parsing. We aim to provide a simple way for you to customize the default behaviour of Zog through simple declarative code inside your project. You can find the options you can tweak & override in the conf package (github.com/Oudwins/zog/conf).

Overriding the default coercer functions

Lets go through an example of overriding the float64 coercer function, because we want to support floats that use a comma as the decimal separator.

import (
  // import the conf package
    "github.com/Oudwins/zog/conf"
)

// we override the coercer function for float64
conf.Coercers.Float64 = func(data any) (any, error) {
  str, ok := data.(string)
  // identify the case we want to override
  if !ok && strings.Contains(str, ",") {
    return MyCustomFloatCoercer(str)
  }
  // fallback to the original function
  return conf.DefaultCoercers.Float64(data)
}

Overriding the default error formatting

There are two different things you can override from the errors configuration:

  1. The error formatter function
  2. The error messages map. Which is a map[zogType][ErrCode] or just specific keys to customize the error messages for specific zog types or codes.
// override the error formatter function
conf.ErrorFormatter = func(e p.ZogError, p z.ParseCtx) {
  // do something with the error
  ...
  // fallback to the default error formatter
  conf.DefaultErrorFormatter(e, p)
}

// override specific error messages
// For this I recommend you import `zod/zconst` which contains zog constants
conf.DefaultErrMsgMap[zconst.TypeString]["my_custom_error_code"] = "my custom error message"
conf.DefaultErrMsgMap[zconst.TypeString][zconst.ErrCodeRequired] = "Now all required errors will get this message"

Zog Schema Parsign Execution Structure

Zog Schema Parsign Execution Structure

  1. Pretransforms
    • On error all parsing and validation stops and error is returned.
    • Can be caught by catch
  2. Default Check -> Assigns default value if the value is nil value
  3. Optional Check -> Stops validation if the value is nil value
  4. Casting -> Attempts to cast the value to the correct type
    • On error all parsing and validation stops and error is returned
    • Can be caught by catch
  5. Required check ->
    • On error: aborts if the value is its nil value and returns required error.
    • Can be caught by catch
  6. Tests -> Run all tests on the value (including required)
    • On error: validation errors are added to the errors. All validation functions are run even if one of them fails.
    • Can be caught by catch
  7. PostTransforms -> Run all postTransforms on the value.
    • On error you return: aborts and adds your error to the list of errors
    • Only run on valid values. Won't run if an error was created before the postTransforms

Roadmap

These are the things I want to add to zog before v1.0.0

Acknowledgments

License

This project is licensed under the MIT License - see the LICENSE file for details.