Validated fields not supporting Unicode?

[OverflowCat]

When using CJK characters in fields with x-www-form-urlencoded, a 500 Internal Server Error occurs.

I didn't put slug in options, so it appears it isn't validated, allowing me to use CJK characters there.

[zanechua]

@OverflowCat there's two distinct issues here right?

1. Support for Unicode so we can get CJK
2. Validation not happening for something?

EDIT:

Please also provide the body in text format so I can validate this.

EDIT 2:

Please also provide your staticman.yml configuration or a repro project.

[OverflowCat]

Nope, I mean validation may cause such a bug. I found that Unicode cannot be used in fields, but it can be in options. Additionally, this issue persists when submitting via application/json, so it's not related to the submission method used.

Body:

{
  "fields": {
    "name": "hello世界",
    "email": "someone@example.com",
    "message": "Hello"
  },
  "options": {
    "slug": "test/page"
  }
}

Remove 世界 in the json and it should work.

Config file:

comments:
  allowedFields: ["name", "email", "url", "message", "twitter", "activitypub"]

  branch: "src"

  commitMessage: "comment: add staticman data"

  filename: "entry{@timestamp}"

  format: "yaml"

  generatedFields:
    date:
      type: date
      options:
        format: "timestamp-seconds"

  moderation: true

  name: "my-site.com"

  path: "comments/{options.slug}"

  requiredFields: ["name", "email", "message"]

  transforms:
    email: md5

[zanechua]

Thanks.

Let me check in on this in a bit. I think I should support unicode either way otherwise we won't get emoji support either.

For options.. that's an interesting one because I referenced the original staticman implementation and there wasn't much details around options except in the getting started docs.

Both the fields and options should be getting validated and escaped. Not sure why the behaviour is different.

[zanechua]

UTF8 support added in:

https://github.com/zanechua/comment-worker/commit/4f88f2869e6fc3915fbbead956fc5eeab582d48d