taskcluster / taskgraph

Generates task dependency graphs for Taskcluster CI
Mozilla Public License 2.0
14 stars 39 forks source link

Replace voluptuous with a faster schema validator #8

Open ahal opened 2 years ago

ahal commented 2 years ago

Imported from: https://bugzilla.mozilla.org/show_bug.cgi?id=1652123

When profiling taskgraph in Gecko, we discovered that voluptuous is responsible for a significant performance hit to Taskgraph. So much so that we disbale schema validation entirely when taskgraph.fast is set. We also avoid using voluptuous to set default values and instead set defaults via a transform.

There are many more performant validation libraries out there. In another project, I once successfully replaced voluptuous with validx (but we should investigate all options).

Alphare commented 3 months ago

I have tried migrating over the task_decision_schema to validx and with careful ordering of the OneOf attributes to prevent error creation and fallback and general attention to validx best practices, validating the schema drops from ~4s to ~1s over all calls within a large decision task when run on my computer.

This was an experiment to see whether this would make any impact and it does show promise, but only if we can add native extensions (validx uses Cython) to the decision task, and migrate over more schemas.

See https://hg.mozilla.org/try/rev/fcdcde5e36caba6bd74bf7c5ee41114874fec397