brandonchinn178 / aeson-schemas

Easily consume JSON data on-demand with type-safety
http://hackage.haskell.org/package/aeson-schemas
BSD 3-Clause "New" or "Revised" License
52 stars 1 forks source link

Optimize including schemas #60

Closed brandon-leapyear closed 3 years ago

brandon-leapyear commented 3 years ago

:sparkles: This is an old work account. Please reference @brandonchinn178 for all future communication :sparkles:


When building a schema, keep the type alias in the generated schema, and don't fully realize it until we need to reify it again. Making it constant time regardless of how big the included schema is.

Include given # of schemas master PR
1 15.92 µs 12.30 µs
5 66.15 µs 50.41 µs
10 127.1 µs 95.59 µs
100 1860 µs 1527 µs
Include schema with given # of keys master PR
1 14.13 µs 10.26 µs
5 27.73 µs 10.13 µs
10 44.07 µs 10.29 µs
100 343.6 µs 10.55 µs

This also reduces the memory usage when typechecking schemas that include other schemas. In a private project, I got the following metrics:

1.2.0 master PR
2.5 GB 4 GB 3 GB

We could probably improve it more, but this PR goes a long way.

Changes

This is the key change:

https://github.com/LeapYear/aeson-schemas/pull/60/files#diff-7850cf57b4077811008b45abad3e90c9L184-R184

-SchemaDefInclude other -> toSchemaObjectV <$> reifySchema other
+SchemaDefInclude other -> return $ SchemaInclude $ Left $ NameRef other

when we include a schema, we no longer reify the entire schema + inline it in this schema. Instead, we hold a reference to it. Then we reference the other schema directly when generating the type-level schema:

https://github.com/LeapYear/aeson-schemas/pull/60/files#diff-b2c04c02961d4bc662fa5d47779b7610R182

+SchemaInclude (Left name) -> [t| 'SchemaInclude ('Right $(conT . reifiedSchemaName =<< lookupSchema name)) |]

The type-level schema will then be changed like this:

 type MySchema = 'Schema
-  '[ '( 'NormalKey "other", ToSchemaObject OtherSchema )
+  '[ '( 'NormalKey "other", 'SchemaInclude ('Right OtherSchema) )
    ]

and then from the typechecker's point of view, it's reusing the already-checked OtherSchema type instead of resolving the type family application ToSchemaObject OtherSchema.

codecov-commenter commented 3 years ago

Codecov Report

Merging #60 into master will decrease coverage by 0.45%. The diff coverage is 26.92%.

Impacted file tree graph

@@            Coverage Diff             @@
##           master      #60      +/-   ##
==========================================
- Coverage   26.48%   26.03%   -0.46%     
==========================================
  Files          13       13              
  Lines         487      507      +20     
  Branches       21       23       +2     
==========================================
+ Hits          129      132       +3     
- Misses        349      366      +17     
  Partials        9        9              
Impacted Files Coverage Δ
src/Data/Aeson/Schema/Internal.hs 0.00% <0.00%> (ø)
src/Data/Aeson/Schema/TH/Schema.hs 0.00% <0.00%> (ø)
src/Data/Aeson/Schema/Type.hs 6.25% <0.00%> (-0.90%) :arrow_down:
src/Data/Aeson/Schema/TH/Utils.hs 46.66% <25.00%> (-2.81%) :arrow_down:
src/Data/Aeson/Schema/TH/Unwrap.hs 33.33% <80.00%> (+0.60%) :arrow_up:

Continue to review full report at Codecov.

Legend - Click here to learn more Δ = absolute <relative> (impact), ø = not affected, ? = missing data Powered by Codecov. Last update 1177696...bef3b46. Read the comment docs.