brandonchinn178 / aeson-schemas

Easily consume JSON data on-demand with type-safety
http://hackage.haskell.org/package/aeson-schemas
BSD 3-Clause "New" or "Revised" License
53 stars 1 forks source link

Regression in memory usage with v1.3.0 #64

Closed brandon-leapyear closed 3 years ago

brandon-leapyear commented 4 years ago

When upgrading another project to use aeson-schemas 1.3.x, the builds in Circle CI were starting to fail with

Process exited with code: ExitFailure (-9) (THIS MAY INDICATE OUT OF MEMORY)

In investigating, it seems like GHC is taking twice as much memory to compile the schemas using aeson-schemas-1.3.x, compared to 1.2.x. If I would rerun stack build, it would succeed, because the build would start where it left off. Since this failure occurred in a module without schema quasiquotes, it seems like the issue is in type checking the built schemas.

The change in schema representation is:

 type PushEvent =
-  'SchemaObject
+  'Schema
-    '[ '( 'NormalKey "ref", 'SchemaText),
+    '[ '( 'NormalKey "ref", SchemaScalar Text),
-       '( 'NormalKey "base_ref", 'SchemaMaybe 'SchemaText),
+       '( 'NormalKey "base_ref", 'SchemaMaybe ('SchemaScalar Text)),
-       '( 'NormalKey "after", 'SchemaCustom GitObjectID),
+       '( 'NormalKey "after", 'SchemaScalar GitObjectID),
...
-       '( 'NormalKey "sender", UserShort),
+       '( 'NormalKey "sender", ToSchemaObject UserShort),
...
      ]
-  :: SchemaType
+  :: Schema' ghc-prim-0.5.3:GHC.Types.Symbol *

The only major change I see is the change from UserShort to ToSchemaObject UserShort, which is due to the fact that UserShort now has the kind of Schema, so we need to convert it back into a SchemaType using ToSchemaObject. It might be the case that resolving the type family forces GHC to re-typecheck that AST.

If other people come across this failure, there are some workarounds:

  1. Bump up memory allocation. If you're on the paid plan for Circle CI, you can bump the resource class of the image to get more computational power.
  2. Add {-# OPTIONS_GHC -O0 #-} to the failing module.

I'm currently experimenting with ways to just store a reference to UserShort instead of directly interpolating it with ToSchemaObject. Work started in #60